[Voyage-linux] Best Practices for writing logs & data to CF

Thu Jul 11 19:11:34 HKT 2013

Hi,

I use Voyage-Linux since 3 years as WiFi HotSpot with Coova Chilli, Apache2, PHP5, MySQL and FreeRadius.

As I need to log every users actions, I’ve completely disabled the read-only system.

I use regular CF (MLC ship, not SLC) and I’m very happy with it.

Some of my boxes are running since 3 years and I never encounter any problems on any CF…

I didn’t tweak any parameter, I just use Voyage-Linux stock config and it works great for me.

Regards,

Sylvain

De : Voyage-linux [mailto:voyage-linux-bounces+sylvaingomez=free.fr at list.voyage.hk] De la part de Jordi Soucheiron
Envoyé : jeudi 11 juillet 2013 10:34
À : Nils Toedtmann
Cc : voyage-linux
Objet : Re: [Voyage-linux] Best Practices for writing logs & data to CF

Jordi Soucheiron
Software Engineer
 <mailto:jsoucheiron at dexmatech.com> jsoucheiron at dexmatech.com

DEXMA

Marie Curie, 8-14, Barcelona, ES-08042

5 Penn Plaza, Suite 2300, New York, NY 10001
t: (+34) 93 181 01 96

www.dexmatech.com

Follow us on Twitter <http://www.twitter.com/dexma>  and Linkedin <http://www.linkedin.com/company/247838> 
Síguenos en Twitter <http://www.twitter.com/dexma_ES>  y Linkedin <http://www.linkedin.com/company/dexma-tech> 

2013/7/11 Nils Toedtmann <voyage-linux-mail at nils.toedtmann.net>

Dear Voyage community

First: Voyage Linux is great! It makes building embedded systems
significantly easier - thanks to everyone who contributed!

I have a number of questions about writing data to CF on ALIX (before
you rant about even wanting to do this - see my use case and motivation
in the PS below).

I apologize in advance if i haven't found existing answers in the Voyage
documentation. I did do some research, but admit that being only a
sysadmin, i didn't dive into the depths of Voyage's development docs.

Here are my questions:

 * Most documentation says that CF wears down quickly when many write
operations happen. But those docs are 5-10 years old. Do modern
'industrial grade' CF cards still suffer from this problem? I wonder
because nowadays it is pretty common to use flash based SSDs in
write-heavy systems.

Heavy writing in SSD is not usually a good idea. The disks will inevitably wear down eventually. It's only a matter of time, but it may be worth it depending on what you need to do. 

 * PC Engine recommend [1] to have partitions aligned to flash's 2K or
4K page size. But most partitioning software assumes 63 sectors (512
bytes each) per track. How would i partition a CF card such that the
partitions are aligned properly, e.g. using Voyage's fdisk? Maybe just
use multiples of '8 tracks = 63*4K'?

 * I have the rootfs on a ro partition, and write my data to a separate
rw partition. Is it safe to assume that even if a powercycle corrupts my
data partition, that the rootfs partition is still intact?

It should work just fine. I've known many people that has a factory default values in ro partitions and restore it in case of failure of the rw partition. I use rw for the whole disk most of the time and I mount the logs directories in ram. I usually sync this directories every hour or so to reduce the cf wear. 

 * Besides your recommendations to adjust the vm.dirty_* syscontrols
when writing to CF [2] - What are your recommended file systems and
mount options for rw data partitions to reduce the risk of data loss on
a powercycle? E.g. i see people recommending to use ext3 or ext4 with
'journal=data' and/or 'sync' (at the expense of reduced write speed). Is
that sound?

Journaling systems increase the number of writes to the CF. You should consider If it's worth it. Remember that ext3 is basically ext2 + journaling. 

 * Does Voyage Linux do a file system check when it mounts additional
file systems if they have fs_passno=2 (6th field) in /etc/fstab? Does
the fsck call include the "-y" option to force automatic fixing mode?

 * Would it work if i symlinked /var/log/ into a rw data partition and
removed it from 'SYNC_DIRS' in /etc/init.d/voyage-sync ?

You don't need to symlink it, just mount it there ;) And yes, it should work

Sorry but I can't help you with the others

Any advise would be greatly appreciated!

Cheers, /nils.

[1] http://www.pcengines.ch/cfperf.htm
[2] http://linux.voyage.hk/content/getting-started-v09x

PS: Disclaimer & use case:

I know that CF is not advised for permanent storage when powercycles can
happen at any time. We use ALIX + Voyage linux to collect and cache, and
send sensor data to central servers. The data we write to CF is not
critical. However, for a number of reasons it would be very helpful if
the data survived powercycles:

 * Cached sensor data that hasn't been sent out yet (some of our ALIXes
have unstable uplinks, e.g. 3G) would not get lost

 * Stored states would allow the data collection to resume significantly
quicker

 * Logs up to just before a system fault (e.g. OOM) would be very
helpful for debugging

_______________________________________________
Voyage-linux mailing list
Voyage-linux at list.voyage.hk
http://list.voyage.hk/mailman/listinfo/voyage-linux

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.voyage.hk/pipermail/voyage-linux/attachments/20130711/4a7c04bf/attachment.html>