[Voyage-linux] Best Practices for writing logs & data to CF

[Jordi Soucheiron]

*Jordi Soucheiron*
Software Engineer
jsoucheiron at dexmatech.com

*DEXMA*
Marie Curie, 8-14, Barcelona, ES-08042*
*
5 Penn Plaza, Suite 2300, New York, NY 10001
t: (+34) 93 181 01 96
www.dexmatech.com

Follow us on Twitter <http://www.twitter.com/dexma> and
Linkedin<http://www.linkedin.com/company/247838>
Síguenos en Twitter <http://www.twitter.com/dexma_ES> y
Linkedin<http://www.linkedin.com/company/dexma-tech>


2013/7/11 Nils Toedtmann <voyage-linux-mail at nils.toedtmann.net>

> Dear Voyage community
>
>
> First: Voyage Linux is great! It makes building embedded systems
> significantly easier - thanks to everyone who contributed!
>
>
> I have a number of questions about writing data to CF on ALIX (before
> you rant about even wanting to do this - see my use case and motivation
> in the PS below).
>
> I apologize in advance if i haven't found existing answers in the Voyage
> documentation. I did do some research, but admit that being only a
> sysadmin, i didn't dive into the depths of Voyage's development docs.
>
>
> Here are my questions:
>
>  * Most documentation says that CF wears down quickly when many write
> operations happen. But those docs are 5-10 years old. Do modern
> 'industrial grade' CF cards still suffer from this problem? I wonder
> because nowadays it is pretty common to use flash based SSDs in
> write-heavy systems.
>
Heavy writing in SSD is not usually a good idea. The disks will inevitably
wear down eventually. It's only a matter of time, but it may be worth it
depending on what you need to do.

>
>  * PC Engine recommend [1] to have partitions aligned to flash's 2K or
> 4K page size. But most partitioning software assumes 63 sectors (512
> bytes each) per track. How would i partition a CF card such that the
> partitions are aligned properly, e.g. using Voyage's fdisk? Maybe just
> use multiples of '8 tracks = 63*4K'?
>
>  * I have the rootfs on a ro partition, and write my data to a separate
> rw partition. Is it safe to assume that even if a powercycle corrupts my
> data partition, that the rootfs partition is still intact?
>
It should work just fine. I've known many people that has a factory default
values in ro partitions and restore it in case of failure of the rw
partition. I use rw for the whole disk most of the time and I mount the
logs directories in ram. I usually sync this directories every hour or so
to reduce the cf wear.

>
>  * Besides your recommendations to adjust the vm.dirty_* syscontrols
> when writing to CF [2] - What are your recommended file systems and
> mount options for rw data partitions to reduce the risk of data loss on
> a powercycle? E.g. i see people recommending to use ext3 or ext4 with
> 'journal=data' and/or 'sync' (at the expense of reduced write speed). Is
> that sound?
>
Journaling systems increase the number of writes to the CF. You should
consider If it's worth it. Remember that ext3 is basically ext2 +
journaling.

>
>  * Does Voyage Linux do a file system check when it mounts additional
> file systems if they have fs_passno=2 (6th field) in /etc/fstab? Does
> the fsck call include the "-y" option to force automatic fixing mode?
>
>  * Would it work if i symlinked /var/log/ into a rw data partition and
> removed it from 'SYNC_DIRS' in /etc/init.d/voyage-sync ?
>
You don't need to symlink it, just mount it there ;) And yes, it should work

Sorry but I can't help you with the others

>
>
> Any advise would be greatly appreciated!
>
>
> Cheers, /nils.
>
>
> [1] http://www.pcengines.ch/cfperf.htm
> [2] http://linux.voyage.hk/content/getting-started-v09x
>
>
> PS: Disclaimer & use case:
>
> I know that CF is not advised for permanent storage when powercycles can
> happen at any time. We use ALIX + Voyage linux to collect and cache, and
> send sensor data to central servers. The data we write to CF is not
> critical. However, for a number of reasons it would be very helpful if
> the data survived powercycles:
>
>  * Cached sensor data that hasn't been sent out yet (some of our ALIXes
> have unstable uplinks, e.g. 3G) would not get lost
>
>  * Stored states would allow the data collection to resume significantly
> quicker
>
>  * Logs up to just before a system fault (e.g. OOM) would be very
> helpful for debugging
>
> _______________________________________________
> Voyage-linux mailing list
> Voyage-linux at list.voyage.hk
> http://list.voyage.hk/mailman/listinfo/voyage-linux
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://list.voyage.hk/pipermail/voyage-linux/attachments/20130711/c709e1b4/attachment.html>