[Voyage-linux] Best Practices for writing logs & data to CF

[John Coltrane]

Hello Nils,

On 07/11/2013 02:56 AM, Nils Toedtmann wrote:
> Dear Voyage community
>
>
> First: Voyage Linux is great! It makes building embedded systems
> significantly easier - thanks to everyone who contributed!
>
>
> I have a number of questions about writing data to CF on ALIX (before
> you rant about even wanting to do this - see my use case and motivation
> in the PS below).
>
> I apologize in advance if i haven't found existing answers in the Voyage
> documentation. I did do some research, but admit that being only a
> sysadmin, i didn't dive into the depths of Voyage's development docs.
>
>
> Here are my questions:
>
>   * Most documentation says that CF wears down quickly when many write
> operations happen. But those docs are 5-10 years old. Do modern
> 'industrial grade' CF cards still suffer from this problem? I wonder
> because nowadays it is pretty common to use flash based SSDs in
> write-heavy systems.
Industrial grade means SLC in my opinion. As far as I remember I have 
seen data sheets with a factor ~10 higher write cycles as MLC. But YMMV


>
>   * PC Engine recommend [1] to have partitions aligned to flash's 2K or
> 4K page size. But most partitioning software assumes 63 sectors (512
> bytes each) per track. How would i partition a CF card such that the
> partitions are aligned properly, e.g. using Voyage's fdisk? Maybe just
> use multiples of '8 tracks = 63*4K'?
gparted has features which allow aligning partitions to boundaries 
different than the default ones. Maybe you should have a look at it


>
>   * I have the rootfs on a ro partition, and write my data to a separate
> rw partition. Is it safe to assume that even if a powercycle corrupts my
> data partition, that the rootfs partition is still intact?
hopefully


>
>   * Besides your recommendations to adjust the vm.dirty_* syscontrols
> when writing to CF [2] - What are your recommended file systems and
> mount options for rw data partitions to reduce the risk of data loss on
> a powercycle? E.g. i see people recommending to use ext3 or ext4 with
> 'journal=data' and/or 'sync' (at the expense of reduced write speed). Is
> that sound?
I have seen recommendations to use EXT4 without journaling, to get the 
advantages of EXT4 without the disadvantages of EXT3 and older. So far I 
have no bad experience. You can format a fresh partition by doing a

mke2fs -t ext4 -O ^has_journal /dev/sdaX

or remove the journal of an existing FS with

tune2fs -O ^has_journal /dev/sdaX


HtH

>
>   * Does Voyage Linux do a file system check when it mounts additional
> file systems if they have fs_passno=2 (6th field) in /etc/fstab? Does
> the fsck call include the "-y" option to force automatic fixing mode?
>
>   * Would it work if i symlinked /var/log/ into a rw data partition and
> removed it from 'SYNC_DIRS' in /etc/init.d/voyage-sync ?
>
>
> Any advise would be greatly appreciated!
>
>
> Cheers, /nils.
>
>
> [1] http://www.pcengines.ch/cfperf.htm
> [2] http://linux.voyage.hk/content/getting-started-v09x
>
>
> PS: Disclaimer&  use case:
>
> I know that CF is not advised for permanent storage when powercycles can
> happen at any time. We use ALIX + Voyage linux to collect and cache, and
> send sensor data to central servers. The data we write to CF is not
> critical. However, for a number of reasons it would be very helpful if
> the data survived powercycles:
>
>   * Cached sensor data that hasn't been sent out yet (some of our ALIXes
> have unstable uplinks, e.g. 3G) would not get lost
>
>   * Stored states would allow the data collection to resume significantly
> quicker
>
>   * Logs up to just before a system fault (e.g. OOM) would be very
> helpful for debugging
>
> _______________________________________________
> Voyage-linux mailing list
> Voyage-linux at list.voyage.hk
> http://list.voyage.hk/mailman/listinfo/voyage-linux