[Voyage-linux] Best Practices for writing logs & data to CF

[Nils Toedtmann]

You are luckier than me.

I use ALIX with CF for a number of years now (OpenWRT first, now
Voyage), and i have experienced a fair amount of forced fscks and
filesystems falling back into read-only mode because of errors.

Hence i started my quest to find the most stable write-to-CF settings
there are.

/n


On 11.07.2013 13:11, Sylvain Gomez wrote:
> Hi,
> 
>  
> 
> I use Voyage-Linux since 3 years as WiFi HotSpot with Coova Chilli,
> Apache2, PHP5, MySQL and FreeRadius.
> 
> As I need to log every users actions, I’ve completely disabled the
> read-only system.
> 
> I use regular CF (MLC ship, not SLC) and I’m very happy with it.
> 
> Some of my boxes are running since 3 years and I never encounter any
> problems on any CF…
> 
> I didn’t tweak any parameter, I just use Voyage-Linux stock config and
> it works great for me.
> 
>  
> 
>  
> 
> Regards,
> 
> Sylvain
> 
>  
> 
> *De :*Voyage-linux
> [mailto:voyage-linux-bounces+sylvaingomez=free.fr at list.voyage.hk] *De la
> part de* Jordi Soucheiron
> *Envoyé :* jeudi 11 juillet 2013 10:34
> *À :* Nils Toedtmann
> *Cc :* voyage-linux
> *Objet :* Re: [Voyage-linux] Best Practices for writing logs & data to CF
> 
>  
> 
>  
> 
> 
> *Jordi Soucheiron*
> Software Engineer
> jsoucheiron at dexmatech.com <mailto:jsoucheiron at dexmatech.com>
> 
> *DEXMA*
> 
> Marie Curie, 8-14, Barcelona, ES-08042
> 
> 5 Penn Plaza, Suite 2300, New York, NY 10001
> t: (+34) 93 181 01 96
> 
> www.dexmatech.com <http://www.dexmatech.com>
> 
> Follow us on Twitter <http://www.twitter.com/dexma> and Linkedin
> <http://www.linkedin.com/company/247838>
> Síguenos en Twitter <http://www.twitter.com/dexma_ES> y Linkedin
> <http://www.linkedin.com/company/dexma-tech>
> 
>  
> 
> 2013/7/11 Nils Toedtmann <voyage-linux-mail at nils.toedtmann.net
> <mailto:voyage-linux-mail at nils.toedtmann.net>>
> 
> Dear Voyage community
> 
> 
> First: Voyage Linux is great! It makes building embedded systems
> significantly easier - thanks to everyone who contributed!
> 
> 
> I have a number of questions about writing data to CF on ALIX (before
> you rant about even wanting to do this - see my use case and motivation
> in the PS below).
> 
> I apologize in advance if i haven't found existing answers in the Voyage
> documentation. I did do some research, but admit that being only a
> sysadmin, i didn't dive into the depths of Voyage's development docs.
> 
> 
> Here are my questions:
> 
>  * Most documentation says that CF wears down quickly when many write
> operations happen. But those docs are 5-10 years old. Do modern
> 'industrial grade' CF cards still suffer from this problem? I wonder
> because nowadays it is pretty common to use flash based SSDs in
> write-heavy systems.
> 
> Heavy writing in SSD is not usually a good idea. The disks will
> inevitably wear down eventually. It's only a matter of time, but it may
> be worth it depending on what you need to do. 
> 
> 
>      * PC Engine recommend [1] to have partitions aligned to flash's 2K or
>     4K page size. But most partitioning software assumes 63 sectors (512
>     bytes each) per track. How would i partition a CF card such that the
>     partitions are aligned properly, e.g. using Voyage's fdisk? Maybe just
>     use multiples of '8 tracks = 63*4K'?
> 
>      * I have the rootfs on a ro partition, and write my data to a separate
>     rw partition. Is it safe to assume that even if a powercycle corrupts my
>     data partition, that the rootfs partition is still intact?
> 
> It should work just fine. I've known many people that has a factory
> default values in ro partitions and restore it in case of failure of the
> rw partition. I use rw for the whole disk most of the time and I mount
> the logs directories in ram. I usually sync this directories every hour
> or so to reduce the cf wear. 
> 
> 
>      * Besides your recommendations to adjust the vm.dirty_* syscontrols
>     when writing to CF [2] - What are your recommended file systems and
>     mount options for rw data partitions to reduce the risk of data loss on
>     a powercycle? E.g. i see people recommending to use ext3 or ext4 with
>     'journal=data' and/or 'sync' (at the expense of reduced write speed). Is
>     that sound?
> 
> Journaling systems increase the number of writes to the CF. You should
> consider If it's worth it. Remember that ext3 is basically ext2 +
> journaling. 
> 
> 
>      * Does Voyage Linux do a file system check when it mounts additional
>     file systems if they have fs_passno=2 (6th field) in /etc/fstab? Does
>     the fsck call include the "-y" option to force automatic fixing mode?
> 
>      * Would it work if i symlinked /var/log/ into a rw data partition and
>     removed it from 'SYNC_DIRS' in /etc/init.d/voyage-sync ?
> 
> You don't need to symlink it, just mount it there ;) And yes, it should work
> 
>  
> 
> Sorry but I can't help you with the others
> 
> 
> 
>     Any advise would be greatly appreciated!
> 
> 
>     Cheers, /nils.
> 
> 
>     [1] http://www.pcengines.ch/cfperf.htm
>     [2] http://linux.voyage.hk/content/getting-started-v09x
> 
> 
>     PS: Disclaimer & use case:
> 
>     I know that CF is not advised for permanent storage when powercycles can
>     happen at any time. We use ALIX + Voyage linux to collect and cache, and
>     send sensor data to central servers. The data we write to CF is not
>     critical. However, for a number of reasons it would be very helpful if
>     the data survived powercycles:
> 
>      * Cached sensor data that hasn't been sent out yet (some of our ALIXes
>     have unstable uplinks, e.g. 3G) would not get lost
> 
>      * Stored states would allow the data collection to resume significantly
>     quicker
> 
>      * Logs up to just before a system fault (e.g. OOM) would be very
>     helpful for debugging
> 
>     _______________________________________________
>     Voyage-linux mailing list
>     Voyage-linux at list.voyage.hk <mailto:Voyage-linux at list.voyage.hk>
>     http://list.voyage.hk/mailman/listinfo/voyage-linux
> 
>  
>