[Voyage-linux] aufs: Sync to flash with our reboot ?

[Robert Rawlins - Think Blue]

Hi Punky,

I'll be testing this stuff out today to see how it goes. So, we have to remountrw before we run the sync script, is that correct?

I have a few files which I need to keep in sync with a filesystem, as I said below at the moment I've been testing with a RW partition on the drive. Which approach would you advocate? The RW partiion? Or a scheduled sync script?

Thanks Punky, I appreciate the advice!

Rob

-----Original Message-----
From: Kim-man 'Punky' TSE [mailto:punkytse at punknix.com] 
Sent: 08 June 2009 02:28
To: Mike O'Connor
Cc: Robert Rawlins - Think Blue; voyage-linux at voyage.hk
Subject: Re: [Voyage-linux] aufs: Sync to flash with our reboot ?

Hi Mike and Robert,

Did you try the voyage-sync script I posted in another thread?

   
http://svn.voyage.hk/repos/voyage/trunk/voyage-util/overlay_fs/etc/init.d/voyage-sync 


This will uses tmpfs for /var/log and /var/tmp.  /etc/inid.d/voyage-sync 
sync will sync all files under /.sync/ (you need to manually do 
remountrw first).  Let me know you progress and share your thoughts.

Regards,
Punky

Mike O'Connor wrote:
> Hi Robert
>
> Yes I've been using the script which was put forward supplied by Guido.
> My issue with it not working was simply the fact that I was using a
> custom kernel which did not have the aufs patches.
>
> I'm not writing out the changes except when I think I need to.
>
> I'm of the option that any one who need permanate storage should use a
> SSD module and run a normal linux OS.
>
> For the parts of the file system which are changing a lot like /var/log
> I would just mount a TMPFS over the top and leave the rest of on the disk.
>
> Mike
>
>
> Robert Rawlins - Think Blue wrote:
>   
>> Hey Mike,
>>
>> Did you ever make much progress on your solution for this? We're in a very similar situation to this where we have data we need to persist on the disk so that it's available even after a power loss. I've been toying with creating second partitions on the disk, which appears to work O.K. however is still prone to corruption on occasion.
>>
>> How are you implementing this? Are you using a cron entry to call the resync script on a regular basis to ensure that any data changes are persisted to the disk? Have you found it to be reliable?
>>
>> I'm really looking to find a solution to this as data loss is proving a real pain in the neck for me :-(
>>
>> Cheers mate,
>>
>> Robert
>>
>> -----Original Message-----
>> From: voyage-linux-bounces+robert.rawlins=thinkbluemedia.co.uk at list.voyage.hk [mailto:voyage-linux-bounces+robert.rawlins=thinkbluemedia.co.uk at list.voyage.hk] On Behalf Of Mike O'Connor
>> Sent: 08 May 2009 07:03
>> To: Guido De Rosa
>> Cc: voyage-linux at voyage.hk
>> Subject: Re: [Voyage-linux] aufs: Sync to flash with our reboot ?
>>
>> Hi Guido
>>
>> I'm trying out this script, I seem to have a problem, the /var/log
>> directory is not being overlaided with aufs.
>>
>> voyage:/tmp/voyage-0.6.1/etc/rc1.d# cat /proc/mounts
>> rootfs / rootfs rw 0 0
>> none /sys sysfs rw,nosuid,nodev,noexec 0 0
>> none /proc proc rw,nosuid,nodev,noexec 0 0
>> udev /dev tmpfs rw,size=10240k,mode=755 0 0
>> /dev/disk/by-label/ROOT_FS / ext2 rw,noatime,errors=continue 0 0
>> tmpfs /lib/init/rw tmpfs rw,nosuid,mode=755 0 0
>> varrun /var/run tmpfs rw,nosuid,mode=755 0 0
>> varlock /var/lock tmpfs rw,nosuid,nodev,noexec 0 0
>> usbfs /proc/bus/usb usbfs rw,nosuid,nodev,noexec 0 0
>> tmpfs /dev/shm tmpfs rw,nosuid,nodev 0 0
>> devpts /dev/pts devpts rw,nosuid,noexec,gid=5,mode=620 0 0
>> tmpfs /tmp tmpfs rw,nosuid,nodev 0 0
>>
>> Which means the remount,ro during the boot process fails.
>>
>> I see during the boot process it doing the voyage-sync start, no errors
>> are printed to the screen.
>>
>> I think the issue is that syslogd has been restarted before the mounting
>> of aufs.
>>
>> One thing to note is that I have done a apt-get upgrade before starting
>> the proccess of getting the new voyage-sync
>>
>> Mike
>>
>>
>>
>> Guido De Rosa wrote:
>>   
>>     
>>> 2009/3/30 Kim-man 'Punky' TSE <punkytse at punknix.com>:
>>>
>>>   
>>>     
>>>       
>>>> I would try I could have some update on it later this week. �After I tested
>>>> Guido's update and some of my additions.
>>>>     
>>>>       
>>>>         
>>> Hi! Please always consider the last update:
>>>
>>> http://bitbucket.org/gderosa/net-appliances/src/tip/voyage/etc/init.d/voyage-sync
>>>
>>> Specifically, since rev9, some extra check has been added:
>>>
>>> 			# verify that $SYNC_DIR is actually mounted as a
>>> 			# $UNIONFS; otherwise, avoid dangerous over-writings
>>> 			# of $SYNCFS_ROOT/$SYNC_DIR
>>> 			if (mount | grep $SYNC_DIR | grep "type $UNIONFS" > /dev/null) ; then
>>> 				echo -n "  Sync'ing $SYNC_DIR to $SYNCFS_ROOT$SYNC_DIR ... "
>>> 				rsync $RSYNC_OPTS $SYNC_DIR $SYNCFS_ROOT/`dirname $SYNC_DIR`
>>> 				log_action_end_msg $?
>>> 			fi
>>>
>>> Think about the following scenario:you've been using AUFS/UNIONFS for
>>> a while and recently you have upgraded the kernel, but forgot to add
>>> aufs support; you also have added to VOYAGE_SYNC_DIRS some directories
>>> with important data; in this scenario, after the first reboot,
>>> mounting of $SYNC_DIR fails, and you get stuck with old $SYNC_DIR
>>> contained in your root partition. After another reboot, new data in
>>> $SYNCFS_ROOT are overwritten by older ones which is definitely not
>>> what you want. That's why an extra check is necessary.
>>>
>>> So, just for the first time, when you just added a new directory to
>>> VOYAGE_SYNC_DIR, you will have to manually copy it to  $SYNCFS_ROOT/ .
>>>
>>> (Maybe an helper function could be added to
>>> /usr/local/sbin/voyage-sync to do this syncing-without-checking
>>> automatically, but definitely not in /etc/init.d )
>>>
>>>   
>>>     
>>>       
>> _______________________________________________
>> Voyage-linux mailing list
>> Voyage-linux at list.voyage.hk
>> http://list.voyage.hk/mailman/listinfo/voyage-linux
>>
>>   
>>     
>
>
> _______________________________________________
> Voyage-linux mailing list
> Voyage-linux at list.voyage.hk
> http://list.voyage.hk/mailman/listinfo/voyage-linux
>   


-- 
Regards,
Punky

Voyage Solutions (http://solution.voyage.hk)
* Embedded Solutions and Systems
  - Mesh Networking, Captive Portal, IP Surveillance, VoIP/PBX
  - Network Engineering, Development Platform and Consultation