ObjectiveFS

Transparently Rotating Filesystems to Comply with Retention Schedule

This article provides an example of how to create a transparently rotating filesystem with a two-week retention schedule. It utilizes the ObjectiveFS filesystem pool feature together with an overlay filesystem.

In this example, we use two filesystems in a filesystem pool to store our weekly filesystem data: one for the current week and one for the previous week, since our retention schedule is two weeks. We then mount these two filesystems on an overlay filesystem: one as the primary filesystem for reading and writing this week’s data and the other as a read-only filesystem for last week’s data. At the end of the week, we recycle the older weekly filesystem that we don’t need anymore per our retention schedule. We then mount the freshly recycled filesystem for reading and writing this week’s data and move last week’s filesystem into read-only mode on our overlay filesystem.

Install AUFS.

 # apt-get update
 # apt-get install linux-image-extra-virtual
 # apt-get install linux-image-extra-4.4.0-53-generic

Create two filesystems in a filesystem pool to be used for rotation.

 # mount.objectivefs create <pool name>/weekly-a
 # mount.objectivefs create <pool name>/weekly-b

On the target machine, create the mount directories for the overlay and rotating filesystems.
```
 # mkdir -p /ofs/weekly-{a,b}
 # mkdir -p /ofs/overlay
```
Initial setup: mount the rotating filesystems and overlay filesystem.
```
 #  mount.objectivefs <pool name>/weekly-a /ofs/weekly-a
 #  mount.objectivefs <pool name>/weekly-b /ofs/weekly-b
 #  mount -t aufs -o br=/ofs/weekly-b:/ofs/weekly-a none /ofs/overlay
 #  ln -s /ofs/overlay /ofs-target
```
Your applications can read and write to /ofs-target/*as usual.
All data written to /ofs-target/*will be stored on the weekly-bfilesystem.
Reads from /ofs-target/*will contain data from both weekly-aand weekly-b.

At the end of each week, use the following weekly rotation schedule:
Note: Make sure that the filesystem is unmounted on all machines before running destroy.

a. Week N

 #  ln -s /ofs/weekly-b /ofs.new && mv -Tf /ofs.new /ofs-target
 #  umount /ofs/overlay
 #  umount /ofs/weekly-a
 // Recycle the weekly-a filesystem
 // Verify weekly-a is unmounted on all machines before running destroy
 #  mount.objectivefs destroy <pool name>/weekly-a
 #  mount.objectivefs create <pool name>/weekly-a
 #  mount.objectivefs <pool name>/weekly-a /ofs/weekly-a
 // Create an overlay filesystem with weekly-a on top
 #  mount -t aufs -o br=/ofs/weekly-a:/ofs/weekly-b none /ofs/overlay
 #  ln -s /ofs/overlay /ofs.new && mv -Tf /ofs.new /ofs-target

Your applications can continue to read and write to /ofs-target/*as usual.
All data written to /ofs-target/*will be stored on the weekly-afilesystem.
Reads from /ofs-target/*will contain data from both weekly-aand weekly-b.

b. Week N+1

 #  ln -s /ofs/weekly-a /ofs.new && mv -Tf /ofs.new /ofs-target
 #  umount /ofs/overlay
 #  umount /ofs/weekly-b
 // Recycle the weekly-b filesystem
 // Verify weekly-b is unmounted on all machines before running destroy
 #  mount.objectivefs destroy <pool name>/weekly-b
 #  mount.objectivefs create <pool name>/weekly-b
 #  mount.objectivefs <pool name>/weekly-b /ofs/weekly-b
 // Create an overlay filesystem with weekly-b on top
 #  mount -t aufs -o br=/ofs/weekly-b:/ofs/weekly-a none /ofs/overlay
 #  ln -s /ofs/overlay /ofs.new && mv -Tf /ofs.new /ofs-target

Your applications can continue to read and write to /ofs-target/*as usual.
All data written to /ofs-target/*will be stored on the weekly-bfilesystem.
Reads from /ofs-target/*will contain data from both weekly-aand weekly-b.

Reference

Filesystem Pool

by ObjectiveFS staff, January 25, 2017
ObjectiveFS is a shared file system for OS X and Linux that automatically scales and gives you scalable cloud storage. If you have questions or article idea suggestions, please email us at お問い合わせはこちら