Saturday, December 3, 2016

linux - Real-Time Multi-Client Synchronization with Conflict Management


I need to setup a synchronization between my Computers with a NAS in the local network. The setup is somewhat complicated, though, so I am assuming there is not an off-the-shelf product that does what I need.


Details:


Devices I use:



  • Desktop PC, dual-boot Windows 10 / Arch Linux

  • Laptop (Arch Linux)

  • NAS (self-build, FreeNAS running on it)


I've used rsync so far to backup to the NAS (one-way, my dual-boot setup uses the same storage for both OS).


The problem is, I would like to get the Laptop in the sync.
Therefore I need to do some kind of two-way sync.


Problems that arise here:



  1. I often save files only seconds before the shutdown, so I need to monitor file changes instead of running a sync tool periodically



    • I cannot sync my laptop outside of the local network (and in the local network, I use the desktop PC most of the times); the local (updated) version could be quite out of date and give some conflicts.


  2. I do want to use the local network only (probably the existing network share of the NAS), not an ownCloud or something similar, because we have quite limited bandwidth where I live and I have some rather huge files to keep synced.



My ideas


I found this question: How do I setup rsync and any other scripts to mimic a minimal viable replica of DropBox?, and looked into a solution using lsyncd (Live Syncing Daemon), which synchronizes local directories with remote targets. This could solve problem #1, but I still need to be prepared for conflicts. Furthermore, I could not find any docs about lsyncd on Windows, so I assume it won't work there.


I also looked at the potential to incorporate Unison into a solution; it seems to do two-way syncs. Maybe I could use it together with lsyncd.
I'm not sure about the conflict handling, though. I found a hint that with -batch, conflicts are just skipped. But how do I handle conflicted files in the long term (I want to sync them, after all)?


So my question is, how can I accomplish this?


Answer



Depending on your NAS - I'm pretty fond of BitTorrent sync for this. It runs on most systems (including raspberry pis and mips systems) with support from quite a few consumer NAS systems (like my seagate). You just pick a folder to share, generate a key, and use the key to set up the share on each system you want it on. Its smart enough to detect you're on a LAN, works outside of it. The free version works well enough for my needs - I mainly use it to sync downloads from a system I keep on all the time and backups of many small files that I add to infrequently.


It'll do what you need - one way or two way sync, using the lan if needed. It deals with conflict by keeping a copy of deleted files, (which I turn off for my use cases) and generally runs well enough that I hardly notice it.


No comments:

Post a Comment

hard drive - Leaving bad sectors in unformatted partition?

Laptop was acting really weird, and copy and seek times were really slow, so I decided to scan the hard drive surface. I have a couple hundr...