The original post: /r/linux by /u/the_bueg on 2024-09-18 13:50:32.

These files get updated/added/deleted fairly often on any machine.

Dropbox’s syncing tech works well for this, but Dropbox to go (for reasons). Some other things I’ve tried:

• I’m currently using Nextcloud. It has its pros and cons, and I’m actually going to use it to replace part of Dropbox. [Eg for getting photos off of phone and other larger file syncing.] But: A) it’s a little too complex for long-term sustainability, IMO. [Eg for “decadeS” of use like dropbox will soon be approaching], granted comparing paid cloud to self-hosted open-source; B) its not suitable for script syncing, as it currently doesn’t sync the execute bit. [There’s a github issue addressing it, but it was closed and may remain so.] I could hack together a little script daemon to keep the exec bit set - as long as that doesn’t trigger an infinite sync storm [which would be ironic] - but I’ve got so many kludges like that to fix bugged stuff, it makes for a fragile system, I want to avoid that if at all possible.
• I tried SyncThings a couple of years ago, but it was super fussy to get going on multiple machines - I don’t typically even mind “fussy”. (I used a server as a “master” to avoid endless syncing.) But the bigger problem is that I don’t want the insecurity of UPnP open, and any solution also need to work, say, through a VPN at the airport. (The advantage of Dropbox - and Nextcloud - is that it initiates the connection over HTTPS.)
• NordVPN and mesh networking. It is a cool and super-easy way to get a private WAN up and running. But Nord is not anywhere remotely near reliable enough on Linux to run on a 24/7/365 server. Man I wish it was. If it was, that could solve the Syncthings and UPnP/port-forwarding conundrum.
• Some other always-on, ultra-reliable WAN solution across all devices, plus Syncthings. I haven’t tried another commercial Nord competitor. After my years of experience with Nord across every one of my wide variety of Linux hosts - and it’s inability to not crash and bork its own service and socket to the point of eventually needing a reboot, I kind of wonder if any commercial VPN service could handle a heavy server load indefinitely, while also being inexpensive, user-friendly, run on any device, allow practically unlimited connections, and have a private WAN feature. Seems too much to expect.
• As mentioned in a comment below, use git. I wouldn’t want to use a third-party cloud provider like github, but could host my own long-term. But as I responded to in the comment, that would require an additional layer of a daemon to watch for changes on clients and automatically performing a pull/commit/push - and/or regular pulls to receive other client changes. Which there are at least existing scripts for. But if I were to do that, I might as well user rsync, which seems better suited to this particular task (where I’m not concerned with inter-file diffs or branches or multi-file merges), and is also better suited to handling large files.

Some quasi-requirements (most but not all of which Dropbox, Syncthings, and Nextcloud satisfy):

• Secure connections and encrypted transmission streams (obviously).
• UPnP not required, firewall port mapping not required, VPN WAN not required.
  • So in other words, along with the first requirement, probably leaves us pretty much with a listening server (probably HTTPS), and a bunch of initiating clients. But I’m more than happy to open up one or a few non-HTTPS inbound port[s] for a dedicated server - just not one or more for each client. In other words, p2p is probably, unfortunately, not in the cards.
• Keep lots of scripts and small files in sync, across up to dozens of instances, anywhere in the world, on any device, on any wifi, behind nearly any firewall. (Though one restricting funky ports is OK - I usually use a regular commercial VPN like Nord in that case. This stuff is all within the bounds of reason, some stuff is obviously just not going to work, including VPNs on occasion.)
• Any file can be edited, deleted, added by any node - even off-line - and eventually propagated to the rest. (Though I don’t actually need scripts and other executables to sync to mobile devices.)
• In the case of conflicts, either rename the losing [oldest] file like Dropbox does, or only keep the winner [newest] and log that fact somewhere.
• For relevant select file metadata that target filesystems can’t natively preserve (eg exec bit, extended attributes, etc.), or don’t store the same way or to the same accuracy: store in the server database (eg Dropbox/Nextcloud), or in each sync client database or other metadata store - and retransmit along with each file. For example, maintain:
  • Execute bit across instances, even if linux file is updated on Windows.
  • Extended attributes (eg NTFS ADRs or Xattrs in *nix file systems).
  • Original file encoding scheme (eg UTF-8 or ANSI) for text-based files, even if a file is edited on a system that doesn’t support it. (For non-text-based files, e.g. .docx, leave it to applications.)
  • Created (mbirth) and modified (mtime) timestamps. (Most sync clients already do this anyway to more reliably maintain their own granularity of collision detection, so the “maintain and transmit file metadata separately” framework is presumably already there most sync applications.)
• Interruption-tolerant incremental syncs of arbitrarily large files (eg a 100GB file), checksum-verified and cleaned-up upon completion.
• Extra bonus, iOS and Android photo syncing. Extra extra bonus, optional auto-deletion on device, upon auto-verification of successful upload. (i.e. PhotoSync app.)
• And if it’s not too much to ask, ideally open-source.
• Oh and because I’m not done with my entitled demands: Easy to set up! (Jk, I’m a former dev by profession and currently by hobby, who writes open-source code. And contributes in small ways to various small projects that probably hasn’t helped the world all that much. I know reasonably well how hard and thankless it is. I can ask as a wish-list, but I certainly don’t “expect”.)

I’m curious to know what other tools others use to address similar needs, and/or hacked-together solutions. Thanks.

Edits: Fixed missing section on syncthings, added bullet about git and rsync.