You might not even like rsync. Yeah it’s old. Yeah it’s slow. But if you’re working with Linux you’re going to need to know it.
In this video I walk through my favorite everyday flags for rsync.
Support the channel:
https://patreon.com/VeronicaExplains
https://ko-fi.com/VeronicaExplains
https://thestopbits.bandcamp.com/
Here’s a companion blog post, where I cover a bit more detail: https://vkc.sh/everyday-rsync
Also, @BreadOnPenguins made an awesome rsync video and you should check it out: https://www.youtube.com/watch?v=eifQI5uD6VQ
Lastly, I left out all of the ssh setup stuff because I made a video about that and the blog post goes into a smidge more detail. If you want to see a video covering the basics of using SSH, I made one a few years ago and it’s still pretty good: https://www.youtube.com/watch?v=3FKsdbjzBcc
Chapters:
1:18 Invoking rsync
4:05 The --delete flag for rsync
5:30 Compression flag: -z
6:02 Using tmux and rsync together
6:30 but Veronica… why not use (insert shiny object here)
Why videos? I feel like an old man yelling at clouds every time something that sounds interesting is presented in a fucking video. Videos are so damn awful. They take time, I need audio and I can’t copy&paste. Why have they become the default for things that should’ve been a blog post?
They linked blog post with the video: https://vkc.sh/everyday-rsync/
Thank you for putting into words what ive subconsciously been thinking for years. Every search result prioritizes videos at the top and I’m still annoyed every time. Or even worst I have to hunt through a 10 minute video for the 30 seconds of info I needed. Stoohhhhpppp internet of new! Make it good again!
Ad money.
Blogs can have ads.
Hear hear. Knowledge should be communicated in an easily shareable way that can also be archived as easily, in contrast to a video requiring hundreds of MB:s.
Especially for a command line tool
man rsync
Rsnapshot. It uses rsync, but provides snapshot management and multiple backup versioning.
Yes, but a few hours writing my own scripts will save me from several minutes of reading its documentation…
It took me like 10 min to setup rsnapshot (installing, and writing systemd unit /timer files) on my servers.
I’m sure I could script something similar in under 10 (hours).
Man ilove being autostic
Yah, I really like this approach. Same reason I set up Timeshift and Mint Backup on all the user machines in my house. For others rsync + cron is aces.
I’ll never not upvote Veronica Explains. Excellent creator and excellent info on everything I’ve seen.
I never thought of it as slow. More like very reliable. I dont need my data to move fast, I need it to be copied with 100% reliability.
And not waste time copying duplicate data. And for the typical home user, it’s probably mo slower than other options.
It’s slow?!?
That part threw me off. Last time i used it, I did incremental backups of a 500 gig disk once a week or so, and it took 20 seconds max.
Yes but imagine… 18 seconds.
Compared to something multi threaded, yes. But there are obviously a number of bottlenecks that might diminish the gains of a multi threaded program.
With xargs everything is multithreaded.
Yeah it’s slow
What’s slow about async? If you have a reasonably fast CPU and are merely syncing differences, it’s pretty quick.
It’s single thread, one file at a time.
That would only matter if it’s lots of small files, right? And after the initial sync, you’d have very few files, no?
Rsync is designed for incremental syncs, which is exactly what you want in a backup solution. If your multithreaded alternative doesn’t do a diff, rsync will win on larger data sets that don’t have rapid changes.
For a home setup that seems fine. But I can understand why you wouldn’t want this for a whole enterprise.
Surely restic or borg would be better for backups?
Rsync can send files and not delete stuff, but there’s no versioning or retention settings.
If you add --delete-before, it absolutely can delete stuff.
Yeah but then it’s not really a good backup!
For versioning/retention, just use snapshots in whatever filesystem you’re using (you are using a proper filesystem like ZFS or BTRFS, right?).
How does that get sent over rsync though? Wouldn’t you need snapshots on the remote destination server?
Why not just use a backup utility instead?
Yes, async copies files to the remote server, the remote server takes regular snapshots.
Why not just use a backup utility instead?
What is that utility providing that snapshots + rsync doesn’t. If rsync + snapshots is sufficient, why overcomplicate it with a backup utility?
The main things that come to mind are you have to test/monitor 2 seperate actions instead of 1, and restores of single files could be more difficult since you need to login to the backup server, restore the file from a snapshot, then also copy that file back to your PC.
My point is, how often do you actually need to restore from backup? If it’s frequent, consider a dedicated tool for whatever that thing is. If it’s infrequent, it’ll probably easier to just learn how to do it every five years or whatever.
If you like borg/restic/etc, by all means, use it.
My point is that most people probably don’t need it. Snapshots are something you set up once, and you should probably use them even if you’re using something like borg for any files that aren’t covered (e.g. config files on the server). Rsync is also something you set up once, and checking it is the same as any other service.
It’s not bad if you don’t need historical backups. I kinda think I do, so I use https://github.com/rustic-rs/rustic becase rust
Restic (https://github.com/restic/restic) is probably a better choice if you’re not a rust-freak like me.
Rustic scares me. I will 100% forget what tool I used to backup after 5 years and be unable to recover my files.
I use rsync + ZFS for backups which includes historical backups
Yup, just configure a snapshot policy and you can recover deleted and modified files going back as long as you choose. And it is probably more space efficient than both/restic too.
I was planning to use rsync to ship several TB of stuff from my old NAS to my new one soon. Since we’re already talking about rsync, I guess I may as well ask if this is right way to go?
I couldn’t tell you if it’s the right way but I used it on my Rpi4 to sync 4tb of stuff from my Plex drive to a backup and set a script up to have it check/mirror daily. Took a day and a half to copy and now it syncs in minutes tops when there’s new data
It depends
rsyncis fine, but to clarify a little further…If you think you’ll stop the transfer and want it to resume (and some data might have changed), then yep,
rsyncis best.But, if you’re just doing a 1-off bulk transfer in a single run, then you could use other tools like
xcopy/scpor - if you’ve mounted the remote NAS at a local mount point - just plain oldcpThe reason for that is that
rsynchas to work out what’s at the other end for each file, so it’s doing some back & forwards communications each time which as someone else pointed out can load the CPU and reduce throughput.(From memory, I think Raspberry Pi don’t handle large transfers over
scpwell… I seem to recall a buffer gets saturated and the throughput drops off after a minute or so)Also, on a local network, there’s probably no point in using encryption or compression options - esp. for photos / videos / music… you’re just loading the CPU again to work out that it can’t compress any further.
It’s just a one-off transfer, I’m not planning to stop the transfer, and it’s my media library, so nothing should change, but I figured something resumable is a good idea for a transfer that’s going to take 12+ hours, in case there’s an unplanned stop.
One thing I forgot to mention:
rsynchas an option to preserve file timestamps, so if that’s important for your files, then thst might also be useful… without checking, the other commands probably have that feature, but I don’t recall at the moment.rsync -Prvt <source> <destination>might be something to try, leave for a minute, stop and retry … that’ll prove it’s all working.Oh… and make sure you get the source and destination paths correct with a trailing
/(or not), otherwise you’ll get all your files copied to an extra subfolder (or not)
yes, it’s the right way to go.
rsync over ssh is the best, and works as long as rsync is installed on both systems.
On low end CPUs you can max out the CPU before maxing out network—if you want to get fancy, you can use rsync over an unencrypted remote shell like
rsh, but I would only do this if the computers were directly connected to each other by one Ethernet cable.
I still prefer tar for quick and dirty same box copies.
tar cf - * | (cd /target; tar xfp -)Why not just
cp?
I use syncthing.
Is rsync better?
Syncthing works pretty well for me and my stable of Ubuntu, pi, Mac, and Windows
Different tools for different use cases IMO.
But neither do backups.
I dunno.
I am using it to keep a real time copy of documents on an offsite server.
Feels like a backup to me.
What happens if you accidentally overwrite something important in a document and save it though? If there’s no incremental versioning you can’t recover from that.
That is a good point.
In my case, I was trying to address the shortcomings of Apple Time Machine. I use a Mac mini as the server I work from on all my machines. Time Machine does the version Managment for me.
I just use Sync Thing through a VPN to keep an offsite backup of content files (not a complete OS restore) and to keep a copy of critical files on my laptop in case I am away from my home network and need to see a file.
I still need to implement a regular air gapped backup instead of the ad-hoc that I have now.
I’m not super familiar with Syncthing, but judging by the name I’d say Syncthing is not at all meant for backups.
Syncthing is technically to synchronize data across different devices in real time (which I do with my phone), but I also use it to transfer data weekly via wi-fi to my old 2013 laptop with a 500GB HDD and Linux Mint (I only boot it to transfer data, and even then I pause the transfers to this device when its done transferring stuff) so I can have larger data backups that wouldn’t fit in my phone, since LocalSend is unreliable for large amounts of data while Synchting can resume the transfer if anything goes wrong. On top of that Syncthing also works in Windows and Android out of the box.
its for a different purpose. I wouldn’t use syncthing the way I use rsync
Veeam for image/block based backups of Windows, Linux and VMs.
syncthing for syncing smaller files across devices.Thank you very much.
Use borg/borgmatic for your backups. Use rsync to send your differentials to your secondary & offsite backup storage.
I used to use rsnapshot, which is a thin wrapper around rsync to make it incremental, but moved to restic and never looked back. Much easier and encrypted by default.
I think the there are better alternatives for backup like kopia and restic. Even seafile. Want protection against ransomware, storage compression, encryption, versioning, sync upon write and block deduplication.
comparing seafile to rsync reminds me the old “Space Pen” folk tale.
This exactly. I’d use rsync to sync a directory to a location to then be backed up by kopia, but I wouldn’t use rsync exclusively for backups.
