Rsync: File Copying & Syncing
Rsync (Remote Sync) is the gold standard for moving data in Linux/Unix. Unlike scp, it is “delta-aware,” meaning it only sends the parts of files that have changed, saving massive amounts of time and bandwidth.
The “Trailing Slash” Rule
This is the most common point of confusion with rsync.
-
source/(with slash): Copies the contents of the folder. -
source(no slash): Copies the folder itself and its contents.
Small Files vs. Large Files
The way rsync handles data should change depending on what you are moving.
| File Type / Context | Strategy | Base Recommended Flags | Useful Modifiers & Tweaks |
|---|---|---|---|
| Small Files (e.g., source code, docs) | Focus on compression and metadata preservation. | -avz |
Add -n or --dry-run first to check what will copy without moving data. |
| Large Files (e.g., Databases, images) | Focus on transfer speed, updating files directly, and tracking progress. | --inplace--progress |
Use --info=progress2 to output a single, continuously updating summary line for the entire transfer instead of a flood of individual file logs. |
| Deep Directories (Massive file counts) | Turn off compression to save local CPU cycles. | -av |
Pairing with --info=progress2 keeps your terminal clean when processing millions of files. |
| Shared / Login Nodes (Any cluster staging) | Throttle your transfer speed to protect cluster filesystem storage stability. | -av --bwlimit=20M |
Mandatory flag. Unthrottled runs on shared nodes are subject to termination. |
Note on
-z(Compression): While-zis great for slow networks, it can actually slow down transfers on high-speed local networks (like a LAN) because the CPU takes longer to compress the data than the wire takes to move it.
Key Options & Advanced Tweaks
1. The Power of -a (Archive Mode)
The -a flag is a “collection” of flags (-rlptgoD). It preserves permissions, symlinks, and timestamps.
-
When to NOT use
-a: If you are moving files to a filesystem that doesn’t support Linux permissions (like an exFAT/FAT32 USB drive),-awill throw constant errors. Use-rtvinstead to sync data and times without trying to set Linux ownership.
2. The --inplace Flag
Normally, rsync creates a hidden temporary file and moves it into place once finished.
-
Why use
--inplace? It updates the destination file directly. This is vital for Large Files where you don’t have enough disk space to hold two copies of the file at once. -
Warning: If the transfer fails halfway, the destination file will be corrupted until you run rsync again to finish it.
Important: Shared Storage Etiquette & Bandwidth Throttling
Because our cluster environments share a central high-performance network storage backend (mind-nas), heavy, unthrottled file movements can easily saturate the storage pipe’s metadata performance. When this happens, it causes kernel deadlocks that freeze terminal sessions and login capabilities for all cluster users.
To maintain cluster stability, you must throttle your transfers when moving data on shared infrastructure.
The --bwlimit Flag
rsync has a built-in mechanism to cap its maximum transfer speed. Adding this flag leaves plenty of network and disk performance left over for your peers.
-
Recommended Cap:
20M(20 Megabytes per second). This provides a fast transfer while completely protecting the storage backend from lagging.
Safe Transfer Example:
Bash
# Throttling your transfer to 20MB/s (Mandatory on shared login nodes)
rsync -a --info=progress2 --bwlimit=20M /home/dpane/mysrcdata /user_data/dpane/my-destination-data/
Best Practices Checklist:
-
Never run unthrottled transfers: Large datasets or folders with tens of thousands of sub-files should always be restricted using
--bwlimit. -
Use Compute Nodes for I/O: For data migrations, multi-terabyte datasets, or deep directory
findloops, spin up an interactive session via Slurm (srun -p cpu --cpus-per-task=2 --mem=4G --pty bash) and run your script inside the allocation rather than on the shared login gateway.
Examples
Sync Local to Remote
rsync -avz full_path_to_source -e "ssh -l username" server_address:full_path_to_destination
Sync Remote to Local
rsync -avz -e "ssh -l username" server_address:full_path_to_source full_path_to_destination
Specific Example: Raptor to MacOS (Copying the Folder)
This command copies the folder “testfolder” and its contents to your home directory.
rsync -avz -e "ssh -l dpane" raptor.ni.cmu.edu:/home/dpane/testfolder /Volumes/HD/dpane/
Specific Example: Raptor to MacOS (Copying Contents ONLY)
By adding the trailing / on the source, you copy only the contents inside “testfolder” into the destination.
rsync -avz -e "ssh -l dpane" raptor.ni.cmu.edu:/home/dpane/testfolder/ /Volumes/HD/dpane/
FAQ
Q: Why should I use rsync instead of scp?
A: When you are transferring a large amount of data, if the transfer is disconnected, rsync will pick it up where it left off. scp doesn’t do this. Use scp for one or two small files; use rsync for everything else, especially multi-GB data.
Q: Does rsync verify files?
A: Yes. Rsync performs checksums to verify data integrity. If a file exists, it may skip it if the size and time match, but it always verifies the data it actually transfers.
Q: Can I see what will happen before I run it?
A: Yes! Add -n (or --dry-run) to your command. It will list exactly which files would be moved without actually making any changes.
Additional Information:
There is a wealth of documentation online. This guide provides 10 practical examples.
-
Note: I suggest avoiding example #8 (“Automatically Delete source Files”) unless you are absolutely certain of your workflow.