{"id":361,"date":"2021-05-18T19:21:28","date_gmt":"2021-05-18T19:21:28","guid":{"rendered":"https:\/\/ni.cmu.edu\/computing\/?post_type=ht_kb&#038;p=361"},"modified":"2026-06-29T19:15:14","modified_gmt":"2026-06-29T19:15:14","slug":"copying-and-syncing-files","status":"publish","type":"ht_kb","link":"https:\/\/ni.cmu.edu\/computing\/knowledge-base\/copying-and-syncing-files\/","title":{"rendered":"Copying and Syncing Files"},"content":{"rendered":"<div id=\"model-response-message-contentr_318a4855f2c8986a\" class=\"markdown markdown-main-panel tutor-markdown-rendering enable-updated-hr-color\" dir=\"ltr\" aria-live=\"polite\" aria-busy=\"false\">\n<h2 data-path-to-node=\"2\">Rsync: File Copying &amp; Syncing<\/h2>\n<p data-path-to-node=\"3\"><b data-path-to-node=\"3\" data-index-in-node=\"0\">Rsync<\/b> (Remote Sync) is the gold standard for moving data in Linux\/Unix. Unlike <code data-path-to-node=\"3\" data-index-in-node=\"79\">scp<\/code>, it is &#8220;delta-aware,&#8221; meaning it only sends the parts of files that have changed, saving massive amounts of time and bandwidth.<\/p>\n<h3 data-path-to-node=\"4\">The &#8220;Trailing Slash&#8221; Rule<\/h3>\n<p data-path-to-node=\"5\">This is the most common point of confusion with rsync.<\/p>\n<ul data-path-to-node=\"6\">\n<li>\n<p data-path-to-node=\"6,0,0\"><b data-path-to-node=\"6,0,0\" data-index-in-node=\"0\"><code data-path-to-node=\"6,0,0\" data-index-in-node=\"0\">source\/<\/code> (with slash):<\/b> Copies the <b data-path-to-node=\"6,0,0\" data-index-in-node=\"33\">contents<\/b> of the folder.<\/p>\n<\/li>\n<li>\n<p data-path-to-node=\"6,1,0\"><b data-path-to-node=\"6,1,0\" data-index-in-node=\"0\"><code data-path-to-node=\"6,1,0\" data-index-in-node=\"0\">source<\/code> (no slash):<\/b> Copies the <b data-path-to-node=\"6,1,0\" data-index-in-node=\"30\">folder itself<\/b> and its contents.<\/p>\n<\/li>\n<\/ul>\n<hr data-path-to-node=\"7\" \/>\n<h3 data-path-to-node=\"8\">Small Files vs. Large Files<\/h3>\n<p data-path-to-node=\"9\">The way rsync handles data should change depending on what you are moving.<\/p>\n<table style=\"width: 100%; max-width: 1200px; border-collapse: collapse; margin: 20px 0; font-family: sans-serif; font-size: 14px;\" data-path-to-node=\"10\">\n<thead style=\"background-color: #f4f4f4; border-bottom: 2px solid #ccc;\">\n<tr>\n<th style=\"padding: 12px; text-align: left; font-weight: bold; width: 22%;\">File Type \/ Context<\/th>\n<th style=\"padding: 12px; text-align: left; font-weight: bold; width: 30%;\">Strategy<\/th>\n<th style=\"padding: 12px; text-align: left; font-weight: bold; width: 23%;\">Base Recommended Flags<\/th>\n<th style=\"padding: 12px; text-align: left; font-weight: bold; width: 25%;\">Useful Modifiers &amp; Tweaks<\/th>\n<\/tr>\n<\/thead>\n<tbody style=\"line-height: 1.5;\">\n<tr style=\"border-bottom: 1px solid #eee;\">\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,1,0,0\"><b data-path-to-node=\"10,1,0,0\" data-index-in-node=\"0\">Small Files<\/b> (e.g., source code, docs)<\/span><\/td>\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,1,1,0\">Focus on compression and metadata preservation.<\/span><\/td>\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,1,2,0\"><code style=\"background: #fdfdfd; padding: 3px 6px; border: 1px solid #ddd; border-radius: 4px;\" data-path-to-node=\"10,1,2,0\" data-index-in-node=\"0\">-avz<\/code><\/span><\/td>\n<td style=\"padding: 12px; color: #555; font-size: 0.9em;\">Add <code>-n<\/code> or <code>--dry-run<\/code> first to check what will copy without moving data.<\/td>\n<\/tr>\n<tr style=\"border-bottom: 1px solid #eee;\">\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,2,0,0\"><b data-path-to-node=\"10,2,0,0\" data-index-in-node=\"0\">Large Files<\/b> (e.g., Databases, images)<\/span><\/td>\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,2,1,0\">Focus on transfer speed, updating files directly, and tracking progress.<\/span><\/td>\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,2,2,0\"><code style=\"background: #fdfdfd; padding: 3px 6px; border: 1px solid #ddd; border-radius: 4px;\" data-path-to-node=\"10,2,2,0\" data-index-in-node=\"0\">--inplace<\/code><\/span><br \/>\n<span data-path-to-node=\"10,2,2,0\"><code style=\"background: #fdfdfd; padding: 3px 6px; border: 1px solid #ddd; border-radius: 4px;\" data-path-to-node=\"10,2,2,0\" data-index-in-node=\"0\">--progress<\/code><\/span><\/td>\n<td style=\"padding: 12px; color: #555; font-size: 0.9em;\">Use <code>--info=progress2<\/code> to output a single, continuously updating summary line for the <b>entire<\/b> transfer instead of a flood of individual file logs.<\/td>\n<\/tr>\n<tr style=\"border-bottom: 1px solid #eee;\">\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,3,0,0\"><b data-path-to-node=\"10,3,0,0\" data-index-in-node=\"0\">Deep Directories<\/b> (Massive file counts)<\/span><\/td>\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,3,1,0\">Turn off compression to save local CPU cycles.<\/span><\/td>\n<td style=\"padding: 12px;\"><span data-path-to-node=\"10,3,2,0\"><code style=\"background: #fdfdfd; padding: 3px 6px; border: 1px solid #ddd; border-radius: 4px;\" data-path-to-node=\"10,3,2,0\" data-index-in-node=\"0\">-av<\/code><\/span><\/td>\n<td style=\"padding: 12px; color: #555; font-size: 0.9em;\">Pairing with <code>--info=progress2<\/code> keeps your terminal clean when processing millions of files.<\/td>\n<\/tr>\n<tr style=\"border-bottom: 1px solid #eee; background-color: #fffaf0;\">\n<td style=\"padding: 12px;\"><b>Shared \/ Login Nodes<\/b> (Any cluster staging)<\/td>\n<td style=\"padding: 12px;\">Throttle your transfer speed to protect cluster filesystem storage stability.<\/td>\n<td style=\"padding: 12px;\"><code style=\"background: #fdfdfd; padding: 3px 6px; border: 1px solid #ccbba0; border-radius: 4px; color: #a04000;\">-av --bwlimit=20M<\/code><\/td>\n<td style=\"padding: 12px; color: #a04000; font-size: 0.9em; font-weight: 500;\"><b>Mandatory flag.<\/b> Unthrottled runs on shared nodes are subject to termination.<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<blockquote data-path-to-node=\"11\">\n<p data-path-to-node=\"11,0\"><b data-path-to-node=\"11,0\" data-index-in-node=\"0\">Note on <code data-path-to-node=\"11,0\" data-index-in-node=\"8\">-z<\/code> (Compression):<\/b> While <code data-path-to-node=\"11,0\" data-index-in-node=\"32\">-z<\/code> is great for slow networks, it can actually slow down transfers on high-speed local networks (like a LAN) because the CPU takes longer to compress the data than the wire takes to move it.<\/p>\n<\/blockquote>\n<hr data-path-to-node=\"12\" \/>\n<h3 data-path-to-node=\"13\">Key Options &amp; Advanced Tweaks<\/h3>\n<h4 data-path-to-node=\"14\">1. The Power of <code data-path-to-node=\"14\" data-index-in-node=\"16\">-a<\/code> (Archive Mode)<\/h4>\n<p data-path-to-node=\"15\">The <code data-path-to-node=\"15\" data-index-in-node=\"4\">-a<\/code> flag is a &#8220;collection&#8221; of flags (<code data-path-to-node=\"15\" data-index-in-node=\"40\">-rlptgoD<\/code>). It preserves permissions, symlinks, and timestamps.<\/p>\n<ul data-path-to-node=\"16\">\n<li>\n<p data-path-to-node=\"16,0,0\"><b data-path-to-node=\"16,0,0\" data-index-in-node=\"0\">When to NOT use <code data-path-to-node=\"16,0,0\" data-index-in-node=\"16\">-a<\/code>:<\/b> If you are moving files to a filesystem that doesn&#8217;t support Linux permissions (like an <b data-path-to-node=\"16,0,0\" data-index-in-node=\"108\">exFAT\/FAT32<\/b> USB drive), <code data-path-to-node=\"16,0,0\" data-index-in-node=\"132\">-a<\/code> will throw constant errors. Use <code data-path-to-node=\"16,0,0\" data-index-in-node=\"167\">-rtv<\/code> instead to sync data and times without trying to set Linux ownership.<\/p>\n<\/li>\n<\/ul>\n<h4 data-path-to-node=\"17\">2. The <code data-path-to-node=\"17\" data-index-in-node=\"7\">--inplace<\/code> Flag<\/h4>\n<p data-path-to-node=\"18\">Normally, rsync creates a hidden temporary file and moves it into place once finished.<\/p>\n<ul data-path-to-node=\"19\">\n<li>\n<p data-path-to-node=\"19,0,0\"><b data-path-to-node=\"19,0,0\" data-index-in-node=\"0\">Why use <code data-path-to-node=\"19,0,0\" data-index-in-node=\"8\">--inplace<\/code>?<\/b> It updates the destination file directly. This is vital for <b data-path-to-node=\"19,0,0\" data-index-in-node=\"79\">Large Files<\/b> where you don&#8217;t have enough disk space to hold two copies of the file at once.<\/p>\n<\/li>\n<li>\n<p data-path-to-node=\"19,1,0\"><b data-path-to-node=\"19,1,0\" data-index-in-node=\"0\">Warning:<\/b> If the transfer fails halfway, the destination file will be corrupted until you run rsync again to finish it.<\/p>\n<\/li>\n<\/ul>\n<hr data-path-to-node=\"20\" \/>\n<div id=\"model-response-message-contentr_a63b1e298a95fe51\" class=\"markdown markdown-main-panel enable-luminous-fast-follows enable-updated-hr-color tutor-markdown-rendering\" dir=\"ltr\" aria-busy=\"false\" aria-live=\"polite\">\n<h3 data-path-to-node=\"3\">Important: Shared Storage Etiquette &amp; Bandwidth Throttling<\/h3>\n<p data-path-to-node=\"4\">Because our cluster environments share a central high-performance network storage backend (<code data-path-to-node=\"4\" data-index-in-node=\"91\">mind-nas<\/code>), heavy, unthrottled file movements can easily saturate the storage pipe&#8217;s metadata performance. When this happens, it causes kernel deadlocks that freeze terminal sessions and login capabilities for all cluster users.<\/p>\n<p data-path-to-node=\"5\">To maintain cluster stability, <b data-path-to-node=\"5\" data-index-in-node=\"31\">you must throttle your transfers<\/b> when moving data on shared infrastructure.<\/p>\n<h4 data-path-to-node=\"6\">The <code data-path-to-node=\"6\" data-index-in-node=\"4\">--bwlimit<\/code> Flag<\/h4>\n<p data-path-to-node=\"7\"><code data-path-to-node=\"7\" data-index-in-node=\"0\">rsync<\/code> has a built-in mechanism to cap its maximum transfer speed. Adding this flag leaves plenty of network and disk performance left over for your peers.<\/p>\n<ul data-path-to-node=\"8\">\n<li>\n<p data-path-to-node=\"8,0,0\"><b data-path-to-node=\"8,0,0\" data-index-in-node=\"0\">Recommended Cap:<\/b> <code data-path-to-node=\"8,0,0\" data-index-in-node=\"17\">20M<\/code> (20 Megabytes per second). This provides a fast transfer while completely protecting the storage backend from lagging.<\/p>\n<\/li>\n<\/ul>\n<h4 data-path-to-node=\"9\">Safe Transfer Example:<\/h4>\n<div class=\"code-block ng-tns-c82932357-113 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation\">\n<div class=\"formatted-code-block-internal-container ng-tns-c82932357-113\">\n<div class=\"animated-opacity ng-tns-c82932357-113\">\n<div class=\"code-block-decoration header-formatted gds-emphasized-body-m ng-tns-c82932357-113 ng-star-inserted\">\n<p><span class=\"ng-tns-c82932357-113\">Bash<\/span><\/p>\n<div class=\"buttons ng-tns-c82932357-113 ng-star-inserted\"><\/div>\n<\/div>\n<pre class=\"ng-tns-c82932357-113\"><code class=\"code-container formatted ng-tns-c82932357-113\" role=\"text\" data-test-id=\"code-content\"><span class=\"hljs-comment\"># Throttling your transfer to 20MB\/s (Mandatory on shared login nodes)<\/span>\r\nrsync -a --info=progress2 --bwlimit=20M \/home\/dpane\/mysrcdata \/user_data\/dpane\/my-destination-data\/\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<h4 data-path-to-node=\"11\">Best Practices Checklist:<\/h4>\n<ol start=\"1\" data-path-to-node=\"12\">\n<li>\n<p data-path-to-node=\"12,0,0\"><b data-path-to-node=\"12,0,0\" data-index-in-node=\"0\">Never run unthrottled transfers:<\/b> Large datasets or folders with tens of thousands of sub-files should always be restricted using <code data-path-to-node=\"12,0,0\" data-index-in-node=\"147\">--bwlimit<\/code>.<\/p>\n<\/li>\n<li>\n<p data-path-to-node=\"12,1,0\"><b data-path-to-node=\"12,1,0\" data-index-in-node=\"0\">Use Compute Nodes for I\/O:<\/b> For data migrations, multi-terabyte datasets, or deep directory <code data-path-to-node=\"12,1,0\" data-index-in-node=\"105\">find<\/code> loops, spin up an interactive session via Slurm (<code data-path-to-node=\"12,1,0\" data-index-in-node=\"159\">srun -p cpu --cpus-per-task=2 --mem=4G --pty bash<\/code>) and run your script inside the allocation rather than on the shared login gateway.<\/p>\n<\/li>\n<\/ol>\n<hr data-path-to-node=\"20\" \/>\n<\/div>\n<h3 data-path-to-node=\"21\">Examples<\/h3>\n<h4 data-path-to-node=\"22\">Sync Local to Remote<\/h4>\n<div class=\"code-block ng-tns-c35263107-39 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation\">\n<div class=\"formatted-code-block-internal-container ng-tns-c35263107-39\">\n<div class=\"animated-opacity ng-tns-c35263107-39\">\n<pre class=\"ng-tns-c35263107-39\"><code class=\"code-container formatted ng-tns-c35263107-39\" role=\"text\" data-test-id=\"code-content\">rsync -avz full_path_to_source -e <span class=\"hljs-string\">\"ssh -l username\"<\/span> server_address:full_path_to_destination\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<h4 data-path-to-node=\"24\">Sync Remote to Local<\/h4>\n<div class=\"code-block ng-tns-c35263107-40 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation\">\n<div class=\"formatted-code-block-internal-container ng-tns-c35263107-40\">\n<div class=\"animated-opacity ng-tns-c35263107-40\">\n<pre class=\"ng-tns-c35263107-40\"><code class=\"code-container formatted ng-tns-c35263107-40\" role=\"text\" data-test-id=\"code-content\">rsync -avz -e <span class=\"hljs-string\">\"ssh -l username\"<\/span> server_address:full_path_to_source full_path_to_destination\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<h4 data-path-to-node=\"26\">Specific Example: Raptor to MacOS (Copying the Folder)<\/h4>\n<p data-path-to-node=\"27\"><i data-path-to-node=\"27\" data-index-in-node=\"0\">This command copies the folder \u201ctestfolder\u201d <b data-path-to-node=\"27\" data-index-in-node=\"44\">and<\/b> its contents to your home directory.<\/i><\/p>\n<div class=\"code-block ng-tns-c35263107-41 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation\">\n<div class=\"formatted-code-block-internal-container ng-tns-c35263107-41\">\n<div class=\"animated-opacity ng-tns-c35263107-41\">\n<pre class=\"ng-tns-c35263107-41\"><code class=\"code-container formatted ng-tns-c35263107-41\" role=\"text\" data-test-id=\"code-content\">rsync -avz -e <span class=\"hljs-string\">\"ssh -l dpane\"<\/span> raptor.ni.cmu.edu:\/home\/dpane\/testfolder \/Volumes\/HD\/dpane\/\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<h4 data-path-to-node=\"29\">Specific Example: Raptor to MacOS (Copying Contents ONLY)<\/h4>\n<p data-path-to-node=\"30\"><i data-path-to-node=\"30\" data-index-in-node=\"0\">By adding the trailing <code data-path-to-node=\"30\" data-index-in-node=\"23\">\/<\/code> on the source, you copy only the <b data-path-to-node=\"30\" data-index-in-node=\"58\">contents<\/b> inside &#8220;testfolder&#8221; into the destination.<\/i><\/p>\n<div class=\"code-block ng-tns-c35263107-42 ng-animate-disabled ng-trigger ng-trigger-codeBlockRevealAnimation\">\n<div class=\"formatted-code-block-internal-container ng-tns-c35263107-42\">\n<div class=\"animated-opacity ng-tns-c35263107-42\">\n<pre class=\"ng-tns-c35263107-42\"><code class=\"code-container formatted ng-tns-c35263107-42\" role=\"text\" data-test-id=\"code-content\">rsync -avz -e <span class=\"hljs-string\">\"ssh -l dpane\"<\/span> raptor.ni.cmu.edu:\/home\/dpane\/testfolder\/ \/Volumes\/HD\/dpane\/\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<hr data-path-to-node=\"32\" \/>\n<h3 data-path-to-node=\"33\">FAQ<\/h3>\n<p data-path-to-node=\"34\"><b data-path-to-node=\"34\" data-index-in-node=\"0\">Q: Why should I use rsync instead of scp?<\/b><\/p>\n<p data-path-to-node=\"34\"><b data-path-to-node=\"34\" data-index-in-node=\"42\">A:<\/b> When you are transferring a large amount of data, if the transfer is disconnected, rsync will pick it up where it left off. <code data-path-to-node=\"34\" data-index-in-node=\"169\">scp<\/code> doesn\u2019t do this. Use <code data-path-to-node=\"34\" data-index-in-node=\"194\">scp<\/code> for one or two small files; use <code data-path-to-node=\"34\" data-index-in-node=\"230\">rsync<\/code> for everything else, especially multi-GB data.<\/p>\n<p data-path-to-node=\"35\"><b data-path-to-node=\"35\" data-index-in-node=\"0\">Q: Does rsync verify files?<\/b><\/p>\n<p data-path-to-node=\"35\"><b data-path-to-node=\"35\" data-index-in-node=\"28\">A:<\/b> Yes. Rsync performs checksums to verify data integrity. If a file exists, it may skip it if the size and time match, but it always verifies the data it actually transfers.<\/p>\n<p data-path-to-node=\"36\"><b data-path-to-node=\"36\" data-index-in-node=\"0\">Q: Can I see what will happen before I run it?<\/b><\/p>\n<p data-path-to-node=\"36\"><b data-path-to-node=\"36\" data-index-in-node=\"47\">A:<\/b> Yes! Add <code data-path-to-node=\"36\" data-index-in-node=\"59\">-n<\/code> (or <code data-path-to-node=\"36\" data-index-in-node=\"66\">--dry-run<\/code>) to your command. It will list exactly which files would be moved without actually making any changes.<\/p>\n<hr data-path-to-node=\"37\" \/>\n<p data-path-to-node=\"38\"><b data-path-to-node=\"38\" data-index-in-node=\"0\">Additional Information:<\/b><\/p>\n<p data-path-to-node=\"38\">There is a wealth of documentation online. <a class=\"ng-star-inserted\" href=\"https:\/\/www.tecmint.com\/rsync-local-remote-file-synchronization-commands\/\" target=\"_blank\" rel=\"noopener\">This guide<\/a> provides 10 practical examples.<\/p>\n<ul data-path-to-node=\"39\">\n<li>\n<p data-path-to-node=\"39,0,0\"><b data-path-to-node=\"39,0,0\" data-index-in-node=\"0\">Note:<\/b> I suggest <b data-path-to-node=\"39,0,0\" data-index-in-node=\"16\">avoiding<\/b> example #8 (&#8220;Automatically Delete source Files&#8221;) unless you are absolutely certain of your workflow.<\/p>\n<\/li>\n<\/ul>\n<\/div>\n","protected":false},"excerpt":{"rendered":"<p>Rsync: File Copying &amp; Syncing Rsync (Remote Sync) is the gold standard for moving data in Linux\/Unix. Unlike scp, it is &#8220;delta-aware,&#8221; meaning it only sends the parts of files that have changed, saving massive amounts of time and bandwidth. The &#8220;Trailing Slash&#8221; Rule This is the most common point&#8230;<\/p>\n","protected":false},"author":1,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"footnotes":""},"ht-kb-category":[17],"ht-kb-tag":[],"class_list":["post-361","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-unix-help"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/361","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/comments?post=361"}],"version-history":[{"count":36,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/361\/revisions"}],"predecessor-version":[{"id":727,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/361\/revisions\/727"}],"wp:attachment":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/media?parent=361"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-category?post=361"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-tag?post=361"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}