{"id":770,"date":"2026-11-19T18:15:49","date_gmt":"2026-11-19T18:15:49","guid":{"rendered":"https:\/\/ni.cmu.edu\/computing\/?post_type=ht_kb&#038;p=770"},"modified":"2026-11-19T18:15:49","modified_gmt":"2026-11-19T18:15:49","slug":"transferring-data-with-globus-on-condor-and-raptor","status":"publish","type":"ht_kb","link":"https:\/\/ni.cmu.edu\/computing\/knowledge-base\/transferring-data-with-globus-on-condor-and-raptor\/","title":{"rendered":"Transferring Data with Globus on Condor and Raptor"},"content":{"rendered":"\r\n<p class=\"wp-block-paragraph\"><strong>Globus Connect Personal (GCP)<\/strong> allows you to transfer large datasets to and from <strong>Condor<\/strong> and <strong>Raptor<\/strong> without opening inbound firewall ports or exposing the servers to external IP addresses. Because GCP runs in user-space and initiates outbound connections only, any lab with an active account on Condor or Raptor can set up a personal collection in a few minutes without root access.<\/p>\r\n\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Prerequisites<\/h2>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li>An active account and SSH access to <strong>Condor<\/strong> or <strong>Raptor<\/strong>.<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li>A Globus account and credentials (log in at <a href=\"https:\/\/app.globus.org\" target=\"_blank\" rel=\"noreferrer noopener\">app.globus.org<\/a>\u00a0via Web Login).<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li>Read\/write Linux filesystem permissions on the directory you intend to transfer data into or out of.<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Step 1: Download and Install Globus Connect Personal<\/h2>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">SSH into <strong>Condor<\/strong> or <strong>Raptor<\/strong> and install Globus Connect Personal in your home directory. Moving the extracted folder to a fixed path (<code>~\/.globusconnectpersonal<\/code>) ensures your commands remain consistent across future updates.<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-code\"><code>cd ~\r\nwget https:\/\/downloads.globus.org\/globus-connect-personal\/linux\/stable\/globusconnectpersonal-latest.tgz\r\ntar xzf globusconnectpersonal-latest.tgz\r\nmv globusconnectpersonal-* ~\/.globusconnectpersonal\r\nrm globusconnectpersonal-latest.tgz<\/code><\/pre>\r\n\r\n\r\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Step 2: Register Your Collection<\/h2>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Because Condor and Raptor are headless Linux servers, run the setup script in CLI mode:<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-code\"><code>~\/.globusconnectpersonal\/globusconnectpersonal -setup<\/code><\/pre>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>The terminal will print a login URL. Copy and paste that URL into a web browser on your local computer.<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>Log in with your account credentials and click <strong>Allow<\/strong> to grant permissions.<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>Copy the <strong>Authorization Code<\/strong> displayed in your browser and paste it back into your SSH terminal prompt.<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>When prompted for a <strong>Collection Name<\/strong>, use a descriptive label that identifies both the server and your Account id (for example, <code>CMU-NI-Condor-&lt;lab-user-name&gt;<\/code>or <code>CMU-NI-Raptor-&lt;lab-user-name&gt; where &lt;lab-user-name&gt; is replaced with your lab account name<\/code>).<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Setup credentials and host certificates are saved automatically in <code>~\/.globusonline\/<\/code>. Keep this directory private and do not alter its permissions.<\/p>\r\n\r\n\r\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Step 3: Configure Accessible Directories<\/h2>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">By default, Globus Connect Personal only allows transfers to and from your home directory (<code>~\/<\/code>). If your lab data lives on a shared mount or data volume (such as <code>\/data\/...<\/code> or <code>\/scratch\/...<\/code>), you must explicitly allow access to that path in <code>~\/.globusonline\/lta\/config-paths<\/code>.<\/p>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Open the configuration file:<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-code\"><code>emacs ~\/.globusonline\/lta\/config-paths<\/code><\/pre>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Each line follows a three-column CSV format: <code>path,shareable,writable<\/code> (<code>1<\/code> = enabled, <code>0<\/code> = disabled). Add your project directory on a new line:<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-code\"><code>~\/,0,1\r\n\/path\/to\/your\/data_directory,1,1<\/code><\/pre>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li><strong><code>1,1<\/code><\/strong>: Allows Globus to read\/write to the directory <strong>and<\/strong> allows you to create shared Guest Collections on it (if your CMU Globus account has subscription privileges).<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li><strong><code>0,1<\/code><\/strong>: Allows Globus to read\/write to the directory for your personal transfers only.<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Save and exit (<code>Ctrl+O<\/code>, <code>Enter<\/code>, <code>Ctrl+X<\/code>).<\/p>\r\n<blockquote>Any time you edit <code>config-paths<\/code>, you must restart Globus Connect Personal for the changes to take effect.<\/blockquote>\r\n\r\n\r\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Step 4: Start and Run Globus in the Background<\/h2>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Globus Connect Personal must be running on the server while transfers are active.<\/p>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Option A: Manual Start (Quick Transfers)<\/h3>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Start the client in the background:<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-code\"><code>~\/.globusconnectpersonal\/globusconnectpersonal -start &amp;<\/code><\/pre>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">To check if it is running or stop it after your transfer completes:<\/p>\r\n\r\n\r\n\r\n<pre class=\"wp-block-code\"><code>~\/.globusconnectpersonal\/globusconnectpersonal -status\r\n~\/.globusconnectpersonal\/globusconnectpersonal -stop<\/code><\/pre>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Option B: Persistent Background Service (Recommended for Large Transfers)<\/h3>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">To keep your collection online across SSH disconnects and server reboots, configure a <code>systemd<\/code> user service:<\/p>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>Create the user service directory and unit file:\r\n<pre class=\"wp-block-code\"><code>mkdir -p ~\/.config\/systemd\/user\r\ncat &lt;&lt; 'EOF' &gt; ~\/.config\/systemd\/user\/globusconnectpersonal.service\r\n[Unit]\r\nDescription=Globus Connect Personal\r\nAfter=network.target\r\n\r\n[Service]\r\nExecStart=%h\/.globusconnectpersonal\/globusconnectpersonal -start\r\nExecStop=%h\/.globusconnectpersonal\/globusconnectpersonal -stop\r\nRestart=on-failure\r\n\r\n[Install]\r\nWantedBy=default.target\r\nEOF<\/code><\/pre>\r\n<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>Enable and start the service:\r\n<pre class=\"wp-block-code\"><code>systemctl --user daemon-reload\r\nsystemctl --user enable --now globusconnectpersonal<\/code><\/pre>\r\n<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>Allow your user services to run when you are not actively logged in via SSH:\r\n<pre class=\"wp-block-code\"><code>loginctl enable-linger $USER<\/code><\/pre>\r\n<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Step 5: Transferring and Sharing Data<\/h2>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Personal Transfers<\/h3>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>Open <a href=\"https:\/\/app.globus.org\" target=\"_blank\" rel=\"noreferrer noopener\">app.globus.org<\/a> and navigate to the <strong>File Manager<\/strong>.<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>In the left <strong>Collection<\/strong> field, search for your new collection (e.g., <code>CMU-NI-Condor-<\/code><code>&lt;lab-user-name&gt;<\/code>.<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>In the right <strong>Collection<\/strong> field, select the remote endpoint (such as another university cluster or your personal workstation).<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<ol class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ol class=\"wp-block-list\">\r\n<li>Select the files or folders to move and click <strong>Start<\/strong>. Globus will handle checksum verification, retries, and email notification upon completion.<\/li>\r\n<\/ol>\r\n<\/li>\r\n<\/ol>\r\n\r\n\r\n\r\n<h3 class=\"wp-block-heading\">Collaborating with External Researchers<\/h3>\r\n\r\n\r\n\r\n<p class=\"wp-block-paragraph\">Because Condor and Raptor remain closed to inbound external traffic, use one of the following methods to exchange data with collaborators at other institutions:<\/p>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li><strong>Pulling Data from a Collaborator (Standard):<\/strong> Ask your collaborator to share their source folder on their institution&#8217;s Globus endpoint with your Globus identity.\u00a0Once shared, open their shared collection on one side of the Globus File Manager and your Condor\/Raptor collection on the other to initiate the transfer.<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li><strong>Allowing a Collaborator to Push Data to Condor\/Raptor (Requires CMU Globus Subscription):<\/strong> If your CMU account has Globus subscription privileges, open <strong>Collections<\/strong> in <a href=\"https:\/\/app.globus.org\" target=\"_blank\" rel=\"noreferrer noopener\">app.globus.org<\/a>, select your Condor or Raptor collection, and click <strong>Share<\/strong> \u2192 <strong>Add Guest Collection<\/strong>. Specify the target subfolder and grant <strong>Read\/Write<\/strong> permissions to your collaborator&#8217;s institutional Globus ID.<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n<hr class=\"wp-block-separator has-alpha-channel-opacity\" \/>\r\n\r\n\r\n<h2 class=\"wp-block-heading\">Troubleshooting<\/h2>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li><strong><code>500 Command failed : Path not allowed<\/code>:<\/strong> The target path is either missing from <code>~\/.globusonline\/lta\/config-paths<\/code> or GCP was not restarted after editing the file. Verify the path entry and run <code>~\/.globusconnectpersonal\/globusconnectpersonal -stop<\/code> followed by <code>-start &amp;<\/code> (or <code>systemctl --user restart globusconnectpersonal<\/code>).<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li><strong>Collection shows &#8220;Offline&#8221; in the web interface:<\/strong> Verify the background process is active (<code>~\/.globusconnectpersonal\/globusconnectpersonal -status<\/code>) and ensure outbound connections to <code>TCP 2223<\/code> and <code>TCP\/UDP 50000\u201351000<\/code> are permitted from the host.<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n\r\n\r\n\r\n<ul class=\"wp-block-list\">\r\n<li style=\"list-style-type: none;\">\r\n<ul class=\"wp-block-list\">\r\n<li><strong>Setting up both Condor and Raptor:<\/strong> Run Steps 1\u20134 separately on each server using distinct collection names (e.g., <code>CMU-NI-Condor-&lt;andrewID&gt;<\/code> and <code>CMU-NI-Raptor-&lt;andrewID&gt;<\/code>). If your home directory (<code>~\/<\/code>) is shared across servers via NFS, use <code>--dir<\/code> to store separate configuration directories per host:\r\n<pre class=\"wp-block-code\"><code>~\/.globusconnectpersonal\/globusconnectpersonal -dir ~\/.globusonline-condor -setup\r\n~\/.globusconnectpersonal\/globusconnectpersonal -dir ~\/.globusonline-condor -start &amp;<\/code><\/pre>\r\n<\/li>\r\n<\/ul>\r\n<\/li>\r\n<\/ul>\r\n","protected":false},"excerpt":{"rendered":"<p>Globus Connect Personal (GCP) allows you to transfer large datasets to and from Condor and Raptor without opening inbound firewall ports or exposing the servers to external IP addresses. Because GCP runs in user-space and initiates outbound connections only, any lab with an active account on Condor or Raptor can&#8230;<\/p>\n","protected":false},"author":1,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"footnotes":""},"ht-kb-category":[13],"ht-kb-tag":[],"class_list":["post-770","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-raptor-condor-file-servers"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/770","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/comments?post=770"}],"version-history":[{"count":2,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/770\/revisions"}],"predecessor-version":[{"id":772,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/770\/revisions\/772"}],"wp:attachment":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/media?parent=770"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-category?post=770"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-tag?post=770"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}