{"id":722,"date":"2026-06-15T15:43:37","date_gmt":"2026-06-15T15:43:37","guid":{"rendered":"https:\/\/ni.cmu.edu\/computing\/?page_id=722"},"modified":"2026-06-15T15:43:37","modified_gmt":"2026-06-15T15:43:37","slug":"using-ssh-safely-alongside-slurm","status":"publish","type":"page","link":"https:\/\/ni.cmu.edu\/computing\/using-ssh-safely-alongside-slurm\/","title":{"rendered":"Using SSH Safely Alongside Slurm"},"content":{"rendered":"

On the Mind cluster, you are only permitted to SSH into a compute node after Slurm has officially granted you an active allocation on that node (via an active srun<\/code> or sbatch<\/code> job).<\/p>\n

However, opening a separate, raw SSH window to check on your code or run interactive process like vscode bypasses Slurm’s automatic environment protections. If you do not manually bind your new SSH window to your assigned GPUs, your code will blindly default to Physical GPU 0, accidentally hijacking another user’s hardware and possibly causing their jobs to crash with Out-of-Memory (OOM) errors.<\/p>\n

The following explains how you how can identify your exact physical GPUs allocated to you and safely route your SSH GPU processing to them.<\/p>\n

Step 1: Find Your Assigned Physical GPU IDs<\/h2>\n

Before running anything in a direct SSH window, you must find out which specific physical hardware slots Slurm has reserved for your active job.<\/p>\n

From your active Slurm terminal window, run the following command (Slurm will automatically substitute your active Job ID for you):<\/p>\n

\n
\n
\n
Bash<\/span><\/p>\n
<\/div>\n<\/div>\n
scontrol show job $SLURM_JOB_ID<\/span> -d | grep \"Nodes=\"<\/span>\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n
How to read the output:<\/h3>\n
Look at the end of the returned line for the GRES<\/code> and IDX<\/code> labels.<\/p>\n
    \n
  • \n
    If it says GRES=gpu:3(IDX:5-7)<\/code>, Slurm has assigned you physical GPUs 5, 6, and 7.<\/p>\n<\/li>\n
  • \n
    If it says GRES=gpu:L40S:2(IDX:2-3)<\/code>, Slurm has assigned you physical GPUs 2 and 3.<\/p>\n<\/li>\n<\/ul>\n
    Step 2: Bind Your SSH Window to Your GPUs<\/h2>\n
    Once you know your physical IDX<\/code> numbers, open your separate terminal tab and log into your assigned compute node via SSH.<\/p>\n
    Before you launch any Python script, Jupyter notebook, or background daemon in that SSH window, you must set your environmental boundary. Run the export<\/code> command using your exact assigned indices:<\/p>\n
    \n
    \n
    \n
    # Replace the numbers with your actual allocated physical indices<\/span>\r\nexport<\/span> CUDA_VISIBLE_DEVICES=5,6,7\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n
    \n
    What this does behind the scenes<\/h3>\n
    Setting this variable forces the NVIDIA driver and machine learning frameworks (like PyTorch, TensorFlow, etc) to completely hide the rest of the node’s GPUs from your SSH session.<\/p>\n
    To your code, your assigned cards are the only<\/i> cards that exist on the machine. This guarantees your code will never accidentally bleed over onto your neighbor’s hardware.<\/p>\n<\/blockquote>\n
    Best Practices Checklist<\/h2>\n
      \n
    • \n
      You can always double-check that your boundary is active in your current SSH terminal by running echo $CUDA_VISIBLE_DEVICES<\/code>.<\/p>\n<\/li>\n
    • \n
      A clean, neighbor-safe workflow inside an SSH window looks like this:<\/p>\n
      \n
      \n
      \n
      # 1. Lock down your hardware footprint<\/span>\r\nexport<\/span> CUDA_VISIBLE_DEVICES=5,6,7\r\n\r\n# 2. Activate your environment and run your code<\/span>\r\nconda activate my_env\r\npython my_daemon.py &\r\n<\/code><\/pre>\n<\/div>\n<\/div>\n<\/div>\n<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
      On the Mind cluster, you are only permitted to SSH into a compute node after Slurm has officially granted you an active allocation on that node (via an active srun or sbatch job). However, opening a separate, raw SSH window to check on your code or run interactive process like…<\/p>\n","protected":false},"author":1,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-722","page","type-page","status-publish","hentry"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/pages\/722","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/comments?post=722"}],"version-history":[{"count":1,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/pages\/722\/revisions"}],"predecessor-version":[{"id":723,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/pages\/722\/revisions\/723"}],"wp:attachment":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/media?parent=722"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}