Recommendations for Remote Development using SSH and VSCode
The Visual Studio Code Remote – SSH extension allows you to open a remote folder on any remote machine, virtual machine, or container with a running SSH server and take full advantage of VS Code’s feature set. Once connected to a server, you can interact with files and folders anywhere on the remote filesystem.
Users Gabriel Sarch and Nicholas Blauch hypothesize that the resources used by a large number of users running Visual Studio code (or other remote development tools) on the head node is contributing to some of the slowdowns users have been seeing. The following is a solution they found to connect these processes to the compute nodes instead of the head node. If their hypothesis is correct, we expect users adopting this method will reduce resources used by VSCode on the head node and improve its performance for all users.
How to use VsCode interactive Python capabilities on a compute node
The following content is from this Stack Overflow solution.
- ssh to the remote cluster
ssh username@mind.cs.cmu.edu - Once on the cluster, add your public key to the authorized keys. You will typically append the content of
~/.ssh/id_rsa.pub(on your local machine) to~/.ssh/authorized_keys(on the cluster) Note: To configure SSH authorized keys you may need to generate an SSH key pair on your local computer (i.e. terminal for unix, powershell for windows). To do this, usessh-keygen, which is included with the standard OpenSSH suite of tools. You will need to find the the .pub file based on the output path of the ssh-keygen tool. - Allocate some resources inside the cluster (
srun --pty bash). You can do this in a screen session so that the job stays running even after you log out. - Get the name of the compute node, typically visible in the command line as
username@mind-x-x). - On your local machine, modify the
~/.ssh/configfile. Using Gabriel’s example, modify your ssh config file, replacing Gabriel’s userid with yours :
Host mind.cs.cmu.edu
HostName mind.cs.cmu.edu
User gsarch
Host mind-*
HostName %h
ProxyJump mind.cs.cmu.edu
User gsarch
Once the changes to the ssh config file are applied, you will be able to access the node directly from VSCode by:
- CTRL+P > Remote-SSH: Connect to Host…
- mind-x-x (e.g. mind-1-1)
- you get connected to the node, now every interactive python you run (including jupyter and jupytext) will have access to your allocated resources
The node can also be directly accessed with “ssh mind-x-x”.
Useful tip:
Command to find the GPU ID slurm allocated to your job.
$ scontrol show job -d 12345 | grep -i 'Gres=gpu'
Nodes=mind-1-30 CPU_IDs=26-29 Mem=14336 GRES=gpu(IDX:1)Where IDX:1 == /dev/nvidia1
Problem solving
(Vladislav Ayzenberg provided information for this section)
Solving Initial Connection issues
When initially configuring VSCode, pay close attention to the permissions of the files and directories (i.e. private and public certificates and .ssh directory).
The following solution partially taken from this stack exchange solution [https://superuser.com/questions/215504/permissions-on-private-key-in-ssh-folder/215506#215506 ]
MacOS/UNIX File/Directory File Permissions
(Windows computer private/public key management and permissions differ from these instructions. However, they are equally as important to configure correctly.)
This help page has general information about File and Directory Permissions.
On your local machine, open a terminal window and confirm you are in your home directory.
imac:~ jim$ pwd /Users/jim
If you are not in your home directory, you can use the following `cd` command to change to it.
imac:~ jim$ cd ~
Confirm that the .ssh directory has the correct permission. Running the following command should give you permission results similar to this..
imac:~ jim$ ls -la | grep .ssh drwx------ 6 jim staff 192 Oct 11 2019 .sshimac:~ jim$ chmod 700 ~/.ssh/
You can check the file permission in the .ssh directory and should confirm they have the following permissions.
imac:~ jim$ ls -l ~/.ssh/ -rw------- 1 jim staff 0 Jun 29 11:43 authorized_keys -rw------- 1 jim staff 0 Jun 29 11:43 id_rsa.pub -rw-r--r-- 1 jim staff 0 Jun 29 11:42 id_rsa
If they don’t have the correct permission, use the following commands to change them to the appropriate settings.
chmod 644 ~/.ssh/id_rsa.pub chmod 600 ~/.ssh/id_rsa chmod 600 ~/.ssh/authorized_keys
VSCode connection suddenly stops working
If you suddenly experience issues connecting to the cluster from VSCode after it had been working (regardless of what node you use). Try resolving the issue by simply deleting the .vscode-server/ directory in your home directory and then try reconnecting. This forces VScode to re-install the dependencies it needs the next time it reconnects.
(Note: The time to delete the .vscode-server/ directory may take a while. You may need to manually reinstall some extensions next time you connect.)