{"id":114,"date":"2020-01-22T19:50:51","date_gmt":"2020-01-22T19:50:51","guid":{"rendered":"https:\/\/ni.cmu.edu\/computing\/?post_type=ht_kb&p=114"},"modified":"2023-10-11T12:57:10","modified_gmt":"2023-10-11T12:57:10","slug":"getting-started","status":"publish","type":"ht_kb","link":"https:\/\/ni.cmu.edu\/computing\/knowledge-base\/getting-started\/","title":{"rendered":"Cluster Quick Start Guide"},"content":{"rendered":"

This article is intended for our new users to begin using the cluster quickly. It summarizes basic information and points to the more detailed information. Users who understand and follow our policies and procedures will meet the our expectations of being good citizens. Please take the time to read our documentation so you don’t inconvenience your fellow community members.<\/p>\n

Documentation and Policies<\/h3>\n

All new “Mind” cluster users are expected<\/strong> to read and understand our Cluster Policies<\/a> page(s).\u00a0 This should only take a few minutes to read over. If you have questions, ask David before proceeding.<\/p>\n

Connecting to the Mind Cluster<\/h3>\n

You applied for a cluster account,<\/a> and you’ve been notified that your account on the cluster is created. This article explains how to connect to the MIND cluster.\u00a0<\/a><\/p>\n

File and Data Storage<\/h3>\n

By default, users land in their home directory ( \/home\/<username> ) when logging onto the cluster.\u00a0 There are various storage areas on the cluster for specified usage.\u00a0 Understand what volumes your expected to use for code, configurations, data and results.\u00a0 The Data Management<\/a> article will help you with this.<\/p>\n

Two critical rules our users must follow regarding their data storage:<\/h6>\n
    \n
  1. Users should only have data currently being used for processing and the results of recent processing saved on the cluster.<\/li>\n
  2. Use these three storage areas correctly.\n
      \n
    1. \/home\/<username><\/strong> is the user’s home directory and should only be used to store small files such as configuration files, documents, and source code [keep this to a maximum of 300GB]<\/li>\n
    2. \/user_data<username><\/strong> space is a 1TB partition for users to save their work.<\/li>\n
    3. \/lab_data\/<whateverlab><\/strong> is a 10TB partition for lab members to share and save their work.<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

      These volumes are accessible cluster wide (on all compute nodes as well as on the head node).<\/p>\n

      This is an example of navigating to the various volumes.<\/h6>\n
      [dpane@mind ~]$ cd \/lab_data\/fisherlab\/\r\n[dpane@mind fisherlab]$ ls\r\n[dpane@mind fisherlab]$ pwd\r\n\/lab_data\/fisherlab\r\n[dpane@mind fisherlab]$ cd\r\n[dpane@mind ~]$ cd \/user_data\/qianoum\/\r\n[dpane@mind qianoum]$ ls\r\n[dpane@mind qianoum]$ ls -ltra\r\ntotal 9\r\ndrwxr-xr-x\u00a0\u00a0 2 qianoum qianoum\u00a0\u00a0 2 Jan\u00a0 9 09:28 .\r\ndrwxrwxrwx 172 root\u00a0\u00a0\u00a0 root\u00a0\u00a0\u00a0 172 Jan 10 10:51 ..\r\n[dpane@mind qianoum]$\r\n<\/pre>\n
      You may also find this article on file and directory permissions<\/a> helpful.<\/p>\n
      Data transfer<\/h3>\n
      Transferring files can be done using any secure transfer protocol. Samba is NOT a secure protocol and is NOT supported on the cluster.\u00a0 Various commands such as scp, sftp, rsync over ssh, etc. are supported.<\/p>\n
      An example command from a Linux or MacOS computer to the MIND cluster<\/h6>\n
      rsync -avz <src directory path> -e \"ssh -l <username>\" mind.cs.cmu.edu:<full destination path on mind><\/pre>\n
      Tectia File Transfer(Windows) , Fetch (MacOs), or some other sftp software will work.
      \n      https:\/\/www.cmu.edu\/computing\/software\/<\/a><\/p>\n
      Software<\/h3>\n
      As new versions of software packages become available, we can made them available by installing them and then change the default user environment. A easy way to change the individual user environment is using the module command. The Modules package is a tool that simplifies shell initialization and lets users easily modify their environment during the session.<\/p>\n
      [dpane@mind-1-1 ~]$ module avail\r\n\r\n--------------------------------------- \/usr\/share\/Modules\/modulefiles ----------------------------------------\r\nanaconda3              dsi-studio             julia-1.2.0            python36-extras\r\ncuda-10.0              freesurfer-5.3.0       matlab-9.11            qt-4.8.2\r\ncuda-10.1              freesurfer-6.0.0       matlab-9.5             qt-4.8.5\r\ncuda-10.2              freesurfer-7.1.0       matlab-9.7             R-3.6.1\r\ncuda-11.1.1            fsl-6.0.3              module-git             rocks-openmpi\r\ncuda-9.2               gcc-4.7.4              module-info            rocks-openmpi_ib\r\ncudnn-10.0-7.3         gcc-4.9.2              modules                rstudio-1.2.5033\r\ncudnn-10.1-v7.6.5.32   gcc-6.3.0              mrtrix3-3.0.0-git      singularity\r\ncudnn-10.2-v7.6.5.32   git-2.23               null                   use.own\r\ncudnn-11.1.1-v8.0.4.30 git-2.30               openmpi-1.10-x86_64\r\ncudnn-9.2-7.6          glx-indirect           openmpi-1.8-x86_64\r\ndot                    julia-0.3.6            openmpi-x86_64\r\n[dpane@mind-1-1 ~]$ \r\n\r\n<\/pre>\n
      Use module load <module filename><\/em> to activate the software. (e.g.module load anaconda3 ) and module unload <module filename><\/em> to deactivate the package.<\/p>\n
      Example of changing versions of Python and pip using the anaconda3 module file.<\/h6>\n
      [dpane@mind ~]$ which python\r\n\/usr\/bin\/python\r\n[dpane@mind ~]$ python --version\r\nPython 2.7.5\r\nx[dpane@mind ~]$ which pip\r\n\/usr\/bin\/pip\r\n[dpane@mind ~]$ pip --version\r\npip 8.1.2 from \/usr\/lib\/python2.7\/site-packages (python 2.7)\r\n[dpane@mind ~]$ module load anaconda3\r\n[dpane@mind ~]$ which python\r\n\/opt\/anaconda3\/bin\/python\r\n[dpane@mind ~]$ which pip\r\n\/opt\/anaconda3\/bin\/pip\r\n[dpane@mind ~]$ pip --version\r\npip 18.1 from \/opt\/anaconda3\/lib\/python3.7\/site-packages\/pip (python 3.7)\r\n[dpane@mind ~]$ python --version\r\nPython 3.7.1\r\n[dpane@mind ~]$\r\n[dpane@mind ~]$ module unload anaconda3\r\n[dpane@mind ~]$\r\n<\/pre>\n
      When users are creating batch job scripts, they should load the appropriate module within the script so their software will run.<\/p>\n
      Running jobs<\/h3>\n
      Users – Avoid processing on the login node ( or “head node”). This includes VSCode usage (see this article on VSCode setup<\/a>)<\/strong> <\/span>The head node is the machine that all users log into – hostname: mind.cs.cmu.edu<\/strong>. To run jobs, users need to schedule their jobs using the cluster job scheduler.\u00a0 The scheduler will assign the job to a compute node.\u00a0 The Slurm Workload Manager (or Slurm for short) job scheduler is the only job scheduler we use on the mind cluster. Users can request resources (machines, cpus, cpus, memory, time) and Slurm with allocate the appropriate resources (a compute node that meets the requested resources).\u00a0 If the resources aren’t available, the scheduler will keep the job request in the queue until a machine(s) with the resources are available.\u00a0 User requests that cannot be met due to the cluster not having the required resources will never get scheduled.\u00a0 The scheduler uses a number of factors and a complicated algorithm to allocate resources fairly.<\/p>\n
      There are two main job queues which all users are able to submit their jobs to.<\/p>\n
        \n
      1. cpu –\u00a0 this encompasses all of the cpu nodes<\/li>\n
      2. gpu –\u00a0 this encompasses all of the gpu nodes<\/li>\n<\/ol>\n
        You are expected to only submit to the GPU queue when your job(s) will be using GPU(s) for its’ processing.<\/h6>\n
        Users are able to request batch or interactive jobs.\u00a0 An interactive\u00a0 job will assign the user a session on a compute node so they can work interactively.<\/p>\n
        Example interactive session request on a specific cpu node (you can eliminate specifying a node if you want):<\/h6>\n
        srun -p cpu --cpus-per-task=1 --gres=gpu:0 --mem=10GB --time=4:00:00 --nodelist=mind-0-11 --pty bash<\/pre>\n
        Example of interactive session request with X11 display on non gpu node:<\/h6>\n
        srun --x11 -p cpu --cpus-per-task=1 --gres=gpu:0 --mem=0 --time=4:00:00 --pty $SHELL<\/pre>\n
        The following is an example of requesting an interactive session on a compute node. In the example, I simply ran the command `hostname` when I was on the head node; requested an interactive session on a node in the cpu queue where I ran the same ‘host name’ command; I then exited the interactive session and got back onto the head node.<\/p>\n
        [dpane@mind ~]$ hostname\r\nmind.cs.cmu.edu\r\n[dpane@mind ~]$ srun -p cpu --cpus-per-task=1 --gres=gpu:0 --mem=10GB --time=4:00:00 --nodelist=mind-0-11 --pty bash\r\n[dpane@mind-0-11 ~]$\r\n[dpane@mind-0-11 ~]$\r\n[dpane@mind-0-11 ~]$ hostname\r\nmind-0-11.eth\r\n[dpane@mind-0-11 ~]$ exit\r\nexit\r\n[dpane@mind ~]$\r\n<\/pre>\n
        More information and examples can be found on the Slurm Scheduler<\/a> page.<\/p>\n","protected":false},"excerpt":{"rendered":"
        This article is intended for our new users to begin using the cluster quickly. It summarizes basic information and points to the more detailed information. Users who understand and follow our policies and procedures will meet the our expectations of being good citizens. Please take the time to read our…<\/p>\n","protected":false},"author":1,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"footnotes":""},"ht-kb-category":[12],"ht-kb-tag":[],"class_list":["post-114","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-getting-started"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/114","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/comments?post=114"}],"version-history":[{"count":39,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/114\/revisions"}],"predecessor-version":[{"id":597,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/114\/revisions\/597"}],"wp:attachment":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/media?parent=114"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-category?post=114"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-tag?post=114"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}