{"id":100,"date":"2020-01-22T19:36:47","date_gmt":"2020-01-22T19:36:47","guid":{"rendered":"https:\/\/ni.cmu.edu\/computing\/?post_type=ht_kb&p=100"},"modified":"2020-01-22T19:37:32","modified_gmt":"2020-01-22T19:37:32","slug":"lens-software","status":"publish","type":"ht_kb","link":"https:\/\/ni.cmu.edu\/computing\/knowledge-base\/lens-software\/","title":{"rendered":"Lens Software"},"content":{"rendered":"
This needs updated for the SLURM scheduler!<\/h5>\n

A sample PBS submit script can be found below. There are a few parts.<\/p>\n

Firstly, you will need to set the following environment variables in your .bashrc file located in your home directory.<\/p>\n

export LENSDIR=\/data2\/plautlab\/Lens
\nexport HOSTTYPE=x86_64
\nexport PATH=$PATH:$LENSDIR\/Bin\/$HOSTTYPE
\nexport LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LENSDIR\/Bin\/${HOSTTYPE}
\nexport TCL_LIBRARY=\/data2\/plautlab\/Lens\/Bin\/x86_64\/<\/p>\n

The next time you start a shell session, your environmental variable declaration will be read and passed on to the shell environment. You can force your current session to read the file now by typing in the shell that you are using:<\/p>\n

source ~\/.bashrc<\/p>\n

Or you can log out of the cluster and back in so they are used.<\/p>\n

The basic script is test.qsub<\/em>, and the two important lines in that script are the line that sets the number of processors to use and the line calling the Lens software.<\/p>\n

Currently, the script reads: #PBS -l nodes=4:ppn=4<\/em> which tells PBS to use 4 of the compute nodes (out of the current 7 on psych-o) and use 4 processors on each node (out of the 8 processors per node). Lens will use the first processor as the lens server, and the rest (15 in this case) as the clients for computation. You can change these numbers in the script before using it to set the number of nodes or processors you want. (nodes=2:ppn=8 would also give you 16 processors, using all the processors on 2 of the nodes, for example).<\/p>\n

The other important line is: lens -b server.qsub.tcl > server.out<\/em><\/p>\n

You can alter this line to use different input or output files. server.qsub.tcl has been changed from the default server.tcl file so you no longer need to edit it to list which machines to use for the job. That’s now handled through PBS, by setting the number of processors in test.qsub. It makes use of an additional little script, ‘tailnodelist’, which needs to be in the same location as the server.qsub.tcl file (see below) along with the rest of the scripts.<\/p>\n

So to run Lens via PBS:<\/p>\n

    \n
  1. Edit test.qsub to set the requested number of nodes\/processors, as needed.<\/li>\n
  2. Edit test.qsub to change the input or output files, as needed.<\/li>\n
  3. Run the command ‘qsub test.qsub’<\/li>\n<\/ol>\n

    One can always make different versions of test.qsub (and give them different filenames), and then just run them with the ‘qsub’ command.<\/p>\n

    \n

    <\/a> Begin test.qsub<\/h4>\n
    \n

    #!\/bin\/sh
    \n# specify a jobname
    \n#PBS -N test<\/p>\n

    # specify number of nodes (ppn should be 8 to reserve all cores on the node)
    \n#PBS -l nodes=4:ppn=4<\/p>\n

    # misc other PBS settings:
    \n#PBS -j eo<\/p>\n

    # echo “Moving to plautlab directory.”
    \ncd \/data2\/plautlab\/Lens
    \n# echo “Beginning lens job.”
    \nlens -b server.qsub.tcl > server.out
    \n# echo “Lens job completed.”<\/p>\n

    \n

    <\/a> End test.qsub<\/h4>\n
    \n

    <\/a> Begin server.qsub.tcl<\/h4>\n
    \n

    # This script is run on the server with the following (assuming “excecutable” below is set to “lens”)<\/span>
    \n#\u00a0<\/span>.\/lens -b server.tcl > server.out &<\/span><\/p>\n

    # LD_LIBRARY_PATH on both server and client machines must include current directory<\/span>
    \n# (containing libtcl8.3.so and libtk8.3.so)<\/span><\/p>\n

    # The base of the network file name (and its directory)<\/span>
    \nset filename\u00a0<\/span>rand100<\/span>
    \nset workingDirectory \/data2\/plautlab\/Lens<\/span>
    \nset networkScript\u00a0<\/span>$workingDirectory\/$filename.in<\/span>
    \nset clientScript\u00a0<\/span>client.tcl<\/span>
    \nset fixedPort\u00a0<\/span>2001<\/span>
    \nset executable\u00a0<\/span>$workingDirectory\/lens<\/span><\/p>\n

    # Starting epoch, total number to run, and learning algorithm<\/span>
    \nset epoch\u00a0<\/span>0<\/span>
    \nset nepochs\u00a0<\/span>100<\/span>
    \nset algorithm\u00a0<\/span>dougsMomentum<\/span><\/p>\n

    # number of epochs to run only steepestDescent before switching to the specified algorithm<\/span>
    \nset nsteepest\u00a0<\/span>0<\/span><\/p>\n

    # Checkpointing of weight files<\/span>
    \nset checkpointInterval 1000<\/span>
    \nset minSaveInterval\u00a0<\/span>100<\/span>
    \nset maxSaveInterval\u00a0<\/span>100<\/span><\/p>\n

    # Here is where you list the client machines. <\/span>
    \n# To run two or more client processes on the same machine, just list it multiple times.<\/span>
    \n#set clientMachines\u00a0<\/span>{<\/span>
    \n#compute-0-10 compute-0-10<\/span>
    \n#}<\/span>
    \nset clientMachines [exec $workingDirectory\/tailnodelist $::env(PBS_NODEFILE)]<\/span>
    \n#echo “List of client machines:”<\/span>
    \n#echo $clientMachines<\/span><\/p>\n

    #############################################################################<\/span>
    \n# shouldn’t need to change anything below this<\/span>
    \n#############################################################################<\/span><\/p>\n

    proc sourceIfExists {file} {<\/span>
    \n if { [file exists $file] } {<\/span>
    \n\u00a0<\/span>puts “Reading parameter file $file”<\/span>
    \n\u00a0<\/span>puts [source $file]<\/span>
    \n }<\/span>
    \n}<\/span><\/p>\n

    proc checkpoint { filename } {<\/span>
    \n\u00a0<\/span>global checkpointInterval<\/span>
    \n\u00a0<\/span>global minSaveInterval<\/span>
    \n\u00a0<\/span>global maxSaveInterval<\/span>
    \n\u00a0<\/span>set epoch [getObj totalUpdates] <\/span>
    \n\u00a0<\/span>set saveInterval [expr int(pow(10,floor(log10($epoch))))]<\/span>
    \n\u00a0<\/span>if { $saveInterval < $minSaveInterval } {<\/span>
    \n\u00a0<\/span>set saveInterval $minSaveInterval<\/span>
    \n\u00a0<\/span>} elseif { $saveInterval > $maxSaveInterval } {<\/span>
    \n\u00a0<\/span>set saveInterval $maxSaveInterval<\/span>
    \n\u00a0<\/span>}<\/span>
    \n\u00a0<\/span>if { [expr $epoch % $saveInterval] = 0 } {<\/span>
    \n\u00a0<\/span>puts “Saving weights to $filename.$epoch.wt.bz2”
    \n\u00a0<\/span>saveWeights $filename.$epoch.wt.bz2 -values 3
    \n\u00a0<\/span>} elseif { [expr $epoch % $checkpointInterval] = 0 } {<\/span>
    \n\u00a0<\/span>puts “Checkpointing to $filename.ckp.wt.bz2”<\/span>
    \n\u00a0<\/span>saveWeights $filename.ckp.wt.bz2 -values 3<\/span>
    \n\u00a0<\/span>}<\/span>
    \n\u00a0<\/span>sourceIfExists $filename.prm<\/span>
    \n\u00a0<\/span>sourceIfExists $filename.$epoch.prm<\/span>
    \n}<\/span><\/p>\n

    # Start the server<\/span>
    \nset port [startServer $fixedPort]<\/span>
    \nset hostname [exec hostname -f]<\/span><\/p>\n

    # Write the customized client script<\/span>
    \nset customClientScript $workingDirectory\/client[getSeed].tcl<\/span>
    \nregsub -all \/ $networkScript \\\\\/ netScript<\/span>
    \nsed “s\/SCRIPT\/$netScript\/; s\/SERVER\/$hostname\/; s\/PORT\/$port\/” \\<\/span>
    \n\u00a0<\/span>$clientScript > $customClientScript<\/span>
    \necho “Copying script to clients”<\/span>
    \nforeach client $clientMachines {<\/span>
    \n echo “$client”<\/span>
    \n scp $customClientScript $client:$workingDirectory<\/span>
    \n}<\/span><\/p>\n

    # Here we define a command for launching clients using ssh<\/span>
    \nproc launchClients {executable customClientScript machines} {<\/span>
    \n\u00a0<\/span>global env<\/span>
    \n\u00a0<\/span>set i 0<\/span>
    \n\u00a0<\/span>foreach client $machines {<\/span>
    \n\u00a0<\/span>puts ” launching on $client…”<\/span>
    \n\u00a0<\/span>exec \/usr\/bin\/ssh $client -n \\<\/span>
    \n\u00a0<\/span>“$executable -batch $customClientScript > \/dev\/null” &<\/span>
    \n\u00a0<\/span>incr i<\/span>
    \n\u00a0<\/span>}<\/span>
    \n\u00a0<\/span>return $i<\/span>
    \n}<\/span><\/p>\n

    # Now we use the command<\/span>
    \nputs “Launching clients…”<\/span>
    \nset numClients [launchClients $executable $customClientScript $clientMachines]<\/span><\/p>\n

    # Load the network and training set<\/span>
    \nsource $networkScript<\/span>
    \nsetObj postUpdateProc { checkpoint $filename }<\/span><\/p>\n

    # maybe load previously saved weights (to restart)<\/span>
    \nif { [file exists wts\/$filename.$epoch.wt.bz2] } {<\/span>
    \n\u00a0<\/span>loadWeights wts\/$filename.$epoch.wt<\/span>
    \n\u00a0<\/span>echo loadWeights wts\/$filename.$epoch.wt<\/span>
    \n} elseif { [file exists $filename.$epoch.wt.bz2] } {<\/span>
    \n\u00a0<\/span>loadWeights $filename.$epoch.wt<\/span>
    \n\u00a0<\/span>echo loadWeights $filename.$epoch.wt<\/span>
    \n}<\/span>
    \nset epoch [getObj totalUpdates] <\/span><\/p>\n

    # Now wait for the clients to connect.<\/span>
    \nputs “Waiting for $numClients clients…”<\/span>
    \nwaitForClients $numClients<\/span><\/p>\n

    # Start training and wait for it to finish,<\/span>
    \n# but don’t wait if it didn’t start correctly.<\/span>
    \nputs “Training…”<\/span><\/p>\n

    if { $epoch < $nsteepest } {<\/span>
    \n\u00a0<\/span>puts [trainParallel [expr $nsteepest – $epoch] -algorithm steepest ]<\/span>
    \n\u00a0<\/span>set epoch $nsteepest<\/span>
    \n}<\/span>
    \nputs [trainParallel [expr $nepochs – $epoch] -algorithm $algorithm ]<\/span><\/p>\n

    # Now break the barrier holding the clients so they can exit.<\/span>
    \nputs “Releasing clients…”<\/span>
    \nwaitForClients<\/span>
    \nexec rm [glob $customClientScript]<\/span>
    \nputs “Ba-bye”<\/span>
    \nexit<\/span><\/p>\n

    \n

    <\/a> End server.qsub.tcl<\/h4>\n","protected":false},"excerpt":{"rendered":"

    This needs updated for the SLURM scheduler! A sample PBS submit script can be found below. There are a few parts. Firstly, you will need to set the following environment variables in your .bashrc file located in your home directory. export LENSDIR=\/data2\/plautlab\/Lens export HOSTTYPE=x86_64 export PATH=$PATH:$LENSDIR\/Bin\/$HOSTTYPE export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$LENSDIR\/Bin\/${HOSTTYPE} export TCL_LIBRARY=\/data2\/plautlab\/Lens\/Bin\/x86_64\/…<\/p>\n","protected":false},"author":1,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"footnotes":""},"ht-kb-category":[11],"ht-kb-tag":[],"class_list":["post-100","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-software"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/100","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/comments?post=100"}],"version-history":[{"count":2,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/100\/revisions"}],"predecessor-version":[{"id":102,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/100\/revisions\/102"}],"wp:attachment":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/media?parent=100"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-category?post=100"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-tag?post=100"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}