{"id":752,"date":"2026-08-20T19:15:15","date_gmt":"2026-08-20T19:15:15","guid":{"rendered":"https:\/\/ni.cmu.edu\/computing\/?post_type=ht_kb&p=752"},"modified":"2026-08-20T19:33:11","modified_gmt":"2026-08-20T19:33:11","slug":"using-slurm-job-arrays-for-large-job-batches","status":"publish","type":"ht_kb","link":"https:\/\/ni.cmu.edu\/computing\/knowledge-base\/using-slurm-job-arrays-for-large-job-batches\/","title":{"rendered":"Using SLURM job arrays for large job batches"},"content":{"rendered":"
\n
Why this matters:<\/strong> Submitting thousands of individual jobs (for example, a script that calls sbatch<\/code> in a loop) floods the SLURM controller with requests, slows down login node response times for everyone, and delays your own job’s placement. Job arrays fix this and are the recommended way to submit large batches of similar jobs.<\/div>\n

What is a job array?<\/h2>\n

A job array lets you submit one job that spawns many sub-tasks, instead of submitting each task as a separate job. SLURM treats the whole array as a single submission internally, so it places far less load on the scheduler than the equivalent number of individual sbatch<\/code> calls \u2014 even though the same number of tasks eventually run.<\/p>\n

Before: a submission loop (avoid this)<\/h2>\n

If your workflow looks like this, it’s exactly the pattern that causes scheduler slowdowns:<\/p>\n

for i in {1..1000}; do\r\n  sbatch -p cpu run_task.sh $i\r\ndone<\/pre>\n
This submits 1,000 separate jobs, each a separate request to the controller.<\/p>\n
After: job arrays on CPU vs. GPU queues<\/h2>\n
Replace the loop with a single script and an --array<\/code> directive. Select the appropriate queue with --partition<\/code> (e.g., cpu<\/code> or gpu<\/code>).<\/p>\n
Example 1: CPU Queue Job Array<\/h3>\n
#!\/bin\/bash\r\n#SBATCH --job-name=cpu_array\r\n#SBATCH --partition=cpu\r\n#SBATCH --array=1-1000%20\r\n#SBATCH --cpus-per-task=2\r\n#SBATCH --mem=4G\r\n#SBATCH --time=01:00:00\r\n#SBATCH --output=logs\/cpu_task_%A_%a.out\r\n\r\n# Ensure the log directory exists before running\r\nmkdir -p logs\r\n\r\n# $SLURM_ARRAY_TASK_ID holds the index for this sub-task (1, 2, 3, ... 1000)\r\n.\/run_cpu_task.sh $SLURM_ARRAY_TASK_ID<\/pre>\n
Example 2: GPU Queue Job Array<\/h3>\n
#!\/bin\/bash\r\n#SBATCH --job-name=gpu_array\r\n#SBATCH --partition=gpu\r\n#SBATCH --gres=gpu:1\r\n#SBATCH --array=1-100%5\r\n#SBATCH --cpus-per-task=4\r\n#SBATCH --mem=16G\r\n#SBATCH --time=02:00:00\r\n#SBATCH --output=logs\/gpu_task_%A_%a.out\r\n\r\nmkdir -p logs\r\n\r\n# $SLURM_ARRAY_TASK_ID selects the dataset chunk or model parameters\r\npython train_model.py --batch-id $SLURM_ARRAY_TASK_ID<\/pre>\n
Submit either script once using sbatch<\/code>:<\/p>\n
sbatch run_cpu_array.sh\r\nsbatch run_gpu_array.sh<\/pre>\n
Useful Array Variables<\/h2>\n
SLURM automatically sets environment variables for every sub-task inside an array:<\/p>\n\n\n\n\n\n\n
Variable<\/th>\nDescription<\/th>\nExample Value<\/th>\n<\/tr>\n
$SLURM_ARRAY_JOB_ID<\/code><\/td>\nMaster job ID assigned to the whole array<\/td>\n12345<\/code> (or %A<\/code> in output)<\/td>\n<\/tr>\n
$SLURM_ARRAY_TASK_ID<\/code><\/td>\nIndex number of the specific sub-task<\/td>\n42<\/code> (or %a<\/code> in output)<\/td>\n<\/tr>\n
$SLURM_JOB_ID<\/code><\/td>\nUnique ID for the individual running task<\/td>\n12345_42<\/code><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Re-running failed sub-tasks<\/h2>\n
If specific sub-tasks fail (for instance, tasks 4, 12, and 88 encounter a network timeout), you do not need to resubmit the entire range. Modify the --array<\/code> flag or pass it directly on the command line to target only those task IDs:<\/p>\n
# Re-run specific non-sequential failed tasks\r\nsbatch --array=4,12,88 run_cpu_array.sh\r\n\r\n# Re-run a sub-range of failed tasks\r\nsbatch --array=100-105 run_cpu_array.sh<\/pre>\n
Useful commands<\/h2>\n\n\n\n\n\n\n\n
Command<\/th>\nWhat it does<\/th>\n<\/tr>\n
squeue -u $USER<\/code><\/td>\nShows the status of your array and its sub-tasks across CPU\/GPU queues<\/td>\n<\/tr>\n
scancel <jobid><\/code><\/td>\nCancels the entire array<\/td>\n<\/tr>\n
scancel <jobid>_<taskid><\/code><\/td>\nCancels a single sub-task (e.g., scancel 12345_12<\/code>)<\/td>\n<\/tr>\n
sacct -j <jobid> --format=JobID,JobName,State,ExitCode<\/code><\/td>\nChecks exit codes for all sub-tasks to quickly find failed runs<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Questions<\/h2>\n
If your workflow doesn’t fit the array pattern, or you’re not sure how to convert an existing script, contact David and he will do his best to help you set it up.<\/p>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Why this matters: Submitting thousands of individual jobs (for example, a script that calls sbatch in a loop) floods the SLURM controller with requests, slows down login node response times for everyone, and delays your own job’s placement. Job arrays fix this and are the recommended way to submit large…<\/p>\n","protected":false},"author":1,"comment_status":"closed","ping_status":"closed","template":"","format":"standard","meta":{"footnotes":""},"ht-kb-category":[7],"ht-kb-tag":[],"class_list":["post-752","ht_kb","type-ht_kb","status-publish","format-standard","hentry","ht_kb_category-cluster"],"jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/752","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb"}],"about":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/types\/ht_kb"}],"author":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/comments?post=752"}],"version-history":[{"count":6,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/752\/revisions"}],"predecessor-version":[{"id":758,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb\/752\/revisions\/758"}],"wp:attachment":[{"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/media?parent=752"}],"wp:term":[{"taxonomy":"ht_kb_category","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-category?post=752"},{"taxonomy":"ht_kb_tag","embeddable":true,"href":"https:\/\/ni.cmu.edu\/computing\/wp-json\/wp\/v2\/ht-kb-tag?post=752"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}