Using SLURM job arrays for large job batches

What is a job array?

A job array lets you submit one job that spawns many sub-tasks, instead of submitting each task as a separate job. SLURM treats the whole array as a single submission internally, so it places far less load on the scheduler than the equivalent number of individual sbatch calls — even though the same number of tasks eventually run.

Before: a submission loop (avoid this)

If your workflow looks like this, it’s exactly the pattern that causes scheduler slowdowns:

for i in {1..1000}; do
  sbatch -p cpu run_task.sh $i
done

This submits 1,000 separate jobs, each a separate request to the controller.

After: job arrays on CPU vs. GPU queues

Replace the loop with a single script and an --array directive. Select the appropriate queue with --partition (e.g., cpu or gpu).

Example 1: CPU Queue Job Array

#!/bin/bash
#SBATCH --job-name=cpu_array
#SBATCH --partition=cpu
#SBATCH --array=1-1000%20
#SBATCH --cpus-per-task=2
#SBATCH --mem=4G
#SBATCH --time=01:00:00
#SBATCH --output=logs/cpu_task_%A_%a.out

# Ensure the log directory exists before running
mkdir -p logs

# $SLURM_ARRAY_TASK_ID holds the index for this sub-task (1, 2, 3, ... 1000)
./run_cpu_task.sh $SLURM_ARRAY_TASK_ID

Example 2: GPU Queue Job Array

#!/bin/bash
#SBATCH --job-name=gpu_array
#SBATCH --partition=gpu
#SBATCH --gres=gpu:1
#SBATCH --array=1-100%5
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=02:00:00
#SBATCH --output=logs/gpu_task_%A_%a.out

mkdir -p logs

# $SLURM_ARRAY_TASK_ID selects the dataset chunk or model parameters
python train_model.py --batch-id $SLURM_ARRAY_TASK_ID

Submit either script once using sbatch:

sbatch run_cpu_array.sh
sbatch run_gpu_array.sh

Useful Array Variables

SLURM automatically sets environment variables for every sub-task inside an array:

Re-running failed sub-tasks

If specific sub-tasks fail (for instance, tasks 4, 12, and 88 encounter a network timeout), you do not need to resubmit the entire range. Modify the --array flag or pass it directly on the command line to target only those task IDs:

# Re-run specific non-sequential failed tasks
sbatch --array=4,12,88 run_cpu_array.sh

# Re-run a sub-range of failed tasks
sbatch --array=100-105 run_cpu_array.sh

Useful commands

Questions

If your workflow doesn’t fit the array pattern, or you’re not sure how to convert an existing script, contact David and he will do his best to help you set it up.