Duke Research Computing - DCCUserGuide (2024)

tm103@dscr-slogin-02 ~ $ srun --pty bash -i srun: job 186535 queued and waiting for resources srun: job 186535 has been allocated resources tm103@dcc-core-11 ~ $

tm103@dcc-core-11 ~ $ squeue -u tm103 JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 186535 common bash tm103 R 0:14 1 dcc-core-11 

#SBATCH --mem=1G#SBATCH --output=matlab.out

hostname 1>&2 #prints hostname to the error file• This job will run in low priority on a lab node in the “scavenger” partitionShort-form commands exampleSLURM short commands don’t use “=“ signs

hostname 1>&2 #prints hostname to the error fileR example script

This loads the environment module for R/3.6.0 and runs a single R script (“Rcode.R”)The “#SBATCH --mem=4G ” requests additional RAMSlurm memory directivesThis is a hard limit – always request a little more`--mem=<MB>`The amount of memory required per node` --mem-per-cpu=<MB>`The amount of memory per CPU core For multi-threaded jobsNote: --mem and --mem-per-cpu are mutually exclusiveSlurm parallel directivesAll parallel directives have defaults of 1 -N <number> How many nodes (machines)-n <number> or --ntasks=<number> How many parallel jobs (“tasks”)-c, --cpus-per-task=<ncpus>Use -c for multi-threaded jobsThe --ntasks default is one CPU core per task, but the --cpus-per-task option will change this default.Multi-threaded (multi-core) example~~~!/bin/bash#SBATCH –J test #SBATCH –o test.out#SBATCH –c 4#SBATCH –-mem-per-cpu=500 #(500 MB) myApplication –n $SLURM_CPUS_PER_TASK~~~The value of $SLURM_CPUS_PER_TASK is the number after “-c”This example starts a single, multi-threaded job that uses 4 CPU cores and 2 GB (4x500MB) of RAMOpenMP multicore example!/bin/bash#SBATCH –J openmp-test #SBATCH –o slurm.out#SBATCH –c 4export OMP_NUM_THREADS=$SLURM_CPUS_PER_TASKmyOpenMPapp # will run on 4 CPU coresThis sets $OMP_NUM_THREADS to the value of $SLURM_CPUS_PER_TASKSlurm job arraysWhat are job arrays?--array (or -a) optionSLURM_ARRAY_TASK_ID environment variableExample with sequentially named filesExamples with non-sequentially named files“Unrolling” existing for loops Slurm job arraysa mechanism for submitting and managing collections of similar jobsone job script and one application programEach can be up 100,000 job tasks on the DCCJob arrays are only supported for batch jobs Job array “tasks” must be independent http://slurm.schedmd.com/job_array.html--array (or –a)For example, in a job script, add the line'#SBATCH --array=1-30or, alternatively,'#SBATCH -a 1-30to submit 30 job tasks. The job array indices can also be specified on the command line, e.g.sbatch -a 1-30 myjob.shArray indices, cont.The index values can be continuous, e.g.-a 0-31 (32 tasks, numbered from 0,1,2,...,31)or discontinuous, e.g.-a 3,5,7-9,12 (6 tasks, numbers 3,5,7,8,9,12)It can also be a single job task, e.g.-a 7The discontinous notation is useful for resubmitting specific job tasks that had previously failed$SLURM_ARRAY_TASK_IDEach job task is assigned the enviromental variable $SLURM_ARRAY_TASK_ID set to it’s index value.tm103@dcc-slogin-02 ~/misc/jobarrays $ cat array-test.sh #!/bin/bashecho $SLURM_ARRAY_TASK_IDtm103@dcc-slogin-02 ~/misc/jobarrays $ sbatch -a 1-3 array-test.sh Submitted batch job 24845830tm103@dcc-slogin-02 ~/misc/jobarrays $ ls slurm-24845830*slurm-24845830_1.out slurm-24845830_2.out slurm-24845830_3.outtm103@dcc-slogin-02 ~/misc/jobarrays $ cat slurm-24845830*123tm103@dcc-slogin-02 ~/misc/jobarrays $Python job array example'#!/bin/bash'#SBATCH -e slurm_%A_%a.err'#SBATCH -o slurm_%A_%a.out '#SBATCH --array=1-5000 python myCode.py$ cat test.pyimport ostaskID=int(os.environ['SLURM_ARRAY_TASK_ID'])...Start 5000 Python jobs, each with a different “taskID”, initialized from SLURM_ARRAY_TASK_IDImporting Environmental VariablesPython:numCPUs=int(os.environ['SLURM_CPUS_PER_TASK'])taskID=int(os.environ['SLURM_ARRAY_TASK_ID'])R:numCPUs <- as.integer(Sys.getenv('SLURM_CPUS_PER_TASK')) taskID <- as.integer(Sys.getenv('SLURM_ARRAY_TASK_ID')) MATLAB:numCPUs = str2num(getenv('SLURM_CPUS_PER_TASK'))taskID = str2num(getenv('SLURM_ARRAY_TASK_ID'))Processing separate input filesProcess an existing file list, e.g. files.txt#!/bin/bashreadarray -t FILES < files.txtFILENAME=${FILES[(($SLURM_ARRAY_TASK_ID - 1))]}myapp $FILENAMEDynamically generate a file list from “ls”'#!/bin/bashexport FILES=($(ls -1 myfile*))FILENAME=${FILES[(($SLURM_ARRAY_TASK_ID - 1))]}myapp $FILENAMEExample: Using the taskID as part of the file name and output directoryFor the case with input file names of the form input1,input2,...,inputN for -a 1-N, e.g.'#!/bin/bash'#SBATCH -e slurm_%A_%a.err'#SBATCH -o slurm_%A_%a.out mkdir out_${SLURM_ARRAY_TASK_ID} cd out_${SLURM_ARRAY_TASK_ID}myapp ../input_${SLURM_ARRAY_TASK_ID}.txtwhere output directories out_1, out_2, ... are created for input files input_1.txt, input_2.txt,...“Unrolling” for loopsOriginal “serial” code (Python)fibonacci = [0,1,1,2,3,5,8,13,21]for i in range(len(fibonacci)): print(i,fibonacci[i])Job array versionimport osi=int(os.environ['SLURM_ARRAY_TASK_ID'])fibonacci = [0,1,1,2,3,5,8,13,21]#for i in range(len(fibonacci)): print(i,fibonacci[i])where the for loop is commented out and each job task is doing a single “iteration”“Unrolling” for loop, cont.`tm103@dcc-slogin-02 ~/misc/jobarrays $ cat fib-array.sh '#!/bin/bash'#SBATCH -e slurm.errmodule load Python/2.7.11python fibonacci.pytm103@dcc-slogin-02 ~/misc/jobarrays $ sbatch –a 1-8 fib-array.sh Submitted batch job 24856052tm103@dcc-slogin-02 ~/misc/jobarrays $ ls slurm-24856052_*slurm-24856052_1.out slurm-24856052_3.out slurm-24856052_5.out slurm-24856052_7.outslurm-24856052_2.out slurm-24856052_4.out slurm-24856052_6.out slurm-24856052_8.outtm103@dcc-slogin-02 ~/misc/jobarrays $ cat slurm-24856052*(1, 1)(2, 1)(3, 2)(4, 3)(5, 5)(6, 8)(7, 13)(8, 21)`tm103@dcc-slogin-02 ~/misc/jobarrays $##Running MPI jobsSupported MPI versionsIntel MPIOpenMPISLURM MPI jobs use “--ntasks=(num)“https://wiki.duke.edu/display/SCSC/Running+MPI+JobsCompiling with OpenMPI`tm103@dcc-slogin-02 ~ $ module load OpenMPI/2.1.0###OpenMPI 2.1.0~~~tm103@dcc-slogin-02 ~ $ which mpicctm103@dcc-slogin-03 ~ $ cd /dscrhome/tm103/misc/slurm/openmpitm103@dcc-slogin-03 ~/misc/slurm/openmpi $ mpicc -o openhello hello.ctm103@dscr-slogin-02 ~/misc/slurm/openmpi ls -l openhello -rwxr-xr-x. 1 tm103 scsc 9184 Sep 1 16:08 openhello`

#SBATCH -p gpu-common --gres=gpu:1 #SBATCH -c 6