StarCCM+

Specifying nodes and processes per node on Slurm systems

This is in a job which had “nodes=3” and “ntasks-per-node=2”:

[brownm12@ca028 ~]$ STARNL=$(srun hostname | sort -u | awk -v NT=$SLURM_NTASKS_PER_NODE '{printf $1":"NT","}' | sed s/.$//)
[brownm12@ca028 ~]$ echo $STARNL
ca028:2,ca099:2,ca100:2

Breakdown:

  • STARNL=$( ... ) set bash variable STARNL to the output of the command enclosed in parentheses:

  • cmd1 | cmd2 | cmd3 send output of cmd1 as input to cmd2 and the resulting output of that to cmd3 (pipes)

  • srun hostname run SLURM_NTASKS instances of hostname command, one on each task (process) allocated to the job

  • sort -u sort the input and remove duplicate lines

  • awk -v NT=$SLURM_NTASKS_PER_NODE '{ ... }' run an awk script on the input, additionaly set the awk variable NT to a value

  • printf $1":"NT"," awk script to print the first column of each line of input, followed by a colon, the value stored in NT, and then a comma

  • sed s/.$// remove the last character of the input line using sed’s “swap” capability

Example usage in a script

So in your job script, prior to the starccm+ command, you would have:

STARNL=$(srun hostname | sort -u | awk -v NT=$SLURM_NTASKS_PER_NODE '{printf $1":"NT","}' | sed s/.$//)
echo "Using nodelist: `echo $STARNL`"

So an example job script would have:

#!/bin/bash
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=6

module load starccm+
# Get hostlist and number of processes for each host and save in a variable:
STARNL=$(srun hostname | sort -u | awk -v NT=$SLURM_NTASKS_PER_NODE '{printf $1":"NT","}' | sed s/.$//)
# Print value of the variable for diagnostics
echo "Using nodelist: `echo $STARNL`"
starccm+ <your options and settings> -np $SLURM_NTASKS -on $STARNL <simfile>