Common mistakes to avoid

Execution environment

1. Do not use module purge. It removes too much and will break Slurm and other cluster functionality. Use module reset to remove all loaded modules and rebase to the default set for the current context.

2. Avoid using a /projects directory as your working directory. /projects is for bulk storage of shared data. /scratch and TMPDIR are for staging and running jobs. The separation helps prevent accidental deletion, data corruption, and provides better performance.

Misaligned resource selection

1. Do not request --mem=2GB on Tinkercliffs normal_q or --mem=8GB on Owl normal_q. Some node memory is reserved for the OS and Slurm, so the proportionate share of memory available per core is slightly less than these amounts. Without setting any value for --mem, you will already get the correct proportionate share of memory (e.g., 1944MB on TC).

Scaling

1. Profile the workload with a representative example as a benchmark before scaling.

2. Right-size the resource request (cpus, memory, gpus, nodes).

3. Maximize the utilization and parallelism inside a single job before scaling to job arrays.