Monitoring Your Work on Compute Nodes

Monitoring means:

Identifying ALL of your Slurm jobs or a PARTICULAR Slurm job
Inspecting a PARTICULAR RUNNING slurm job by observing its characteristics and performance.

Types of Jobs That are Slurm Jobs

The following types of jobs are all considered Slurm Jobs:

Batch jobs where you submit an sbatch Slurm script to the Slurm scheduler.
Interactive jobs where you request resources on one or more compute nodes and when Slurm gives you these resources, you have free-reign to use them for the specified duration.
Open OnDemand (OOD) jobs.

Guidance on Monitoring Slurm Jobs

Every Slurm (and OOD) job, of different types, that you have running should be inspected each day three times—morning, noon, and night—to ensure that it is running properly.

If you have many running jobs of one type, then at least one of those running jobs should be inspected each day three times—morning, noon, and night—to ensure that it is running properly.

When any user wastes resources, everyone suffers.

We need to look out for ourselves and for each other by monitoring our jobs.

Identifying Your Slurm Jobs on a Cluster

Video Number	Video Name and Link	Description
1	How to identify slurm jobs (on compute nodes) via the command line	How to use Slurm commands `squeue` and `scontrol show job --details`.
2	How to identify Open OnDemand-based slurm jobs	For OOD jobs, how to get the slurm job ID.

From these videos, you now have, for each job of interest:

The Slurm Job ID for your job.
Confirmation that the job is running.
The compute node(s) on which your job is running.
The CPU IDs and GPU indexes for the CPUs and GPUs that your job is using.
Miscellaneous things about job submission:
- The submission command used.
- The path and name of the sbatch slurm script.
- The the working directory.

You will need one or more of these quantities in the procedures in the section below.

Inspecting Your Running Slurm Jobs on a Cluster

In the previous subsection, you can identify queued, running, and if you issue the command just right, you can even see when the process of shutting down a job occurs. But this shut down process happens very quickly. Here, in this subsection, we are interested in running processes and observing their behavior.

In the previous subsection, there is a list of parameters that should be recorded for each job that you want to inspect. You will need this information in the videos below.

You can view these videos in any order, but some of them build on earlier videos, and although each video is standalone, you may get more out of later videos if you have watched the previous ones.

Video Number	Video Name and Link	Description
1	How to show details of your running job	How to use the command `showjobusage`. For higher level CPU and GPU instantaneous utilization.
2	How to show all running jobs on a compute node	How to use the command `shownodeusage`. For higher level CPU and GPU instantaneous utilization.
3	How to use Grafana dashboards to get running job details	How to use Grafana to inspect a particular job. Particularly for detailed GPU utilization.
4	How to directly zero-in on your job within Grafana dashboards	How to use `getjobutilurl`. Particularly for detailed GPU utilization.
5	How to ssh onto a compute node that you are running on and inspect job	How to use `ssh`, `scontrol show job --details`, and both `top` and `htop`. Particularly for detailed CPU utilization.
6	How to observe CPU resource utilization on compute nodes	How to use `ssh` and `ps` to inspect CPU utilization of individual processes.
7	TODO: How to observe Nvidia GPU resource utilization on compute nodes	How to use `ssh` and `nvidia-smi` to inspect GPU utilization of individual processes.
8	TODO: How to observe AMD GPU resource utilization on compute nodes	How to use `ssh` and `rocm-smi` to inspect GPU utilization of individual processes.