Your Responsibilities For Running Jobs on Compute Nodes

We address here a universal problem: wasting of resources. It is a problem across universities (VT is not immune to it). It is well-known and discussed among cluster administrators across universities.

Big picture view:

1. Significant resources get wasted daily on ARC clusters.

2. Everyone should try to eliminate this waste.

3. It is every user’s responsibility to do so.

4. Every user benefits from the diligence of each user inspecting their jobs through the job’s life.

   • This means at the job’s start and every few hours while the job is running. Not while you are asleep, of course.

5. The effort required to inspect your jobs is quite minimal. While the instructions below may seem long, once you do them a couple of times, they will become very natural and very fast.

A Sense of the Issue

To build a sense of community and that all users are “in the same boat,” see this Waiting Jobs Dashboard that shows jobs that are queued and are awaiting resources to run. What you see will vary with time, but across all clusters, it is not uncommon to have some 1200 to 1300 jobs waiting to run on GPU nodes and 1700-2000 jobs waiting to run on CPU nodes. There is a lot of demand.

By following these procedures, we can reduce the numbers of waiting jobs and reduce their wait times.

Overview: Common Problems Leading to Resource Wastage

1. For batch jobs: A. Requesting a GPU node and then not using the GPU.

   • Because once GPU resources are allocated to you, no one else can use them, even if you do not use them. B. Requesting more CPUs than your job requires.

   • Because once CPU resources are allocated to you, no one else can use them, even if you do not use them. C. Requesting more memory than your job requires.

   • Because once memory resources are allocated to you, no one else can use them, even if you do not use them.

2. For interactive jobs (both from the command line and through OOD [Open OnDemand]): A. Not inspecting your interactive job requests to see whether and when they are fulfilled, i.e., when Slurm provide you the resources that you can start using.

   • Because once resources are allocated to you, no one else can use them, even if you do not use them. B. Not giving resources back when you are done with them.

   • Because once resources are allocated to you, the two ways that resources are re-couped by Slurm and can then be used by others are:

     1. Your interactive job time limit expires.

     2. You give back resources when you are done.

   • These two cases mean, for example, if you request resources for two days, and finish your work in seven hours, then if you do (2), the resources can be used by someone else immediately after seven hours. But if instead you leave the terminal and do not give back the resources, then for 48 hours only you can use the resources, meaning that 41 hours of those 48 hours are completely wasted.

3. For both batch and interactive jobs: A. Not looking at resource utilization while your job is running to confirm that your job is running well.

Your Batch Job Responsibilities

Dashboards

The Grafana dashboard is here:

From there, you can select dashboards from the list.

• Use grafana dashboards

  • Tinkercliffs

  • Owl

  • Falcon

• To see first-hand how we are all in the same boat, look at the dashboard that shows how many jobs are queued and waiting to run, by (cluster, partition) pairs.

Your End-of-Job Responsibilities

Given a slurm job ID of SJI, type

seff SJI

to see CPU and CPU-based memory usage aggregated over your job.

This enables you to build up intuition about the resources that your jobs need.

We are not saying that you should cut resources down to the bone, but if, say, you use 20 CPUs and from seff your CPU utilization is 20%, then you should consider whether you can decrease the number of CPUs for the job and increase the utilization of those.