Abaqus

Introduction

Abaqus delivers a scalable suite of unified analysis products. The Abaqus Unified FEA product suite offers powerful and complete solutions for both routine and sophisticated engineering problems covering a vast spectrum of industrial applications.

Availability

Abaqus is available on several ARC systems. Virginia Tech maintains a limited quantity of Abaqus network licenses through the university’s IT Procurement and Licensing Solutions.

License availability

You can query the status of licensing with the command abaqus licensing lmdiag -n. The -n option specifies “non-interactive” mode of output which dumps all the license component statuses at once.

abaqus licensing lmstat -S ABAQUSLM

Interface

The primary interface for using Abaqus on ARC resources is via batch scripts and using the command line. For example on Tinkercliffs, you can get detailed command line help for using Abaqus like this:

This shows an exploratory Abaqus command running on a login node, but most computations should be submitted as jobs via a [traditional job submission](slurm).

[mypid@tinkercliffs1 ~]$ module load ABAQUS/2018
[mypid@tinkercliffs1 ~]$ abaqus -help 
In the following execution procedures, "abaqus" refers to the command
used to run Abaqus.

 Obtaining information

 abaqus  {help | information={environment | local | memory | release |
           support | system | all} [job=job-name] | whereami}


 Abaqus/Standard and Abaqus/Explicit execution

 abaqus     job=job-name
            [analysis | datacheck | parametercheck | continue |
              convert={select | odb | state | all} | recover |
              syntaxcheck | information={environment | local |
              memory | release | support | system | all}]
...

Tip

In your job script, you will need to unset the SLURM_GTIDS environment variable due to avoid errors like

<IBM Platform MPI>: : warning, dlopen of libhwloc.so failed (null)/lib/linux_amd64/libhwloc.so: cannot open shared object file: No such file or directory 
Traceback (most recent call last): 
Exception: can't parse host/port from umbrella

Add this line to your job script before launching abaqus:

unset SLURM_GTIDS

This seems to be due to a conflict between Slurm and the MPI Abaqus uses by default.

Parallel Computing with Abaqus

There are two primary modes of obtaining parallelism in Abaqus as specfied with the optional mp_mode={ mpi | threads } setting.

Threads: Shared Memory Parallel. Execution is limited to a single node since the threads require shared access to the same memory space.
MPI: Message Passing Parallel. Several or many processes are launched and run as if each is on its own computer with dedicated memory. The discretization of the domain is divided equally (more or less) between the processes (ie. “domain decomposition”) and each process is carries out the simulation on its subdomain. Neighboring subdomains affect each other, so processes must pass messages (MPI) to share the necessary data. This mode can scale to a large number of processors across many machines, but the overhead of subdividing the domain and passing messages becomes significant.

The default mode of parallelism is “MPI” and Abaqus can be instructed how many processes to launch with the option cpus=XX. It is often convenient to write job scripts which automatically adapt to the Slurm resource allocation size using environment variables provided by the job. For example:

#!/bin/bash
# Filename: abaqusJob.sh
# Run Abaqus analysis on 1 node with 12 cores for up to 4 hours
#SBATCH --account=<your Slurm account>
#SBATCH --partition=normal_q
#SBATCH --time=4:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=12
#SBATCH --cpus-per-task=1
module reset
module load ABAQUS/2018
cd $SLURM_SUBMIT_DIR
unset SLURM_GTIDS
abaqus job=<jobname> analysis cpus=$SLURM_NTASKS interactive

To submit this job, use the sbatch command and specify the filename of the script. For example

sbatch abaqusJob.sh

When resources become available, the script will run on the allocated node and record stdout and stderr to an output file. You can monitor this file for the status of the simulation and can also query Slurm for job status:

squeue -u <username>

These online manuals have much more information about Abaqus usage.

Running ABAQUS with GPU:

To utilize ABAQUS with GPU acceleration, follow these steps:

Create a file named abaqus_v6.env in your working directory and add the following content to it:

import os
os.environ["ABA_ACCELERATOR_TYPE"]="PLATFORM_CUDA"

Modify your batch script to include CUDA and specify the number of GPUs required using the gpus=N parameter, where N is the number of GPUs you are requesting.

Note: In most cases, one GPU is enough. One GPU consumes one licensing token.

Example:

#!/bin/bash
#SBATCH -J ABAQUS-GPU
#SBATCH -N 1
#SBATCH --ntasks-per-node=12
#SBATCH --cpus-per-task=1
#SBATCH --gres=gpu:1
#SBATCH -t 24:00:00
#SBATCH -p dgx_normal_q
#SBATCH -A <your-allocation>

module reset
module load CUDA/12.0.0
module load ABAQUS/2022
unset SLURM_GTIDS
abaqus job=test_job cpus=$SLURM_NTASKS gpus=1 interactive

Note: Use either -p dgx_normal_q or -p a100_normal_q as GPUs are only accessible in these partitions. Note that ABAQUS versions differ between the two partitions (a100_normal_q has ABAQUS/2018 and dgx_normal_q has ABAQUS/2022).

Fetch and Run an Example or Benchmark Problem

Abaqus installations come with a large number of example and benchmark jobs. They can be accessed using the fetch command if you know the jobname or filename you’re interested in:

abaqus fetch job=<jobname>

For example:

abaqus fetch job=rotating_drum_mixer