A high-performance computing cluster is a finite resource that must be shared among many users. To run some computational fluid dynamics simulations on a GPU, we must wait in a queue behind many other researchers. However, when setting up a new simulation, there will inevitably be misconfigurations and errors.

To be productive, we need to

1. Make sure we run the correct simulations, i.e. with the parameters we intend; and
2. Be able to quickly debug when things go wrong, in a fast iterative loop.

I have written about getting simulations done efficiently before, but this post adds a few more ideas to address these two aims and increase productivity. The suggestions might be useful to anyone running long simulations.

Number runs sequentially

It is not possible to encode all the metadata that might be relevant in a folder name, so you should not try. Just stick to a 0001 style identifier number, and use a spreadsheet to record the key settings for each run. More importantly, a spreadsheet provides ample space for remarks on what you are changing and why, what you hope to learn from each simulation, and the actual results. Never reuse a run number, but do clean up old ones that fail or are not needed any more.

Configuration files as a single source of truth

Managing the inherent complexity of a simulation that requires the specification of a lot of input data and choosing many modelling parameters is hard.

Bad options include a proliferation of command line arguments to the main script, sprinkling default values throughout the code, or, worst of all, commenting lines in and out to change behaviour!

I think the following workflow is quite elegant. First, we read a YAML config file containing some settings. The settings are passed as kwargs to initialise a dataclass. Then, in a fresh working directory created for this run, the data class is saved back to a new config file. Finally, in interactive mode, the simulation starts straight away; in batch mode, the new YAML file is passed as an argument to a queue job script.

The YAML–dataclass–YAML method allows us to:

• Quickly check in a single file which settings were used for a given run;
• Enforce types for each setting, and perform other validation in the dataclass.__post_init__ method;
• Specify default values in the code but record them next to the simulation results for future reference, in case they change later on;
• Generate documentation for all settings using automated tools like Sphinx;
• Use grep to search through many config files for runs with a certain setting;
• Neatly version control our settings for specific cases separately to the core general code;
• Run a modified version of a previous simulation by copying and changing the config file.

Backup the source

If we want reproduce an earlier result, it is not sufficient to have the config file — the preprocessing code may have changed too. Previously I have experimented with recording a git commit hash for every run, but this does not play well with iterative, exploratory programming. To be rigorous, each failed attempt to fix a problem will end up with its own git commit, with increasingly exasperated messages. A more pragmatic approach, with a lesser burden on day-to-day work, is to automatically store a compressed backup of the entire source code on every invocation. This may seem like overkill, but the file size is small compared to, say, three-dimensional flow field output data.

Dry run

All job scripts should take a dry-run config flag that will skip running any CFD, but still perform all pre- and post-processing. This is vital to be able to test the script from start to finish on a login node and check for errors before submitting to the queue.

Automated restarts

Large computational fluid dynamics simulations may take several days or weeks of wall time to converge, longer than the typical maximum queue job wall time of 36 hours. A long job must split into smaller identical jobs. It is worthwhile to write a job script that will automate this process. Given the working directory of an initial simulation, we only have to copy the old output file to an input file in a new working directory and repeat the initial job command. Calling the script recursively and setting a job dependency for subsequent steps allows an arbitrary number of restarts.

Holding a node after errors

There are few things more frustrating than watching your job sit in a queue for days, only for the job to throw an error and exit minutes after starting. If the problem is not caught by input validation or a dry run, and only appears at runtime, we will need to manually intervene and restart the job — ideally keeping and using the existing resources we have queued for.

My solution is to trap any errors and start a shell on the allocated compute node to allow me to examine the situation. I have settled on the below snippet that can be pasted at the top of a job submission script:

trap 'handle_error' ERR
handle_error() {
    echo "# Command failed, starting a shell on ${HOSTNAME}. Attach using:" > failed.sh
    echo "ssh -t $HOSTNAME tmux att" >> failed.sh
	# In case the job was submitted from a tmux session, unset $TMUX to prevent erroneous nesting warning
	export TMUX=""
	# Start a shell in a detached tmux session
    # Headless seems flaky
    # Fix this by redirecting to a file handle
    tmux new -d 'exec bash' &> /dev/null
    # Keep the job running until it times out
    sleep 36h
}

From a login node, we can jump into a shell on the failed compute node just by doing:

bash failed.txt

This shell inherits the same environment that the failed job had, so we can, for example, run MPI processes across all allocated nodes. Because the shell is inside the tmux terminal multiplexer, we can open further shell prompts as we like.

There are a couple of limitations to this approach. First, it assumes the tmux binary is on your $PATH; the script could, however, be modified to use GNU screen or even no terminal multiplexer at all. Second, we need to manually check if a job has failed and fix the problem, so it is sensible to add some buffer time to the requested wall time of the job to allow for this. At the cost of further dependencies, we could add some code to send an email notification into our handle_error function.

Comments or suggestions?

You can send comments and other suggestions to my University email and I may add them below.