Running batch jobs at TACC

Compute cluster overview

When you SSH into ls6, your session is assigned to one of a small set of login nodes (also called head nodes). These are separate from the cluster compute nodes that will run your jobs.

Think of a TACC node as a computer, like your laptop, but probably with more cores and memory. Now multiply that computer a thousand or more, and you have a cluster.

Open

The small set of login nodes are a shared resource – type the users command to see everyone currently logged in. Login nodes are not meant for running interactive programs – for that you submit a description of what you want done to a batch system, which distributes the work to one or more compute nodes.

On the other hand, the login nodes are intended for copying files to and from TACC, so they have a lot of network bandwidth while compute nodes have limited network bandwidth.

So follow these guidelines:

• Do not perform substantial computation on the login nodes.
  
  

  • They are closely monitored, and you will get warnings – or worse – from the TACC admin folks!

• Do not perform significant network access from your batch jobs.
  
  

  • Instead, stage your data from a login node onto $SCRATCH before submitting your job.

So what about developing and testing your code and workflows? Initial code development is often performed elsewhere, then migrated to TACC once is is relatively stable. But for local testing and troubleshooting, TACC offers interactive development (idev) nodes. We'll be taking advantage of idev nodes later on.

Lonestar6 and Stampede3 overview and comparison

Here is a comparison of the configurations and ls6 and stampede3. stampede3 is the newer (and larger) cluster, launched in 2024; ls6 was launched in 2022.

User guides for ls6 and stampede3 can be found at:

• https://docs.tacc.utexas.edu/hpc/lonestar6/

  • see https://docs.tacc.utexas.edu/hpc/lonestar6/#running-queues for all available queues

• https://docs.tacc.utexas.edu/hpc/stampede3

Unfortunately, the TACC user guides are aimed towards a different user community – the weather modelers and aerodynamic flow simulators who need very fast matrix manipulation and other High Performance Computing (HPC) features. The usage patterns for bioinformatics – generally running 3rd party tools on many different datasets – is rather a special case for HPC. TACC calls our type of processing "parameter sweep jobs" and has a special process for running them, using their launcher module.

Software at TACC

Programs and your $PATH

When you type in the name of an arbitrary program (ls for example), how does the shell know where to find that program? The answer is your $PATH. $PATH is a predefined environment variable whose value is a list of directories.The shell looks for program names in that list, in the order the directories appear.

To determine where the shell will find a particular program, use the which command. Note that which tells you where it looked if it cannot find the program.

Using which to search $PATH

The module system

The module system is an incredibly powerful way to have literally thousands of software packages available, some of which are incompatible with each other, without causing complete havoc. The TACC staff stages packages in well-known locations that are NOT on your $PATH. Then, when a module is loaded, its binaries are added to your $PATH.

For example, the following module load command makes the apptainer container management system available to you:

How module load affects $PATH

Note that apptainer is another name for singularity.

TACC BioContainers modules

It is quite a large systems administration task to install software at TACC and configure it for the module system. As a result, TACC was always behind in making important bioinformatics software available. To address this problem, TACC moved to providing bioinformatics software via containers, which are similar to virtual machines like VMware and Virtual Box, but are lighter weight: they require less disk space because they rely more on the host's base Linux environment. Specifically, TACC (and many other HPC/High Performance Computing clusters) use Apptainer containers, which are similar to Docker containers but are more suited to the HPC environment – in fact one can build a Docker container then easily convert it to Apptainer for use at TACC.

TACC obtains its containers from BioContainers (https://biocontainers.pro/ and https://github.com/BioContainers/containers), a large public repository of bioinformatics tool Apptainer containers. This has allowed TACC to easily provision thousands of such tools!

These BioContainers are not visible in TACC's "standard" module system, but only after the master biocontainers module is loaded. Once it has been loaded, you can search for your favorite bioinformatics program using module spider.

Notice how the BioContainers module names have "ctr" in their names, version numbers, and other identifying information.

loading a biocontainer module

Once the biocontainers module has been loaded, you can just module load the desired tool, as with the kallisto pseudo-aligner program below.

Note that loading a BioContainer does not add anything to your $PATH. Instead, it defines an alias, which is just a shortcut for executing the command using its container. You can see the alias definition using the type command. And you can ensure the program is available using the command -v utility.

installing custom software

Even with all the tools available at TACC, inevitably you'll need something they don't have. In this case you can build the tool yourself and install it in a local TACC directory. While building 3rd party tools is beyond the scope of this course, it's really not that hard. The trick is keeping it all organized.

For one thing, remember that your $HOME directory quota is fairly small (10 GB on ls6), and that can fill up quickly if you install many programs. We recommend creating an installation area in your $WORK directory and installing programs there. You can then make symbolic links to the binaries you need in your ~/local/bin directory (which was added to your $PATH in your .bashrc).

See how we used a similar trick to make the launcher_creator.py program available to you. Using the ls -l option shows you where symbolic links point to:

Real location of launcher_creator.py

Job Execution

Job execution is controlled by the SLURM batch system on both stampede3 and ls6.

To run a job you prepare 2 files:

1. a commands file file containing the commands to run, one task per line (<job_name>.cmds)

2. a job control file that describes how to run the job (<job_name>.slurm)

The process of running the job involves these steps:

1. Create a commands file containing exactly one task per line.

2. Prepare a job control file for the commands file that describes how the job should be run.

3. You submit the job control file to the batch system.

   1. The job is then said to be queued to run.

4. The batch system prioritizes the job based on the number of compute nodes needed and the job run time requested.

5. When compute nodes become available, the job tasks (command lines in the <job_name>.cmds file) are assigned to one or more compute nodes and begin to run in parallel.

6. The job completes when either:

   1. you cancel the job manually

   2. all job tasks complete (successfully or not!)

   3. the requested job run time has expired

SLURM at a glance

Here are the main components of the SLURM batch system.

Simple example

Let's go through a simple example. Execute the following commands to copy a premade simple.cmds commands file:

Copy simple commands

What are the tasks we want to do? Each task corresponds to one line in the simple.cmds commands file, so let's take a look at it using the cat (concatenate) command that simply reads a file and writes each line of content to standard output (here, your Terminal):

View simple commands

The tasks we want to perform look like this:

simple.cmds commands file

There are 8 tasks. Each task sleeps for 5 seconds, then uses the echo command to output a string containing the task number and date to a log file named for the task number. Notice that we can put two commands on one line if they are separated by a semicolon ( ; ).

You can count the number of lines in the simple.cmds file using the wc (word count) command with the -l (lines) option:

Count file lines

Use the handy launcher_creator.py program to create the job control file.

Create batch submission script for simple commands

You should see output something like the following, and you should see a simple.slurm batch submission file in the current directory.

Submit your batch job then check the batch queue to see the job's status.

Submit simple job to batch queue

The queue status will initially show your job as WAITING until a node becomes available:

Once your job is ACTIVE (running) you'll see something like this:

If you don't see your simple job in either the ACTIVE or WAITING sections of your queue, it probably already finished – it should only run for a few seconds!

Notice in my queue status, where the STATE is Running, there is only one node assigned. Why is this, since there were 8 tasks?

Every job, no matter how few tasks requested, will be assigned at least one node. Each lonestar6 node has 128 physical cores, so each of the 8 tasks can be assigned to a different core.

Exercise: What files were created by your job?

filename wildcarding

You can look at one of the output log files like this:

But here's a cute trick for viewing the contents all your output files at once, using the cat command and filename wildcarding.

Multi-character filename wildcarding

The cat command can take a list of one or more files. The asterisk ( * ) in cmd*.log is a multi-character wildcard that matches any filename starting with cmd then ending with .log.

You can also specify single-character matches inside brackets ( [ ] ) in either of the ways below, this time using the ls command so you can better see what is matching:

Single character filename wildcarding

This technique is sometimes called filename globbing, and the pattern a glob. Don't ask me why – it's a Unix thing. Globbing – translating a glob pattern into a list of files – is one of the handy thing the bash shell does for you. (Read more about Pathname wildcards)

Exercise: How would you list all files starting with "simple"?

Here's what my cat output looks like. Notice the times are all nearly the same because all the tasks ran in parallel. That's the power of cluster computing!

echo

Lets take a closer look at a typical task in the simple.cmds file.

A simple.cmds task line

The echo command is like a print statement in the bash shell.  echo takes its arguments and writes them to standard output. While not always required, it is a good idea to put echo's output string in double quotes ( " ).

backtick evaluation

So what is this funny looking `date` bit doing? Well, date is just another Linux command (try just typing it in) that just displays the current date and time. Here we don't want the shell to put the string "date" in the output, we want it to execute the date command and put the result text into the output. The backquotes ( ` ` ) also called backticks, around the date command tell the shell we want that command executed and its standard output substituted into the string. (Read more about Quoting in the shell)

Backtick evaluation

output redirection

There's still more to learn from one of our simple tasks, something called output redirection.

Every command and Unix program has three "built-in" streams: standard input, standard output and standard error, each with a name, a number, and a redirection syntax.

Open

Normally echo writes its string to standard output, but it could encounter an error and write an error message to standard error. We want both standard output and standard error for each task stored in a log file named for the command number.

A simple.cmds task line

So in the above example the first '>' says to redirect the standard output of the echo command to the cmd3.log file.

The '2>&1' part says to redirect standard error to the same place. Technically, it says to redirect standard error (built-in Linux stream 2) to the same place as standard output (built-in Linux stream 1). Since standard output is going to cmd3.log, any standard error will go there also. (Read more about Standard streams and redirection)

When the TACC batch system runs a job, all outputs generated by tasks in the batch job are directed to one output and error file per job. Here they have names like simple.e2411919 and simple.o24119195.

• simple.o2411919 contains all standard output generated by your task that was not redirected elsewhere

• simple.e2411919 contains all standard error generated by your tasks that was not redirected elsewhere

• both also contain information relating to running your job and its tasks

For large jobs with complex tasks, it is not easy to troubleshoot execution problems using these files.

So a best practice is to separate the outputs of each of the tasks into an individual log file, one per task, as we do here. Why is this important? Suppose we run a job with 100 commands, each one a whole pipeline (alignment, for example). 88 finish fine but 12 do not. Just try figuring out which ones had the errors, and where the errors occurred, if all the standard output is in one intermingled file and all standard error in the other intermingled file!

Job parameters

Now that we've executed a really simple job, let's take a look at some important job submission parameters. These correspond to arguments to the launcher_creator.py script.