Content Comparison

...

1. Create a commands file containing exactly one task per line.
2. Prepare a job control file for the commands file that describes how the job should be run.
3. You submit the job control file to the batch system.
   1. The job is then said to be queued to run.
4. The batch system prioritizes the job based on the number of compute nodes needed and the job run time requested.
5. When compute nodes become available, the job tasks (command lines in the <job_name>.cmds file) are assigned to one or more compute nodes and begin to run in parallel.
6. The job completes when either:
   1. you cancel the job manually
   2. all job tasks complete (successfully or not!)
   3. the requested job run time has expired

...

Here are the main components of the SLURM batch system.

...

Let's go through a simple example. Execute the following commands to copy a pre-made premade simple.cmds commands file:

...

There are 8 tasks. Each task sleeps for 5 seconds, then uses the echo command to output a string containing the task number and date to a log file named for the task number. Notice that we can put two commands on one line if they are separated by a semicolon ( ; ).

You can count the number of lines in the simple.cmds file using the wc (word count) command with the -l (lines) option:

wc -l simple.cmds

Use the handy launcher_creator.py program to create the job control file.

mkdir -p ~/local/bin
cd ~/local/bin
ln -s -f /work/projects/BioITeam/common/bin/launcher_creator.py

launcher_creator.py -j simple.cmds -n simple -t 00:01:00 -a TRA23004 -q developmentOTH21164

You should see output something like the following, and you should see a simple.slurm batch submission file in the current directory.

Project simple.
Using job file simple.cmds.
Using developmentnormal queue.
For 00:01:00 time.
Using TRA23004OTH21164 allocation.
Not sending start/stop email.
Launcher successfully created. Type "sbatch simple.slurm" to queue your job.

...

sbatch simple.slurm 
showq -u

# Output looks something like this:
-------------------------------------------------------------
          Welcome to the Lonestar6 Supercomputer
-------------------------------------------------------------
--> Verifying valid submit host (login2)...OK
--> Verifying valid jobname...OK
--> Verifying valid ssh keys...OK
--> Verifying access to desired queue (normal)...OK
--> Checking available allocation (TRA23004)...OK
Submitted batch job 1722779

...

 desired queue (normal)...OK
--> Checking available allocation (OTH21164)...OK
Submitted batch job 2411919

The queue status will initially show your job as WAITING until a node becomes available:

SUMMARY OF JOBS FOR USER: <abattenh>

ACTIVE JOBS--------------------
JOBID     JOBNAME    USERNAME      STATE   NODES REMAINING STARTTIME
================================================================================

WAITING JOBS------------------------
JOBID     JOBNAME    USERNAME      STATE   NODES WCLIMIT   QUEUETIME
================================================================================
2411919   simple     abattenh      Waiting 1     0:01:00   Wed May 28 15:39:24

Total Jobs: 1     Active Jobs: 0     Idle Jobs: 1     Blocked Jobs: 0

Once your job is ACTIVE (running) you'll see something like this:

SUMMARY OF JOBS FOR USER: <abattenh>

ACTIVE JOBS--------------------
JOBID     JOBNAME    USERNAME      STATE   NODES REMAINING STARTTIME
================================================================================
1722779   simple     abattenh      Running 1      0:00:39  Sat Jun  1 21:55:28

WAITING JOBS------------------------
JOBID     JOBNAME    USERNAME      STATE   NODES WCLIMIT   QUEUETIME
================================================================================

Total Jobs: 1     Active Jobs: 1     Idle Jobs: 0     Blocked Jobs: 0

...

Every job, no matter how few tasks requested, will be assigned at least one node. Each lonestar6 node has 128 physical cores, so each of the 8 tasks can be assigned to a different core.

...

cmd1.log  cmd3.log  cmd5.log  cmd7.log  simple.cmds     simple.o924965 
cmd2.log  cmd4.log  cmd6.log  cmd8.log  simple.e924965  simple.slurm

filename wildcarding

You can look at one of the output log files like this:

...

Command 1 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:47 CDT 2024
Command 2 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:47 CDT 2024
Command 3 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:51 CDT 2024
Command 4 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:48 CDT 2024
Command 5 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:45 CDT 2024
Command 6 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:49 CDT 2024
Command 7 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:51 CDT 2024
Command 8 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:48 CDT 2024

echo

Lets take a closer look at a typical task in the simple.cmds file.

...

The echo command is like a print statement in the bash shell.  echo takes its arguments and writes them to standard output. While not always required, it is a good idea to put echo's output string in double quotes ( " ).

backtick evaluation

So what is this funny looking `date` bit doing? Well, date is just another Linux command (try just typing it in) that just displays the current date and time. Here we don't want the shell to put the string "date" in the output, we want it to execute the date command and put the result text into the output. The backquotes ( ` ` ) also called backticks) , around the date command tell the shell we want that command executed and its standard output substituted into the string. (Read more about Quoting in the shell)

...

The '2>&1' part says to redirect standard error to the same place. Technically, it says to redirect standard error (built-in Linux stream 2) to the same place as standard output (built-in Linux stream 1); and since . Since standard output is going to cmd3.log, any standard error will go there also. (Read more about Standard streams and redirection)

When the TACC batch system runs a job, all outputs generated by tasks in the batch job are directed to one output and error file per job. Here they have names like simple.e924965e2411919 and simple.o924965o24119195.

• simple.

...

• o2411919 contains all standard output

...

• generated by your task that was not redirected elsewhere
• simple.

...

• e2411919 contains all standard error generated by your tasks that was not redirected elsewhere

...

• both also contain information relating to running your job and its tasks

...

For large jobs with complex tasks, it is not easy to troubleshoot execution problems using these files.

...

The launcher module knows how to interpret various job parameters in the <job_name>.slurm batch SLURM submission script and use them to create your job and assign its tasks to compute nodes. Our launcher_creator.py program is a simple Python script that lets you specify job parameters and writes out a valid <job_name>.slurm submission script.

...

# Use spacebar to page forward; Ctrl-c or q to exit
launcher_creator.py | more

...

launcher_creator.py -j simple.cmds -n simple -t 00:01:00 -a TRA23004 -q developmentOTH21164

• The name of your commands file is given with the -j simple.cmds option.
• Your desired job name is given with the -n simple option.
  • The <job_name> (here simple) is the job name you will see in your queue.
  • By default a corresponding <job_name>.slurm batch file is created for you.
    • It contains the name of the commands file that the batch system will execute.

...

• In launcher_creator.py, the queue is specified by the -q argument.
  
  • The default queue is development. Specify -q normal for normal queue jobs.
• The maximum runtime you are requesting for your job is specified by the -t argument.
  • Format is hh:mm:ss
  • Note that your job will be terminated without warning at the end of its time limit!

...

• You specify that allocation name with the -a argument of launcher_creator.py.
• If you have set an $ALLOCATION environment variable to an allocation name, that allocation will be used.
  
  

# This sets the default project allocation for launcher_creator.py
export ALLOCATION=TRA23004OTH21164

• When you run a batch job, your project allocation gets "charged" for the time your job runs, in the currency of SUs (System Units).
• SUs are related in some way to node hours, usually 1 SU = 1 "standard" node hour.
  
  

...

Exercise: How many tasks are specified in the wayness.cmds file?

...

...

wc -l wayness.cmds

...

launcher_creator.py -j wayness.cmds -n wayness -w 4 -t 00:02:00 -a TRA23004OTH21164 -q development
sbatch wayness.slurm
showq -u

Exercise:

• With 16 tasks requested and wayness of 4, how many nodes will this job require?
• How much memory will be available for each task?

 ExerciseExercise:

• If you specified a wayness of 2, how many nodes would this job require?
• How much memory could each task use?

...

cat cmd*log

# or, for a listing ordered by command number (the 2nd space-separated field)
cat cmd*log | sort -k 2k2,2n

The vertical bar ( | ) above is the pipe operator, which connects one program's standard output to the next program's standard input.

...

   4 c303-005.ls6.tacc.utexas.edu
   4 c303-006.ls6.tacc.utexas.edu
   4 c304-005.ls6.tacc.utexas.edu
   4 c304-006.ls6.tacc.utexas.edu

(Read more about awk in Some Linux commands: awk, and read more about Piping a histogram). And we'll be doing more of this sort of thing soon!

Some best practices

Redirect task output and error streams

...

# Redirect both standard output and standard error to a file

my_program input_file1 output_file1 output_file1 > file1.log 2>&1 > file1.log 2>&1

# Redirect standard output and standard error to separate files
my_program 1>out.txt 2>err.log

Combine serial workflows into scripts

...

Here's an example directory structure

$WORK/my_project
                 /01.original      # contains or links to original fastq files
                 /02.fastq_prep    # run fastq QC and trimming jobs here
                 /03.alignment     # run alignment jobs here
                 /04.analysis      # analyze gene overlap here
         /51        /81.test1         # play around with stuff here
         /52        /82.test2         # play around with other stuff here


...

cd $WORK/my_project/02.fastq_prep
ln -sf ../01.original fq
ls ./fq/my_raw_sequences.fastq.gz

relative path syntax

As we have seen, there are several special "directory names" the bash shell understands:

...

idev -m 60 -N 1 -A TRA23004OTH21164 -p normal -r CoreNGS--reservation=core-ngs-class-0603

Notes:

• -p normal requests nodes on the normal queue
  • this is the default for our reservation, while the development queue is the normal default
• -m 60 asks for a 60 minute session
• -A TRA23004 OTH21164 specifies the TACC allocation/project to use
• -N 1 asks for 1 node
• --reservation=CoreNGS-Tuecore-ngs-class-0603 gives us priority access to TACC nodes for the today's class. You normally won't use this option.

When you ask for an idev session, you'll see output as similar to that shown below. Note that the process may repeat the "job status:  PD" (pending) step while it waits for an available node.

  -> Checking on the status of development queue. OK

 -> Defaults file    : ~/.idevrc
 -> System           : ls6
 -> Queue            : developmentnormal         (cmd line: -p        )
 -> Nodes            : 1              (cmd line: -N        )
 -> Tasks per Node   : 128            (Queue default       )
 -> Time (minutes)   : 60             (cmd line: -m        )
 -> Project          : TRA23004OTH21164       (cmd line: -A        )

-----------------------------------------------------------------
          Welcome to the Lonestar6 Supercomputer
-----------------------------------------------------------------

--> Verifying valid submit host (login1login2)...OK
--> Verifying valid jobname...OK
--> Verifying valid ssh keys...OK
--> Verifying access to desired queue (developmentnormal)...OK
--> Checking available allocation (TRA23004OTH21164)...OK
Submitted batch job 2354652412167

 -> After your idev job begins to run, a command prompt will appear,
 -> and you can begin your interactive development session.
 -> We will report the job status every 4 seconds: (PD=pending, R=running).

 -> job status:  PD
 -> job status:  R

 -> Job is now running on masternode= c302c304-005006...OK
 -> Sleeping for 7 seconds...OK
 -> Checking to make sure your job has initialized an env for you....OK
 -> Creating interactive terminal session (login) on master node c302c304-005006.
 -> ssh -Y  -o "StrictHostKeyChecking no" c302c304-005006   

Once the idev session has started, it looks quite similar to a login node environment, except for these differences:

• the hostname command on a login node will return a login server name like login3.ls6.tacc.utexas.edu
  • while in an idev session hostname returns a compute node name like c303-006.ls6.tacc.utexas.edu
• you cannot submit a batch job from inside an idev session, only from a login node
• your idev session will end when the requested time has expired
  • or you can just type exit to return to a login node session

...