Content Comparison

...

Use the handy launcher_creator.py program to create the job control file.

mkdir -p ~/local/bin
cd ~/local/bin
ln -s -f /work/projects/BioITeam/common/bin/launcher_creator.py

launcher_creator.py -j simple.cmds -n simple -t 00:01:00 -a OTH21164 -r core-ngs-class-0603

You should see output something like the following, and you should see a simple.slurm batch submission file in the current directory.

...

cmd1.log  cmd3.log  cmd5.log  cmd7.log  simple.cmds     simple.o924965 
cmd2.log  cmd4.log  cmd6.log  cmd8.log  simple.e924965  simple.slurm

filename wildcarding

You can look at one of the output log files like this:

...

Command 1 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:47 CDT 2024
Command 2 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:47 CDT 2024
Command 3 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:51 CDT 2024
Command 4 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:48 CDT 2024
Command 5 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:45 CDT 2024
Command 6 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:49 CDT 2024
Command 7 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:51 CDT 2024
Command 8 on c304-005.ls6.tacc.utexas.edu - Sat Jun  1 21:55:48 CDT 2024

echo

Lets take a closer look at a typical task in the simple.cmds file.

...

The echo command is like a print statement in the bash shell.  echo takes its arguments and writes them to standard output. While not always required, it is a good idea to put echo's output string in double quotes ( " ).

backtick evaluation

So what is this funny looking `date` bit doing? Well, date is just another Linux command (try just typing it in) that just displays the current date and time. Here we don't want the shell to put the string "date" in the output, we want it to execute the date command and put the result text into the output. The backquotes ( ` ` ) also called backticks, around the date command tell the shell we want that command executed and its standard output substituted into the string. (Read more about Quoting in the shell)

...

launcher_creator.py -j simple.cmds -n simple -t 00:01:00 -a OTH21164 -r core-ngs-class-0603

• The name of your commands file is given with the -j simple.cmds option.
• Your desired job name is given with the -n simple option.
  • The <job_name> (here simple) is the job name you will see in your queue.
  • By default a corresponding <job_name>.slurm batch file is created for you.
    • It contains the name of the commands file that the batch system will execute.

...

launcher_creator.py -j wayness.cmds -n wayness -w 4 -t 00:02:00 -a OTH21164 -r core-ngs-class-0603
sbatch wayness.slurm
showq -u

Exercise:

• With 16 tasks requested and wayness of 4, how many nodes will this job require?
• How much memory will be available for each task?

 ExerciseExercise:

• If you specified a wayness of 2, how many nodes would this job require?
• How much memory could each task use?

...

cat cmd*log

# or, for a listing ordered by command number (the 2nd space-separated field)
cat cmd*log | sort -k2,2n

...

Read more about awk in Some Linux commands: awk, and read more about Piping a histogram. And we'll be doing more of this sort of thing soon!

Some best practices

Redirect task output and error streams

...

# Redirect both standard output and standard error to a file
my_program input_file1 output_file1 > file1.log 2>&1

# Redirect standard output and standard error to separate fielsfiles
my_program 1>out.txt 2>err.log

...

Here's an example directory structure

$WORK/my_project
                 /01.original      # contains or links to original fastq files
                 /02.fastq_prep    # run fastq QC and trimming jobs here
                 /03.alignment     # run alignment jobs here
                 /04.analysis      # analyze gene overlap here
                 /81.test1         # play around with stuff here
                 /82.test2         # play around with other stuff here


...

cd $WORK/my_project/02.fastq_prep
ln -sf ../01.original fq
ls ./fq/my_raw_sequences.fastq.gz

relative path syntax

As we have seen, there are several special "directory names" the bash shell understands:

...