Page Comparison

...

Code Block

language	bash
title	How to display all available options of the launcher_createor.py script
collapse	true

launcher_creator.py -h

Short option	Long option	Required	Description
-n	name	Yes	The name of the job.
-at	allocationtime		The allocation you want to charge the run toYes	Time allotment for job, format must be hh:mm:ss.
-qb	queue	Default: Development	The queue to submit to, like 'normal' or 'largemem', etc.
-w	wayness		Optional The number of jobs in a job list you want to give to each node. (Default is 12 for Lonestar, 16 for Stampede.)
-N	number of nodes		Optional Specifies a certain number of nodes to use. You probably don't need this option, as the launcher calculates how many nodes you need based on the job list (or Bash command string) you submit. It sometimes comes in handy when writing pipelines.
-t	time	Yes	Time allotment for job, format must be hh:mm:ss.
-e	email		Optional Your email address if you want to receive an email from Lonestar when your job starts and ends.
-l	launcher		Optional Filename of the launcher. (Default is `<name>.sge`)
-m	modules		Optional String of module management commands. `module load launcher` is always in the launcher, so there's no need to include that.
-b	Bash commands		Optional String of Bash commands to execute.
-j	Command list		Optional Filename of list of commands to be distributed to nodes.
-Bash commands	-b OR -j must be used	Optional String of Bash commands to execute.
-j	Command list	-b OR -j must be used	Optional Filename of list of commands to be distributed to nodes.
-a	allocation		The allocation you want to charge the run to. If you only have one allocation you don't need this option
-m	modules		Optional String of module management commands. `module load launcher` is always in the launcher, so there's no need to include that. Think to all the times in the class that you had to type 'module load xxxxx' while on the idev node. The same will be true for the launcher script. As you are more familiar with what types of analysis you will be doing, you will likely change your .bashrc file to limit the things you have to specify here.
-q	queue	Default: Development	The queue to submit to, like 'normal' or 'largemem', etc. You will usually want to change this to 'normal'
-w	wayness		Optional The number of jobs in a job list you want to give to each node. (Default is 12 for Lonestar, 16 for Stampede.)
-N	number of nodes		Optional Specifies a certain number of nodes to use. You probably don't need this option, as the launcher calculates how many nodes you need based on the job list (or Bash command string) you submit. It sometimes comes in handy when writing pipelines.
-e	email		Optional Your email address if you want to receive an email from Lonestar when your job starts and ends. If you set an environmental variable EMAIL_ADDRESS it will use that variable if you don't put anything after the -e
-l	launcher		Optional Filename of the launcher. (Default is `<name>.sge`)
-s	stdout		Optional Setting this flag outputs the name of the launcher to stdout.

...

Code Block

language	bash
title	how to make a sample commands file
collapse	true

# remember that things after the # sign are ignored by bash and most all programing languages
cds  # move to your scratch directory
nano commands
 
# the following lines should be entered into nano
echo "My name is _____ and todays date is:" > BDIB.output.txt
date >> BDIB.output.txt
echo "I have just demonstrated that I know how to redirect output to a new file, and to append things to an already created file. Or at least thats what I think I did" >> BDIB.output.txt
 
echo "i'm going to test this by counting the number of lines in the file that I am writing to. So if the next line reads 4 I remember I'm on the right track" >> BDIB.output.txt
wc -l BDIB.output.txt >> BDIB.output.txt
 
echo "I know that normally i would be typing commands on each line of this file, that would be executed on a compute node instead of the head node so that my programs run faster, in parallel, and do not slow down others or risk my tacc account being locked out" >> BDIB.output.txt
 
echo "i'm currently in my scratch directory on lonestar. there are 2 main ways of getting here: cds and cd $SCRATCH:" >>BDIB.output.txt
pwd >> BDIB.output.txt
 
echo "over the last week I've conducted multiple different types of analysis on a variety of sample types and under different conditions. Each of the exercises was taken from the website https://wikis.utexas.edu/display/bioiteam/Genome+Variant+Analysis+Course+20162017" >> BDIB.output.txt
 
echo "using the ls command i'm now going to try to remind you (my future self) what tutorials I did" >> BDIB.output.txt
 
ls -1 >> BDIB.output.txt
 
echo "the contents of those directories (representing the data i downloaded and the work i did) are as follows: ">> BDIB.output.txt
ls */* >> BDIB.output.txt
 
echo "the commands that i have run on the headnode are: " >> BDIB.output.txt
history >> BDIB.output.txt
 
echo "the contents of this, my commands file, which i will use in the launcher_creator.py script are: ">>BDIB.output.txt
cat commands >> BDIB.output.txt
 
echo "finally, I will be generating a job.slurm file using the launcher_creator.py script using the following command:" >> BDIB.output.txt
echo 'launcher_creator.py -w 1 -N 1 -n "what_i_did_at_BDIB_20162017" -t 00:02:00 -a "UT-2015-05-18"' >> BDIB.output.txt # this will create a my_first_job.slurm file that will run for 2 minutes
echo "and i will send this job to the que using the the command: sbatch what_i_did_at_BDIB_20162017.slurm" >> BDIB.output.txt  # this will actually submit the job to the Queue Manager and if everything has gone right, it will be added to the development queue.

 
ctrlo #keyboard command to write your nano output
crtlx # keyboard command to close the nano interface
 
launcher_creator.py -w 1 -N 1 -n "what_i_did_at_BDIB_20162017" -t 00:02:00 -a "UT-2015-05-18"
sbatch what_i_did_at_BDIB_20162017.slurm

Interrogating the launcher queue

...

Based on our example you may have expected 1 new file to have been created during the job submission (BDIB.output.txt), but instead you will find 3 2 extra files as follows: what_i_did.e(job-ID), and what_i_did.peo(job-ID), and what_i_did.o(job-ID). . When things have worked well, these files are typically ignored. When your job fails, these files offer insight into the why so you can fix things and resubmit.

...

Code Block

language	bash
title	make a single final file using the cat command and copy to a useful work directory
collapse	true

# remember that things after the # sign are ignored by bash 
cat BDIB.output.txt > first# remember that things after the # sign are ignored by bash 
cat BDIB.output.txt > end_of_class_job_submission.final.output 
mkdir $WORK/BDIB_GVA_2017
mkdir $WORK/BDIB_GVA_2017/end_of_course_summary/  # each directory must be made in order to avoid getting a no such file or directory error
cp end_of_class_job_submission.final.output  mkdir $WORK/BDIB_GVA_2016
mkdir_2017/end_of_course_summary/
cp what_i_did* $WORK/BDIB_GVA_20162017/end_of_course_summary/  # note this  # each directory must be made in order to avoid getting a no such file or directory error
cp first_job_submission.final.output $WORK/BDIB_GVA_2016/end_of_course_summary/
cp what_i_did*grabs the 2 output files generated by tacc about your job run as well as the .slurm file you created to tell it how to run your commands file

cp commands $WORK/BDIB_GVA_20162017/end_of_course_summary/

Return to GVA2017 to work on any additional tutorials you are interested in.

Versions Compared

Old Version 1

New Version 2

Key

Interrogating the launcher queue