Create SGE or SLURM launchers for Stampede and Lonestar
launcher_creator.py is a convenience utility that automates creating a launcher for TACC clusters based on parameters (and commands) you give it. It outputs a either a SLURM or SGE file that can be submitted to Stampede or Lonestar using sbatch
or qsub
. (It detects which machine it is being executed on, so the same code should work on either cluster.) It can run both Bash commands on a single node and/or a parametric list of commands to be dispatched to multiple nodes.
How To Use It
$ launcher_creator.py <options listed below>
-n |
name |
The name of the job. |
-a |
allocation |
The allocation you want to charge the run to. |
-q |
queue |
The queue to submit to, like 'normal' or 'largemem', etc. |
-w |
wayness |
Optional The number of jobs in a job list you want to give to each node. (Default is 12 for Lonestar, 16 for Stampede.) |
-N |
number of nodes |
Optional Specifies a certain number of nodes to use. You probably don't need this option, as the launcher calculates how many nodes you need based on the job list (or Bash command string) you submit. It sometimes comes in handy when writing pipelines. |
-t |
time |
Time allotment for job, format must be hh:mm:ss. |
-e |
Optional Your email address if you want to receive an email from Lonestar when your job starts and ends. |
|
-l |
launcher |
Optional Filename of the launcher. (Default is |
-m |
modules |
Optional String of module management commands. |
-b |
Bash commands |
Optional String of Bash commands to execute. |
-j |
Command list |
Optional Filename of list of commands to be distributed to nodes. |
-s |
stdout |
Optional Setting this flag outputs the name of the launcher to stdout. |
Note that while the "-b" and "-j" options are both optional, using neither would pass no work to Lonestar, so you want to use at least one of them!
Examples
Distributing Shrimp to four nodes
- Make a list of the Shrimp commands to run. You can do this in a text editor, algorithmically with a Bash loop, whatever. Let's pretend this text is saved as
mapping_commands.list
List of Shrimp commandsgmapper-ls -N 6 --qv-offset 33 -p opp-in --fastq -1 Sample_R1-00.fq -2 Sample_R2-00.fq reference.fasta > Sample_mapped-00.sam gmapper-ls -N 6 --qv-offset 33 -p opp-in --fastq -1 Sample_R1-01.fq -2 Sample_R2-01.fq reference.fasta > Sample_mapped-01.sam gmapper-ls -N 6 --qv-offset 33 -p opp-in --fastq -1 Sample_R1-02.fq -2 Sample_R2-02.fq reference.fasta > Sample_mapped-02.sam gmapper-ls -N 6 --qv-offset 33 -p opp-in --fastq -1 Sample_R1-03.fq -2 Sample_R2-03.fq reference.fasta > Sample_mapped-03.sam
- Generate a launcher using
launcher_creator.py
.Using launcher_creator.pylauncher_creator.py -q normal -a MyAllocation -n MapSample -j mapping_commands.list -w 2 -t 6:00:00 -l map_sample.sge -m "module load shrimp"
- Submit the newly-created launcher.
qsub the new launcher
qsub map_sample.sge