...
| Code Block | ||
|---|---|---|
| ||
4 c303-005.ls6.tacc.utexas.edu 4 c303-006.ls6.tacc.utexas.edu 4 c304-005.ls6.tacc.utexas.edu 4 c304-006.ls6.tacc.utexas.edu |
(Read more about awk in Some Linux commands: awk, and read more about Piping a histogram).
Some best practices
Redirect task output and error streams
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
# Redirect both standard output and standard error to a file | my_program input_file1 output_file1 > file1.log 2>&1 # Redirect standard output and standard error to separate fiels my_program 1>out.txt 2>err.log |
Combine serial workflows into scripts
...
$WORK/my_project
/01.original # contains or links to original fastq files
/02.fastq_prep # run fastq QC and trimming jobs here
/03.alignment # run alignment jobs here
/04.analysis # analyze gene overlap here
/5181.test1 # play around with stuff here
/5282.test2 # play around with other stuff here
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
idev -m 60 -N 1 -A TRA23004OTH21164 -p normal --r reservation=CoreNGS-Tue |
Notes:
- -p normal requests nodes on the normal queue
- this is the default for our reservation, while the development queue is the normal default
- -m 60 asks for a 60 minute session
- -A TRA23004 OTH21164 specifies the TACC allocation/project to use
- -N 1 asks for 1 node
- --reservation=CoreNGS-Tue gives us priority access to TACC nodes for the class. You normally won't use this option.
When you ask for an idev session, you'll see output as similar to that shown below. Note that the process may repeat the "job status: PD" (pending) step while it waits for an available node.
| Code Block |
|---|
-> Checking on the status of development queue. OK -> Defaults file : ~/.idevrc -> System : ls6 -> Queue : normal development (cmd line: -p ) -> Nodes : 1 (cmd line: -N ) -> Tasks per Node : 128 (Queue default ) -> Time (minutes) : 60 (cmd line: -m ) -> Project : TRA23004OTH21164 (cmd line: -A ) ----------------------------------------------------------------- Welcome to the Lonestar6 Supercomputer ----------------------------------------------------------------- --> Verifying valid submit host (login1login2)...OK --> Verifying valid jobname...OK --> Verifying valid ssh keys...OK --> Verifying access to desired queue (developmentnormal)...OK --> Checking available allocation (TRA23004OTH21164)...OK Submitted batch job 2354652412167 -> After your idev job begins to run, a command prompt will appear, -> and you can begin your interactive development session. -> We will report the job status every 4 seconds: (PD=pending, R=running). -> job status: PD -> job status: R -> Job is now running on masternode= c302c304-005006...OK -> Sleeping for 7 seconds...OK -> Checking to make sure your job has initialized an env for you....OK -> Creating interactive terminal session (login) on master node c302c304-005006. -> ssh -Y -o "StrictHostKeyChecking no" c302c304-005006 |
Once the idev session has started, it looks quite similar to a login node environment, except for these differences:
- the hostname command on a login node will return a login server name like login3.ls6.tacc.utexas.edu
- while in an idev session hostname returns a compute node name like c303-006.ls6.tacc.utexas.edu
- you cannot submit a batch job from inside an idev session, only from a login node
- your idev session will end when the requested time has expired
- or you can just type exit to return to a login node session
...