Diagram of stampede2 directories and what connects to what, and how fast
Stampede2 is a collection of 4,200 KNL nodes (computers), 1736 SKX nodes (computers) connected to three file servers, each with unique characteristics.
You need to understand the file servers to know how to use them.
$HOME | $WORK2 | $SCRATCH | |
|---|---|---|---|
Purged? | No | No | Files can be purged if not accessed for 10 days. |
Backed Up? | Yes | No | No |
Capacity | 10GB | 1TB | Basically infinite. 8.5 PB |
Command to Access | cdh | cdw2 | cds |
Purpose | Store Executables | Store Files | Run Jobs |
Executables that aren't available on TACC through the "module" command should be stored in $HOME.
If you plan to be using a set of files frequently or would like to save the results of a job, they should be stored in $WORK.
If you're going to run a job, it's a good idea to keep your input files in a directory in $WORK2 and copy them to a directory in $SCRATCH where you plan to run your job.
This example command might help a bit:
cp $WORK2/my_fastq_data/*fastq $SCRATCH/my_project/
Stampede2's /home and /scratch file systems are mounted and visible only on Stampede2, but the work file system mounted on Stampede2 is part of the global file system hosted on Stockyard. This means /work and /work2 are visible on common among TACC clusters : lonestar5, stampede2, and frontera.
General Guidelines to reduce File I/O load on TACC:
Now let's go on to look at how jobs are run on stampede2.