Linux and Lonestar refresher
- 1 Overview:
- 2 Objectives:
- 3 Tutorial:
- 3.2 Diagram of Lonestar directories: What connects to what, how fast, and for how long.
- 3.3 Understanding "jobs" and compute nodes.
- 3.4 Running a job
- 3.5 Interrogating the launcher queue
- 3.6 Evaluating your first job submission
- 3.7 Moving beyond the preinstalled commands on TACC
Overview:
Based on your pre-class survey answers, the vast majority of you are pretty familiar with the basic linux commands. This portion of the class is devoted to making sure we are all starting from the same starting point on lonestar. Previous versions of portions of this tutorial can be found here, here, here, here, here, and here. Collective thanks to all those that contributed to those works which now appear in a single version. Anyone wishing to use this tutorial is welcome.
Objectives:
Log into lonestar.
Change your lonestar profile to the course specific format.
Refresh understanding of basic linux commands with some course organization.
Review use of the nano text editor program, and become familiar with several other text editor programs.
Tutorial:
Logging into lonestar
Start a new terminal window. For MACs this is done by clicking on the magnifying glass on the right hand side of the toolbar at the top of the page and type "terminal". For windows this should be done by connecting through cygwin. Log into lonestar using your account information.
This brings us to our first "code block". There will be 3 types of code blocks used throughout this class:
Visible
These are code blocks that you would have no idea what to type without help.
These will typically be associated with longer/more detailed text above the text box explaining things.
Hinted
These are code blocks that you can probably figure out what to type with a hint that goes beyond what the tutorial is requesting. Access the hint by clicking the triangle or hint hyperlink.
These will always contain an additional hidden code block incase you don't find the hint as clever as we did.
Hidden
These code blocks represent things that either there is a good chance you know how to do already, something too straightforward to warrant a hint, or are there to give you the answer if the hint doesn't help. Access the answer by clicking "expand source" on the right hand side of the code block.
Text inside of code blocks represent "right" answers, and should either be typed EXACTLY into the terminal window as they are, or copy pasted with a noteable exception. Things that exist within <> symbols represent something that you need to replace before sending it to the terminal. We try to put informative text within the brackets so you know what to replace it with. If you are ever unsure of what to replace the <> text with, just ask.
Using what we have just taught you about code blocks, log into lonestar. Since this is your first code box, it is probably worth expanding even if you know how to log into lonestar already.
When prompted enter your password, and answer "yes" to the security question.
Logging into remote computers
As a matter of internet safety, the terminal window knows you are entering a password and may not want your neighbor to see what it is. For this reason, even as you type to enter your password, nothing will be displayed on the screen. While backspace will work if you know you made a mistake, we often find it better to just hit enter and try again.
If you have never logged into lonestar from the computer you are currently using before, you will be issued a security warning. The same will be true if you log into any of the other TACC resources, or any other remote computer. If you ever see a security warning logging into somewhere that you use commonly you should answer no and try to figure out why you were warned. Otherwise type "yes" to bypass the security check.
Setting up your lonestar profile and other variables
There are many flavors of Linux/Unix shells. The default for TACC's Linux (and most other Linuxes) is bash (bourne again shell), which we will use throughout.
Whenever you login via an interactive shell as you did above, a well-known script is executed by the shell to establish your favorite environment settings. We've set up a common profile for you to start with that will help you know where you are in the file system and make it easier to access some of our shared resources. If you already have a profile set up on lonestar that you like, we want to make sure that we don't destroy it but it will be important to make sure that we change it temporarily. Use the ls command to check if you have a .profile already set up in your home directory.
If you already have a .profile file, use the mv command to change the name to something descriptive (for example ".profile_pre_bdib_backup"). Otherwise continue to creating a new .profile file.
Copy our predefined ngsc_profile_user file from /corral-repl/utexas/BioITeam/scripts/ folder as .profile before using the chmod command to change the permissions to read and write for the user only.
The chmod 600 .profile command marks the file as readable/writable only by you. The .profile script file will not be executed unless it has these permissions settings. Note that the well-known filename is .profile (or .profile_user on some systems), which is specific to the bash shell.
Notice that when you do a normal ls to list the contents of your home directory, this file doesn't appear. That's because it's a hidden "dot file" – a file that has no filename, only an extension. To see these hidden files use the -a (all) switch for ls:
To see even more detail, including file permissions, add the -l (long listing) switch:
Since .profile is executed when you login, to ensure it is set up properly you should first logout:
then log back in:
If everything is working correctly you should now see a prompt like this: tacc:~$
In order to make navigating to the different file systems on lonestar a little easier ($SCRATCH and $WORK), you can set up some shortcuts with these commands that create folders that "link" to those locations. Run these commands when logged into Lonestar with a terminal, from your home directory.
Creating a shortcut to the main Lonestar working directories
cdh
ln -s $SCRATCH scratch
ln -s $WORK work
ln -s $BI BioITeam
Understanding what your .profile file actually does.
Editing files
There are a number of options for editing files at TACC. These fall into three categories:
Linux text editors installed at TACC (nano, vi, emacs). These run in your terminal window. vi and emacs are extremely powerful but also quite complex, so nano may be the best choice as a first local text editor.
Text editors or IDEs that run on your local computer but have an SFTP (secure FTP) interface that lets you connect to a remote computer (Notepad++ or Komodo Edit). Once you connect to the remote host, you can navigate its directory structure and edit files. When you open a file, its contents are brought over the network into the text editor's edit window, then saved back when you save the file.
Software that will allow you to mount your home directory on TACC as if it were a normal disk e.g. MacFuse/MacFusion for Mac, or ExpanDrive for Windows or Mac ($$, but free trial). Then, you can use any text editor to open files and copy them to your computer with the usual drag-drop.
We'll go over nano together in class, but you may find these other options more useful for your day-to-day work so feel free to go over these sections in your free time to familiarize yourself with their workings to see if one is better for you.
As we will be using nano throughout the class, it is a good idea to review some of the basics. nano is a very simple editor available on most Linux systems. If you are able to use ssh, you can use nano. To invoke it, just type:
How to start the nano text editor
nano
You'll see a short menu of operations at the bottom of the terminal window. The most important are:
ctl-o - write out the file
ctl-x - exit nano
You can just type in text, and navigate around using arrow keys. A couple of other navigation shortcuts:ctl-a - go to start of line
ctl-e - go to end of line
Be careful with long lines – sometimes nano will split long lines into more than one line, which can cause problems in our commands files, as you will see.
Stringing commands together and controlling their output
In a linux shell, it is often useful to take output of one command save it to a new file rather than having it print to the screen. It uses a familiar metaphor: "pipes". The linux operating system expects some "standard input pipe" and gives output back through a "standard output pipe". These are called "stdin" and "stdout" in linux. There's also a special "stderr" for errors; we'll ignore that for now. Usually, your shell is filling the operating system's stdin with stuff you type - the commands with options. The shell passes responses back from those commands to stdout, which the shell usually dumps to your screen. The ability to switch stdin and stdout around is one of the key reasons linux has existed for decades and beat out many other operating systems. Let's start making use of this. Change to the scratch directory and make a new folder called "piping" and put list of the full contents of the $BI folder to a new file called whatsHere.
Redirecting STDOUT
cds
mkdir piping
ls -1 $BI > whatsHere
cat whatsHereWhen you execute the ls -1 > whatsHere command, you should have noticed nothing happening on the screen, and when you cat the whatsHere file, you should notice the output you would have exptected from the ls -1 > whatsHere command. Often it is useful to chain commands together using the output of the first command as the input of the second command. Commands are chained together using the "|" character (shift \ above the return key). Use redirection to put the first 2 lines of the $BI directory contents into the whatsHere file.
Again, you should see your answer only showing up after the cat command. Note that by using a single > you are overwriting the existing contents and that there is no warning that this is happening beware of this in the future as linux doesn't have an "undo" feature. We will make use of the redirect output (stdout) character (>), and the "pass output along as input" "|" throughout the course. Not all shells are equal - the bash shell lets you redirect stdout with either > or 1>; stderr can be redirected with 2>; you can redirect both stdout and stderr using &>. If these don't work, use google to try to figure it out. The web site stackoverflow is a usually trustworthy and well annotated site for OS and shell help.
Understanding TACC
Now that we've been using lonestar for a little bit, and have it behaving in a way that is a little more useful to us, let's get more of a functional understanding of what exactly it is and how it works.
Diagram of Lonestar directories: What connects to what, how fast, and for how long.
Lonestar is a collection of 1,888 computers connected to three file servers, each with unique characteristics. You need to understand the file servers to know how to use them effectively.
| $HOME | $WORK | $SCRATCH |
|---|---|---|---|
Purged? | No | No | Files are purged if not accessed for 10 days. |
Backed Up? | Yes | No | No |
Capacity | 1GB | 250GB | Basically infinite. 1.4 PB |
Commands to Access | cdh cd $HOME/ | cdw cd $WORK/ | cds cd $SCRATCH/ |
Purpose | Store Executables | Store Files | Run Jobs |
Executables that aren't available on TACC through the "module" command should be stored in $HOME.
If you plan to be using a set of files frequently or would like to save the results of a job, they should be stored in $WORK.
If you're going to run a job, it's a good idea to keep your input files in a directory in $WORK and copy them to a directory in $SCRATCH where you plan to run your job.
Example command for copying data from a $WORK directory to $SCRATCH
cp $WORK/my_fastq_data/*fastq $SCRATCH/my_project/
Understanding "jobs" and compute nodes.
When you log into lonestar using ssh you are connected to what is known as the login node or "the head node". There are several different head nodes, but they are shared by everyone that is logged into lonestar (not just in this class, or from campus, or even from texas, but everywhere in the world). Anything you type onto the command line has to be executed by the head node. The longer something takes to complete, or the more it will slow down you and everybody else. Get enough people running large jobs on the head node all at once (say a classroom full of Big Data in Biology summer school students) and lonestar can actually crash leaving nobody able to execute commands or even log in for minutes -> hours -> even days if something goes really wrong. To try to avoid crashes, TACC tries to monitor things and proactively stop things before they get too out of hand. If you guess wrong on if something should be run on the head node, you may eventually see a message like the one pasted below. If you do, its not the end of the world, but repeated messages will become revoked TACC access and emails where you have to explain what you are doing to TACC and your PI and how you are going to fix it and avoid it in the future.
Example of how you learn you shouldn't have been on the head node
Message from root@login1.ls4.tacc.utexas.edu on pts/127 at 09:16 ...
Please do not run scripts or programs that require more than a few minutes of
CPU time on the login nodes. Your current running process below has been
killed and must be submitted to the queues, for usage policy see
http://www.tacc.utexas.edu/user-services/usage-policies/
If you have any questions regarding this, please submit a consulting ticket.
So you may be asking yourself what the point of using lonestar is at all if it is wrought with so many issues. The answer comes in the form of compute nodes. There are 1,888 compute nodes that can only be accessed by a single person for a specified amount of time. These compute nodes are divided into different queues called: normal, development, largemem, etc. Access to nodes (regardless of what queue they are in) is controlled by a "Queue Manager" program. You can personify the Queue Manager program as: Heimdall in Thor, a more polite version of Gandalf in lord of the rings when dealing with with the balrog, the troll from the billy goats gruff tail, or any other "gatekeeper" type. Regardless of how nerdy your personification choice is, the Queue Manager has an interesting caveat: it only speaks 1 language "job.sge". "job.sge" is a file that contains information on HOW/WHERE to run things (how many nodes you need, how long you need them for, how to charge your allocation, etc). The Queue manager doesn't care WHAT you are running, only HOW to find what you are running (which is specified by a setenv CONTROL_FILE commands line in job.sge). The WHAT is then handled by the file "commands" which contains what you would normally type into the command line to make things happen.
Further launcher.sge reading
To make things easier on all of us, there is a script called launcher_creator.py that you can use to automatically generate the "job.sge" file. This can all be summarized in the following figure:
Using launcher_creator.py
We have created a Python script called launcher_creator.py that makes creating a launcher.sge script a breeze. You will probably want to use this for the rest of the course. Now run the script with the -h option to show the help message so we can see what other options the script takes:
Short option | Long option | Required | Description |