Linux fundamentals

This page should serve as a reference for the many "things Linux" we use in this course. It is by no means complete – Linux is **huge** – but offers introductions to many important topics.

See also this page, which provides lists of the most common Linux commands, by category, as well as their most useful options: Some Linux commands

Some Linux commands

• This page provides lists of the most common Linux commands, by category, as well as their most useful options:
  
  • Some Linux commands

Terminal programs, shells and commands

You need a Terminal program in order to ssh to a remote computer.

• Macs and Linux have a Terminal program built-in
• Windows options:
  • Windows 10+
    • Command Prompt and PowerShell programs have ssh and scp (may require latest Windows updates)
      • Start menu → Search for Command
    • Putty – http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
      
      • simple Terminal and file copy programs
      • download either the Putty installer or just putty.exe (Terminal) and pscp.exe (secure copy client)
    • Windows Subsystem for Linux – Windows 10 Professional includes a Ubuntu-like bash shells
      • See https://docs.microsoft.com/en-us/windows/wsl/install-win10
      • We recommend the Ubuntu Linux distribution, but any Linux distribution will have an SSH client

Use ssh (secure shell) to login to a remote computers.

# General form:
ssh <user_name>@<full_host_name>

# For example
ssh abattenh@ls6.tacc.utexas.edu

The bash shell REPL and commands

When you type something in at a bash command-line prompt, it Reads the input, Evaluates it, then Prints the results, then does this over and over in a Loop. This behavior is called a REPL – a Read, Eval, Print Loop. The shell executes the command line input when it sees a linefeed, which happens when you press Enter after entering the command.

The input to the bash REPL is a command, which consists of:

• The command name (any of the built-in Linux/Unix commands, or the name of a user-written script or program)
  
• One or more (optional) options, usually noted with a leading dash (-) or double-dash (--).
  • short (1-character) options can be provided separately, prefixed by a single dash (-)
    • or can be combined with the combination prefixed by a single dash
  • long (multi-character or "word") options are prefixed with a double dash (--) and must be supplied separately.
  • Both long and short options can be assigned a value
• One or more command-line arguments, which are often (but not always) file names

Some examples using the ls (list files) command:

ls               # example 1 - no options or arguments
ls -l            # example 2 - one "short" (single character) option only (-l)
ls --help        # example 3 - one "long" (word) option (--help)
ls .profile      # example 4 - one argument, a file name (.profile)
ls --width=20    # example 5 - a long option that has a value (--width is the option, 20 is the value)
ls -w 20         # example 6 - a short option w/a value, as above, where -w is the same as --width
ls -l -a -h      # example 7 - three short options entered separately (-l -a -h)
ls -lah          # example 8 - three short options that can be combined after a dash (-lah)

• The arguments to ls are one or more file or directory names. If no arguments are provided, the contents of the current directory are listed.
  • If an argument is a directory name, the contents of that directory are listed.
    

• Some handy options for ls:

  •  -l shows a long listing, including file permissions, ownership, size and last modified date.
  • -a shows all files, including dot files whose names start with a period ( . ) which are normally not listed
  • -h says to show file sizes in human readable form (e.g. 12M instead of 12201749)

A good place to start learning built-in Linux commands and their options is on the Some Linux commands page.

Getting help

How do you find out what options and arguments a command uses?

1. In the Terminal, type in the command name then the --help long option (e.g. ls --help)
   • Works for most Linux commands; 3rd party tools may use -h or -? or even /? instead
   • May produce a lot of output, so you may need to scroll up quite a bit, or pipe the output to a pager
     • e.g. ls --help | more (a space advances the output by one screen/"page", and typing Ctrl-C will exit more)
2. Use the built-in manual system (e.g. type man ls)
   • This system uses the less pager
   • For now, just know that a space advances the output by one screen/"page", and typing q will exit the display.
3. Ask the Google, e.g. search for ls man page
   • Can be easier to read

Many 3rd party tools, especially bioinformatics tools, may bundle a number of different functions into one command. For these tools, just typing in the command name then Enter may provide top-level usage information. For example, the bwa tool that aligns sequencing reads to a reference genome:

bwa

Produces something like this:

Program: bwa (alignment via Burrows-Wheeler transformation)
Version: 0.7.16a-r1181
Contact: Heng Li <lh3@sanger.ac.uk>

Usage:   bwa <command> [options]

Command: index         index sequences in the FASTA format
         mem           BWA-MEM algorithm
         fastmap       identify super-maximal exact matches
         pemerge       merge overlapping paired ends (EXPERIMENTAL)
         aln           gapped/ungapped alignment
         samse         generate alignment (single ended)
         sampe         generate alignment (paired ended)
         bwasw         BWA-SW for long queries

         shm           manage indices in shared memory
         fa2pac        convert FASTA to PAC format
         pac2bwt       generate BWT from PAC
         pac2bwtgen    alternative algorithm for generating BWT
         bwtupdate     update .bwt to the new format
         bwt2sa        generate SA from BWT and Occ

Note: To use BWA, you need to first index the genome with `bwa index'.
      There are three alignment algorithms in BWA: `mem', `bwasw', and
      `aln/samse/sampe'. If you are not sure which to use, try `bwa mem'
      first. Please `man ./bwa.1' for the manual.

bwa, like many bioinformatics programs, is written as a set of sub-commands. This top-level help displays the sub-commands available. You then type bwa <command> to see help for the sub-command:

bwa index

Displays something like this:

Usage:   bwa index [options] <in.fasta>

Options: -a STR    BWT construction algorithm: bwtsw or is [auto]
         -p STR    prefix of the index [same as fasta name]
         -b INT    block size for the bwtsw algorithm (effective with -a bwtsw) [10000000]
         -6        index files named as <in.fasta>.64.* instead of <in.fasta>.*

Warning: `-a bwtsw' does not work for short genomes, while `-a is' and

Of course Google works on 3rd party tools also (e.g. search for bwa manual)

Terminal input

Literal characters and metacharacters

In the bash shell, and in most tools and programming environment, there are two kinds of input:

• literal characters, that just represent (and print as) themselves
  • e.g. alphanumeric characters A-Z, a-z, 0-9
• metacharacters - these are special characters that are associated with an operation in the environment
  • e.g. the pound sign ( # ) comment character that tells the shell to ignore everything after the #
    

There are many metacharacters in bash:  # \ $ | ~ [ ]  to name a few.

Pay attention to the different metacharacters and their usages – which can depend on the context where they're used.

About command line input

You know the command line is ready for input when you see the command line prompt. It can be configured differently on different systems, but on our system it shows your account name, server name, current directory, then a dollar sign ($). Note the tilde character ( ~ ) signifies your Home directory.

The shell executes command line input when it sees a linefeed character (\n, also called a newline), which happens when you press Enter after entering the command.

More than one command can be entered on a single line – just separate the commands with a semi-colon ( ; ).

cd; ls -lh

A single command can also be split across multiple lines by adding a backslash ( \ ) at the end of the line you want to continue, before pressing Enter.

ls6:~$ ls ~/.bashrc \
> ~/.profile

Notice that the shell indicates that it is not done with command-line input by displaying a greater than sign ( > ). You just enter more text then Enter when done.

Text lines and the Terminal

Sometimes a line of text is longer than the width of your Terminal. In this case the text is wrapped. It can appear that the output is multiple lines, but it is not. For example, FASTQ files often have long lines:

head $CORENGS/misc/small.fq

Note that most Terminals let you increase/decrease the width/height of the Terminal window. But there will always be single lines too long for your Terminal width (and too many lines of text for its height).

So how long is a line? So how many lines of output are there really? And how long is a line? The wc (word count) command can tell us this.

• wc -l reports the number of lines in its input
• wc -c reports the number of characters in its input (including invisible linefeed characters)

And when you give wc -l multiple files, it reports the line count of each, then a total.

wc -l $CORENGS/misc/small.fq           # Reports the number of lines in the small.fq file
cat $CORENGS/misc/small.fq | wc -l     # Reports the number of lines on its standard input
wc -l $CORENGS/misc/*.fq               # Reports the number of lines in all matching *.fq files
tail -1 $CORENGS/misc/small.fq | wc -c # Reports the number of characters of the last small.fq line 

Command input errors

You don't always type in commands, options and arguments correctly – you can misspell a command name, forget to type a space, specify an unsupported option or a non-existent file, or make all kinds of other mistakes.

What happens? The shell attempts to guess what kind of error it is and reports an appropriate error message as best it can. Some examples:

# You mis-type a command name, or a command not installed on your system
ls6:~$ catt
catt: command not found

# You try to use an unsupported option
ls6:~$ ls -z
ls: invalid option -- 'z'
Try 'ls --help' for more information.

# You specify the name of a file that does not exist
ls6:~$ ls xxx
ls: cannot access 'xxx': No such file or directory

# You try to access a file or directory you don't have permissions for
ls6:~$ cat /etc/sudoers
cat: /etc/sudoers: Permission denied

Getting around in the shell

Type as little and as accurately as possible by using keyboard shortcuts!

Command line history and editing

Sometimes you want to repeat a command you've entered before, possibly with some changes.

• The built-in history command lists the commands you've entered, each with a number.
  • You can re-execute any command in the history by typing an exclamation point ( ! ) then the number
  • e.g. !15 re-executes the 15th command in your history.

• Use Up arrow to retrieve any of the last 50+ commands you've typed, going backwards through your history.
  
  • You can then edit the retrieved line, and hit Enter (even in the middle of the command), and the shell will use that command.
• The Down arrow "scrolls" forward from where you are in the command history.

The command line cursor (small thick bar on the command line) marks where you are on the command line.

• Right arrow and Left arrow move the cursor forward or backward on the current command line.
• Use Ctrl-a (holding down the Control key and a) to jump the cursor to the start of the line.
• Use Ctrl-e to jump the cursor to the end of the line.
• Arrow keys are also modified by Ctrl- (Windows) or Option- (Mac)
  
  • Ctrl-right-arrow (Windows) or Option-right-arrow (Mac) will skip by "word" forward
  • Ctrl-left-arrow (Windows) or Option-left-arrow (Mac) will skip by "word" backward

Once the cursor is positioned where you want it:

• Just type in any additional text you want
• To delete text after the cursor, use: Ctrl-d or:
  • Delete key on Windows
  • Function-Delete keys on Macintosh
• To delete text before the cursor, use, use: Ctrl-h or:
  • Backspace key on Windows
  • Delete key on Macintosh
• Use Ctrl-k (kill) to delete everything on the line after the cursor
• Use Ctrl-y (yank) to copy the last killed text to where the cursor is

Tab key completion

Hitting Tab when entering command line text invokes shell completion, instructing the shell to try to guess what you're doing and finish the typing for you. It's almost magic!

On most modern Linux shells you use Tab completion by pressing:

• single Tab – completes file or directory name up to any ambiguous part
  • if nothing shows up, there is no unambiguous match
• Tab twice – display all possible completions
  
  • you then decide where to go next
• shell completion works for commands too (like bowtie)

Absolute and relative pathname syntax

An absolute pathname lists all components of the full file system hierarchy that describes a file. Absolute paths always start with the forward slash ( / ), which is the root of the file system hierarchy. Directory names are separated by the forward slash ( / ) .

You can also specify a directory relative to where you are using one of the special directory names:

1. single period ( . ) means "the current directory"
2. two periods ( . . ) means "directory above the current"
3. tilde ( ~ )  means "my Home directory"

Avoid special characters in filenames

Pathname wildcards

The shell has shorthand to refer to groups of files by allowing wildcards in file names.

Using these wildcards  is sometimes called filename globbing, and the pattern a glob.

• asterisk ( * ) is the most common filename wildcard. It matches any length of any characters
• brackets ( [ ] ) match any character between the brackets
  
  • and you can use a hyphen ( - ) to specify a range of characters (e.g. [A-G])
• braces ( {  } ) enclose a list of comma-separated strings to match (e.g. {dog,pony})

For example:

• ls *.bam – lists all files in the current directory that end in .bam
• ls [A-Z]*.bam – does the same, but only if the first character of the file is a capital letter
• ls [ABcd]*.bam – lists all .bam files whose 1st letter is A, B, c or d.
• ls *.{fastq,fq}.gz – lists all .fastq.gz and .fq.gz files.

Streams and Piping

Standard streams and redirection

Most Linux commands write their results to standard output, a built-in stream that is mapped to your Terminal, but that data can be redirected to a file instead.

In fact every Linux command and program has three standard Unix streams:  standard input, standard output and standard error. Each has a number, a name, and redirection syntax:

• standard output is stream 1
  • redirect standard output to a file with a the > or 1> redirection operator
    • a single > or 1> overwrites any existing data in the target file
    • a double >> or 1>> appends to any existing data in the target file
• standard error is stream 2
  • redirect standard error to a file with a the 2> redirection operator
    • a single 2> overwrites any existing data in the target file
    • a double 2>> appends to any existing data in the target file

It is easy to not notice the difference between standard output and standard error when you're in an interactive Terminal session – because both outputs are sent to the Terminal window. But they are separate streams, with different meanings. In particular, programs write error and/or diagnostic messages to standard error, not to standard output.

Here's a command that shows the difference between standard error and standard output:

ls /etc/fstab xxx.txt

Produces this output in your Terminal:

ls: cannot access 'xxx.txt': No such file or directory
/etc/fstab

What is not obvious, since both streams are displayed on the Terminal, is that:

• the diagnostic text "ls: cannot access 'xxx.txt': No such file or directory" is being written to standard error
• the listing of the existing file ("/etc/passwd") is being written to standard output

To see this, redirect standard output and standard error to different files and look at their contents:

ls /etc/fstab xxx.txt 1> stdout.txt 2>stderr.txt
cat stdout.txt   # Displays "/etc/fstab"
cat stderr.txt   # Displays "ls: cannot access 'xxx.txt': No such file or directory"

What if you want both standard output and standard error to go to the same file? You use this somewhat odd 2>&1 redirection syntax:

# Redirect both standard output and standard error to the out.txt file
ls /etc/fstab xxx.txt > out.txt 2>&1

# Display the contents of the out.txt file
cat out.txt

# produces output like this:
ls: cannot access 'xxx.txt': No such file or directory
/etc/fstab

Two final notes.

• When standard output is redirected to a file, the data is not displayed on the Terminal
  
  • If you want the data written to both standard output (the Terminal) and a file, use the tee command
  • e.g. ls -l ~ | tee home_dir_listing.log
• There is a special Linux file called /dev/null that serves as a "global trash can" – it just throws away anything you write to it.
  • So you can direct standard output and/or standard error to /dev/null to ignore it completely.

When running batch programs and scripts you will want to manipulate standard output and standard error from programs appropriately – especially for 3rd party programs that often produce both results data and diagnostic/progress messages.

Piping

Most programs/commands read input data from some source, then write output to some destination. A data source can be a file, but can also be standard input. Similarly, a data destination can be a file but can also be a stream such as standard output.

The power of the Linux command line is due in no small part to the power of piping. The pipe operator ( | ) connects one program's standard output to the next program's standard input.

A simple example is piping uncompressed data "on the fly" to count its lines using wc -l (word count command with the lines option).

# zcat is like cat, except that it understands the gz compressed format,
# and uncompresses the data before writing it to standard output.
# So, like cat, you need to be sure to pipe the output to a pager if
# the file is large.
zcat big.fq.gz | wc -l

piping a histogram

But the real power of piping comes when you stitch together a string of commands with pipes – it's incredibly flexible, and fun once you get the hang of it.

For example, here's a simple way to make a histogram of mapping quality values from a subset of BAM file records.

# create a histogram of mapping quality scores for the 1st 1000 mapped bam records
samtools view -F 0x4 small.bam | head -1000 | cut -f 5 | sort -n | uniq -c

• samtools view converts the binary small.bam file to text and writes alignment record lines one at a time to standard output.
  • -F 0x4 option says to filter out any records where the 0x4 flag bit is 0 (not set)
  • since the 0x4 flag bit is set (1) for unmapped records, this says to only report records where the query sequence did map to the reference
• | head -1000
  • the pipe connects the standard output of samtools view to the standard input of head
  • the -1000 option says to only write the first 1000 lines of input to standard output
    
• | cut -f 5
  • the pipe connects the standard output of head to the standard input of cut
  • the -f 5 option says to only write the 5th field of each input line to standard output (input fields are tab-delimited by default)
    
    • the 5th field of an alignment record is an integer representing the alignment mapping quality
    •  the resulting output will have one integer per line (and 1000 lines)
• | sort -n
  • the pipe connects the standard output of cut to the standard input of sort
  • the -n option says to sort input lines according to numeric sort order
  • the resulting output will be 1000 numeric values, one per line, sorted from lowest to highest
    
• | uniq -c
  
  • the pipe connects the standard output of sort to the standard input of uniq
  • the -c option option says to just count groups of lines with the same value (that's why they must be sorted) and report the total for each group
  • the resulting output will be one line for each group that uniq sees
  • each line will have the text for the group (here the unique mapping quality values) and a count of lines in each group
    

Viewing text in files

cat, more or less

The most basic way of view file data is the cat command. While the name comes from its ability to concatenate one or more files, it can be used to output the contents of a single file. For example:

cat ~/.profile

# or, to see line numbers in the output:
cat -n ~/.profile

Using cat by itself is fine for small files, but it reads/writes everything in the file without stopping. So for larger files you use a pager such as more, or less. A pager reads text and outputs only one "page" of text at a time, then waits for you to ask it to advance. And a "page" of text is the number of lines that will fit on your visible Terminal. 

Using the more pager:

more ~/.bashrc

• Press the spacebar to see the next page.

• If there is additional output, you'll see the  --More-- indicator again; if not, the command prompt appears again.

• To end the more display, just type q (quit) or Ctrl-c.

Using the less pager:

less ~/.bashrc

# to see line numbers in the output:
less -N ~/.bashrc

# to use case-insensitive matching:
less -I ~/.bashrc

Basic navigation in less:

• Use q to quit less at any time
• space or Ctrl-f advances one page forward; Ctrl-b goes back one page
• down arrow goes down (forward) one line; up arrow goes up (backward) one line

Searching in less:

• /<pattern> – search for <pattern> in forward direction
  • n – goes to the next match of <pattern>
  • N – goes to the previous match of <pattern>
• ?<pattern> – search for <pattern> in backward direction
  • n – previous match going back
  • N – next match going forward

Introducing grep

Another method of text searching is using the grep program, which stands for general regular expression parser. In Unix, the grep program performs regular-expression text searching, and displays lines where the pattern text is found.

Nearly every programming language offers grep functionality, where a pattern you specify – a regular expression or regex – describes how the search is performed. 

There are many grep regular expression metacharacters that control how the search is performed (see the grep command).  

Basic usage is:  grep '<pattern>' <file> where

• '<pattern>' (usually enclosed in single quotes) just contains alphanumeric characters (A-Z, a-z, 0-9).

Common options:

• grep -i will perform a case-insensitive search
• grep -n will display line numbers where the pattern was matched

head and tail

Two other commands that are useful for viewing text are head and tail.

• With no options, head shows the first 10 lines of its input and tail shows the last 10 lines.
• Use the -n option followed by a number to specify how many lines to view']
  • or just put the number you want after a dash (e.g. -5 for 5 lines or -1 for 1 line)
• use the tail -n +<integer> syntax to display all input starting from that line

Examples:

head ~/.bashrc           # view the 1st 10 file lines
head -n 2 ~/.bashrc      # view the 1st 2 file lines
head -5 ~/.bashrc        # view the 1st 5 file lines

tail ~/.bashrc           # view the last 10 file lines
tail -n 3 ~/.bashrc      # view the last 3 file lines
tail -1 ~/.bashrc        # view the last line of the file

# view 7 lines of text starting at line 20
tail -n +20 ~/.bashrc | head -7

Since head and tail do not have an option to display line numbers, you can pipe in text that includes line numbers with cat -n:

cat -n ~/.bashrc | head -4       # view the 1st 4 lines w/line numbers
cat -n ~/.bashrc | tail -5       # view the last 5 lines w/line numbers

# view 6 lines of text starting at line 25
cat -n ~/.bashrc | tail -n +25 | head -6

More Linux concepts

Environment variables

Environment variables are just like variables in a programming language (in fact bash is a complete programming language), they are "pointers" that reference data assigned to them. In bash, you assign an environment variable as shown below:

export varname="Some value, here it's a string"

You set environment variables using the bare name (varname above).

You then refer to or evaluate an environment variable using a dollar sign ( $ ) evaluation operator before the name:

echo $varname

The export keyword when you're setting ensures that any sub-processes that are invoked will inherit this value. Without the export only the current shell process will have that variable set.

Use the env command to see all the environment variables you currently have set.

Quoting in the shell

What different quote marks mean in the shell and when to use can be quite confusing.

When the shell processes a command line, it first parses the text into tokens ("words"), which are groups of characters separated by whitespace (one or more space characters). Quoting affects how this parsing happens, including how metacharacters are treated and how text is grouped.

There are three types of quoting in the shell:

1. single quoting (e.g. 'some text') – this serves two purposes
   • It groups together all text inside the quotes into a single token
   • It tells the shell not to "look inside" the quotes to perform any evaluations
     
     • all metacharacters inside the single quotes are ignored
     • in particular, any environment variables in single-quoted text are not evaluated
2. double quoting (e.g. "some text") – also serves two purposes
   • it groups together all text inside the quotes into a single token
   • it allows environment variable evaluation, but inhibits some metacharcters
     • e.g. asterisk ( * ) pathname globbing and some other metacharacters
       
   • double quoting also preserves any special characters in the text
     • e.g. newlines (\n) or Tabs (\t)
3. backtick quoting (e.g. `date`)
   
   • evaluates the expression inside the backtick marks ( ` ` )
   • the standard output of the expression replaces the text inside the backtick marks ( ` ` )
   • the syntax $( date ) is equivalent

The quote characters themselves ( '  "  ` ) are metacharacters that tell the shell to "start a quoting process" then "end a quoting process" when the matching quote is found. Since they are part of the processing, the enclosing quotes are not included in the output.

single and double quotes

The first rule of quoting is: always enclose a command argument in quotes if it contains spaces so that the command sees the quoted text as one item. In particular, always use single ( ' ) or double ( " ) quotes when you define an environment variable whose value contains spaces.

foo='Hello world'   # correct - defines variable "foo" to have value "Hello world"
foo=Hello world     # error - no command called "world"

These two expressions using double quotes or single quotes are different because the single quotes tell the shell to treat the quoted text as a literal, and not to look inside it for metacharacter processing.

# Inside double quotes, the text "$USER" is evaluated and its value substituted
echo "my account name is $USER"  

# Inside single quotes, the text "$USER" is left as-is  
echo 'the environment variable storing my account name is $USER'

# Inside double quotes, use a backslash ( \ ) to escape the dollar sign ( $ ) metacharacter 
echo "the environment variable storing my account name is \$USER"

backtick quoting and sub-shell evaluation

backtick ( ` ` ) evaluation quoting is one of the underappreciated wonders of Unix. The shell:

• evaluates the expression/command inside the backtick marks ( ` ` )
• the standard output of the expression replaces the text inside the backticks

An example, using the date function that just writes the current date and time to standard output, which appears on your Terminal.

date          # Calling the date command just displays date/time information
echo date     # Here "date" is treated as a literal word, and written to standard output
echo `date`   # The date command is evaluated and its standard output replaces `date`

A slightly different syntax, called sub-shell evaluation, also evaluates the expression inside $( ) and replaces it with the expression's standard output.

today=$( date );          echo $today  # environment variable "today" is assigned today's date
today="Today is: `date`"; echo $today  # "today" is assigned a string including today's date

What is text?

So what exactly is text? That is, what is stored in files that the shell interprets as text?

On standard Unix systems, each text character is stored as one byte – eight binary bits – in a format called ASCII (American Standard Code for Information Interchange). Eight bits can store 2^{^8} = 256 values, numbered 0 - 255.

In its original form values 0 - 127 were used for standard ASCII characters. Now values 128 - 255 comprise an Extended set. See https://www.asciitable.com/

However not all ASCII "characters" are printable -- in fact the "printable" characters start at ASCII 32 (space).

ASCII values 0 - 31 have special meanings. Many were designed for use in early modem protocols, such as EOT (end of transmission) and ACK (acknowledge), or for printers, such as VT (vertical tab) and FF (form feed).

The non-printable ASCII characters we care most about are:

• Tab (decimal 9, hexadecimal 0x9, octal 0o011)
  • backslash escape: \t
• Linefeed/Newline (decimal 10, hexadecimal 0xA, octal 0o012)
  • backslash escape: \n
• Carriage Return (decimal 13, hexadecimal 0xD, octal 0o015)
  • backslash escape: \r

Let's use the hexdump command (really an alias, defined in your ~/.bashrc login script) to look at the actual ASCII codes stored in a file:

tail ~/.bashrc | hexdump

This will produce output something like this:

Each line here describes 16 characters, in three display areas:

• The numeric offset of the 16-character line, in hexadecimal (base 16)
  
  • 16 decimal is 0x10 hex
• The numeric value (ASCII code) for each character, again in hexadecimal
  • each 2-digit hex number represents one 8-bit byte/character
• The translated text, written between a greater than ( > ) and less than ( < ) sign
  
  • The display character associated with each ASCII code, or a period ( . ) for non-printable characters

Notice that spaces are ASCII 0x20 (decimal 32), and the newline characters appear as 0x0a (decimal 10).

Why hexadecimal? Programmers like hexadecimal (base 16) because it is easy to translate hex digits to binary, which is how everything is represented in computers. And it can sometimes be important to know which binary bits are 1s and which are 0s. (Read more about Decimal and Hexadecimal)

Writing multiple text lines

There are several ways to output multi-line text. You can:

• Start the text with a single quote or a double quote
  • press Enter when you want to start a new line
  • keep entering text and Enter until you're satisfied
  • supply the matching single quote or a double quote then Enter

  • example:

echo 'My
name is
Anna'

• Use echo -e
  
  • The -e option tells echo to replace some special backslash escapes characters that represent non-printable characters with their associated ASCII codes
    
    • So \n will be replaced by a newline (linefeed) character and \t will be replaced by a Tab.

  • example:

echo -e "My\nname is\nAnna"

heredoc

Another method for writing multi-line text that can be useful for composing a large block of text in a script, is the heredoc syntax, where a block of text is specified between two user-supplied block delimiters, and that text block is sent to a command. The general form of a heredoc is:

COMMAND << DELIMITER
..text...
..text...
DELIMITER

For example, using the (arbitrary) delimiter EOF and the cat command:

cat << EOF
This text will be output
And this USER environment variable will be evaluated: $USER
EOF

Here the block of text provided to cat is just displayed on the Terminal. To write it to a file just use the 1> or > redirection syntax in the cat command:

cat 1> out.txt << EOF
This text will be output
And this USER environment variable will be evaluated: $USER
EOF

The out.txt file will then contain this text:

This text will be output
And this USER environment variable will be evaluated: student01

Arithemetic in bash

Arithmetic in bash is very weird:

echo $(( 50 * 2 + 1 ))

n=0
n=$(( $n + 5 ))
echo $n

And it only returns integer values, after truncation.

echo $(( 4 / 2 ))
echo $(( 5 / 2 ))

echo $(( 24 / 5 ))

As a result, if I need to do anything other than the simplest arithemetic, I use awk:

awk 'BEGIN{print 4/2}'
echo 3 2 | awk '{print ($1+$2)/2}'

You can also use the printf function in awk to control formatting. Just remember that a linefeed ( \n ) has to included in the format string:

echo 3.1415926 | awk '{ printf("%.2f\n", $1) }'

You can even use it to convert a decimal number to hexadecimal using the %x printf format specifier. Note that the convention is to denote hexadecimal numbers with an initial 0x.

echo 65 | awk '{ printf("0x%x\n", $1) }'

Bash control flow

the bash for loop

As in many programming languages, a for loop performs a series of expressions on one or more item in the for's argument list.

The bash for loop has the general structure:

for <variable_name> in <list of space-separated items>
do <something>
   <somthing else>
done

The <items> should be (or evaluate to) for's argument list: a space-separated list of items (e.g. 1 2 3 4 or `ls -1 *.gz` ).

for num in `seq 4`
do 
  echo $num
done

# or, since bash lets you put multiple commands on one line 
# if they are each separated by a semicolon ( ; )
for num in `seq 4`; do echo $num; done

Gory details:

• The `seq 4` expression uses backtick evaluation to generate a set of 4 numbers: 1 2 3 4.
• The do/done block expressions are executed once for each of the items in the list
• Each time through the loop (the do/done block) the variable named num is assigned one of the values in the list
  
  • Then the value can be used by referencing the variable using $num
  • The variable name num is arbitrary – it can be any name we choose

processing multiple files in a for loop

One common use of for loops is to process multiple files, where the set of files to process is obtained by pathname wildcarding. For example, the code below counts the number of reads in a set of compressed FASTQ files:

for fname in *.gz; do
   echo "$fname has $((`zcat $fname | wc -l` / 4)) sequences"
done

quotes matter

We saw how double quotes allow the shell to evaluate certain metacharacters in the quoted text.

But more importantly when assigning multiple lines of text to a variable, quoting the evaluated variable preserves any special characters in the variable value's text such as Tab or newline characters.

Consider this case where a captured string contains newlines, as illustrated below.

txt=$( echo -e "aa\nbb\ncc" )
echo "$txt"   # inside double quotes, newlines preserved
echo $txt     # without double quotes, newlines are converted to spaces

This difference is very important!

• you do want to preserve newlines when processing one line of text at a time
• you do not want to preserve newlines when specifying the list of values a for loop processes (which must all be on one line)

See the difference:

nums=$( seq 5 )
echo $nums
echo "$nums"

echo $nums| wc -l     # newlines converted to spaces, so only one line
echo "$nums" | wc -l  # newlines preserved, so reports 5

# This loop prints a line for each of the files
for n in $nums; do
  echo "the number is: '$n'"
done

# But this loop prints only one line
for n in "$nums"; do
  echo "the number is: '$n'"
done

the if statement

The general form of an if/then/else statement in bash is:

if [ <test expression> ]
then <expression> [ expression... ]
else <expression> [ expression... ]
fi

Where

• The <test expression> is any expression that evaluates to true or false
  • In the shell, the number 0 (or an empty value) is false
  • Anything else is true
  • There must be at least one space around the <test expression> separating it from the enclosing bracket [ ].
  • Double brackets [[  ]] can also be used to enclose the <test expression>
• When the <test expression> is true the then expressions are evaluated.
• When the <test expression> is false the else expressions are evaluated.

A simple example:

for val in 5 0 "27" "$emptyvar" abc '0'; do
  if [ "$val" ]
    then echo "Value '$val' is true"
    else echo "Value '$val' is false"
  fi
done

A good reference on the many built-in bash conditionals: https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html

reading file lines with while

The read function can be used to read input one line at a time, in a bash while loop.

While the full details of the read commad are complicated (see https://unix.stackexchange.com/questions/209123/understanding-ifs-read-r-line) this read-a-line-at-a-time idiom works nicely. 

while IFS= read line; do
  echo "Line: '$line'"
done < ~/.bashrc

• The IFS= clears all of read's default input field separators, which is normally whitespace (one or more spaces or Tabs).
  
  • This is needed so that read will set the line variable to exactly the contents of the input line, and not strip leading whitespace from it.
• The lines are redirected from ~/.bashrc to the standard input of the while loop by the < ~/.bashrc expression after the done keyword.

If the input data is well structured, its fields can be read directly into variables. Notice we can pipe all the output to more – or could redirect it to a file.

tail /etc/passwd | while IFS=':' read account x uid gid name shell
do 
  echo $account $name
done | more

File attributes

Consider a long listing of our Home directory.

There are 9 whitespace-separated columns in this long listing:

1. file permissions - a 10-character field
2. number of sub-components associated with a directory - rarely important
3. account name of the file owner
4. Unix group associated with the file
5. file size
6. last modification month
7. last modification day
8. last modification year, or last modification hour/minute if within the last year
9. file name

Notice I call everything a file, even directories. That's because directories are just a special kind of file – one that contains information about the directory's contents.

Owner and Group

A file's owner is the Unix account that created the file (here abattenh, me). That account belongs to one or more Unix groups, and the group associated with a file is listed in field 4.

The owner will always be a member of the Unix group associated with a file, and other accounts may also be members of the same group. G-801021 is one of the Unix groups I belong to at TACC. To see the Unix groups you belong to, just type the groups command.

Permissions

File permissions and information about the file type are encoded in that 1st 10-character field. Permissions govern who can access a file, and what actions they are allowed.

• character 1 describes the file type (d for directory, - for regular file, l for symbolic link)
• the remaining 9 characters are 3 sets of 3-character designations
  • characters 2-4: what the owning user account can do
  • characters 5-7: what other members of the associated Unix group can do
  • characters 8-19: what other non-group members (everyone) can do

Each of the 3-character sets describes if read ( r ) write ( w ) and execute ( x or s ) actions are allowed, or not allowed ( - ).

• read ( r ) access means file contents can be read, and copied
• write ( w ) access means a file's contents can be changed, and directory contents can be modified (files added or deleted)
• execute ( x or s )
  • for files, execute ( x ) means it is a program that can be called/executed
    • e.g. /usr/bin/ls, the file that performs the ls command
  • for directories, execute ( x ) means directory operations may be performed/executed
    • the directory can be listed and changed into

Examples:

ls -l ~/.bash_history

ls -l /usr/bin/ls

ls -l -d ~/local (-d says to list directory information, not directory contents)

Copying files between TACC and your laptop

Assume you want to copy the TACC file $SCRATCH/core_ngs/fastq_prep/small_fastqc.html back to your laptop/local computer. You must initiate the copy operation from your local computer rather than at TACC. Why? because the TACC servers have host names and IP addresses that are public in the Internet's Distributed Name Service (DNS) directory. But your local computer (in nearly all cases) does not have a published name and address.

First, on the TACC server figure out what the appropriate absolute path (a.k.a. full pathname) is.

cd $SCRATCH/core_ngs/fastq/prep
pwd -P

This will display something like /scratch/01063/abattenh/core_ngs/fastq_prep

For folks with Mac or Linux laptops or Windows 10+ users with scp available in the Command Prompt program (or Windows subsystem for Linux):

• Open a Terminal window on your local computer
• cd to the directory where you want the files
• Type something like the following, substituting your user name and absolute path:
  
  

scp abattenh@ls6.tacc.utexas.edu:/scratch/01063/abattenh/core_ngs/fastq_prep/small_fastqc.html .

Windows users can use the free WinSCP program (https://winscp.net/eng/index.php) if their Windows version does not support scp.

Editing files

There are three main approaches to editing Unix files:

1. Use a command-line program that lets you enter/edit text in a Terminal window (e.g. nano, vi/vim, emacs)
   1. nano is extremely simple and is a good choice as a first local text editor
      • warning: nano has a tendency to break long single lines into multiple lines
   2. vi and emacs are extremely powerful but also quite complex
      • emacs reference sheet: https://www.gnu.org/software/emacs/refcards/pdf/refcard.pdf
      • vi reference sheet: http://www.atmos.albany.edu/daes/atmclasses/atm350/vi_cheat_sheet.pdf
2. Use a text editor or IDE (Integrated Development Environment) program that runs on your local computer but has an SFTP (secure FTP) interface that lets you connect to a remote computer
   
   • E.g., Komodo IDE (Windows & Mac) or Notepad++ (Windows). Both are no-cost.
     
   • Once you connect to the remote host, you can navigate its directory structure and edit files.
   • When you open a file, its contents are brought over the network into the text editor's edit window, then saved back when you save the file.
3. Use software or protocols that allow you to "mount" remote server directories
   
   • Once mounted, the remote storage appears as a local volume/drive.
     • Then, you can use any text editor or IDE on your local computer to open/edit/save remote files.
   • Software programs that can mount remote data include ExpanDrive for Windows or Mac (costs $$, but has a free trial), TextWrangler for Mac.
   • Remote file system protocols include Samba (Windows, Mac) and NFS (Linux)

Knowing the basics of at least one Linux command-line text editor is useful for creating/editing small files, and we'll explore nano in this class. For editing larger files, you may find options #2 or #3 more useful.

nano

nano is a very simple editor available on most Linux systems. If you are able to ssh into a remote system, you can use nano there.

To invoke nano to edit a new or existing file just type nano <filename>. For example:

nano newfile.txt

You'll see the name of the file (if you supplied one) on the top line of the Terminal window.

Navigation and operations in nano are similar to those we discussed in Command line editing

You can just type in text, and navigate around using arrow keys (up/down/left/right). A couple of other navigation shortcuts:

• Ctrl-a - go to start of line
• Ctrl-e - go to end of line
• Arrow keys are also modified by Ctrl- (Windows) or Option- (Mac)
  
  • Ctrl-right-arrow (Windows) or Option-right-arrow (Mac) will skip by "word" forward
  • Ctrl-left-arrow (Windows) or Option-left-arrow (Mac) will skip by "word" backward

Once you've positioned the cursor where you want it, just type in your text.

To remove text:

• To delete text after the cursor, use Ctrl-d or:
  • Delete key on Windows
  • Function-Delete keys on Macintosh
• To delete text before the cursor, use Ctrl-h or:
  • Backspace key on Windows
  • Delete key on Macintosh
• Use Ctrl-k (kill) to delete everything on the line
  • This is different from Ctrl-k on the command line where it deletes everything after the cursor
• Use Ctrl-u (uncut) to paste the just-killed text at the cursor
  • Recall this operation is Ctrl-y (yank) for command line editing

Once you're satisfied with your edits:

• use Ctrl-o - write out the file
• use Ctrl-x - exit nano

These and other important nano operations are displayed in a menu at the bottom of the Terminal window. Note that the ^ character means Ctrl- in this menu.

emacs

emacs is a complex, full-featured editor available on most Linux systems.

To invoke emacs to edit a new or existing file just type:

emacs <filename>

Here's a reference sheet that list many commands: https://www.gnu.org/software/emacs/refcards/pdf/refcard.pdf. The most important are:

• Ctrl-x/Ctrl-s - write out the file
• Ctrl-x/Ctrl-c - exit emacs

You can just type in text, and navigate around using arrow keys. A couple of other navigation shortcuts:

• Ctrl-a - go to start of line
• Ctrl-e - go to end of line
  
  

Line ending nightmares

The dirty little secret of the computer world is that the three main "families" of computers – Macs, Windows and Linux/Unix – use different, mutually incompatible line endings.

• Linux/Unix uses linefeed ( \n )
• Windows uses carriage return followed by linefeed ( \r\n )
• some Mac programs use carriage return only ( \r )

And guess what? Most Linux programs don't work with files that have Windows or Mac line endings, and what's worse they give you bizarre error messages that don't give you a clue what's going on!

So whatever non-Linux text editor you use, be sure to adjust its "line endings" setting – and it better have one somewhere!

Komodo Edit for Mac and Windows

Komodo Edit is a free, full-featured text editor with syntax coloring for many programming languages and a remote file editing interface. It has versions for both Macintosh and Windows. Download the appropriate install image here.

Once installed, start Komodo Edit and follow these steps to configure it:

• Configure the default line separator for Unix
  • On the Edit menu select Preferences
  • Select the New Files Category
  • For Specify the end-of-line (EOL) indicator for newly created files select UNIX (\n)
  • Select OK
• Configure a connection to TACC
  • On the Edit menu select Preferences
  • Select the Servers Category
  • For Server type select SFTP
  • Give this profile the Name of lonestar6
    
  • For Hostname enter ls6.tacc.utexas.edu
  • Enter your TACC user ID for Username
  • Leave Port and Default path blank
  • Select OK

When you want to open an existing file at lonestar6, do the following:

• Select the File menu -> Open -> Remote File
  • Select your lonestar6 profile from the top Server drop-down menu
  • Once you log in, it should show you all the files and directories in your lonestar6 $HOME directory
• Navigate to the file you want and open it
  • Often you will use your ~/work or ~/scratch directory links to help you here

To create and save a new file, do the following:

• From the Komodo Edit Start Page, select New File
  • Select the file type (Text is good for commands files)
• Edit the contents
• Select the File menu -> Save As Other -> Remote File
  • Select your lonestar6 profile from the Server drop-down menu
  • Once you log in, it should show you all the files and directories in your lonestar6 $HOME directory
• Navigate to where you want the put the file and save it
  • Often you will use your ~/work or ~/scratch directory links to help you here

Rather than having to navigate around TACC's complex file system tree, it helps to use the symbolic links to those areas that we created in your Home directory.

Notepad++ for Windows

Notepad++ is an open source, full-featured text editor for Windows PCs (not Macs). It has syntax coloring for many programming languages (python, perl, bash), and a remote file editing interface.

If you're on a Windows PC download the installer here.

Once it has been installed, start Notepad++ and follow these steps to configure it:

• Configure the default line separator for Unix
  • In the Settings menu, select Preferences
  • In the Preferences dialog, select the New Document/Default Directory tab.
  • Select Unix in the Format section
  • Close
• Configure a connection to TACC
  • In the Plugins menu, select NppFTP, then select Focus NppFTP Window. The top bar of the NppFTP panel should become blue.
  • Click the Settings icon (looks like a gear), then select Profile Settings
  • In the Profile settings dialog click Add new
  • Call the new profile lonestar6
  • Fill in Hostname (ls6.tacc.utexas.edu) and your TACC user ID
  • Connection type must be SFTP
  • Close

To open the connection, click the blue (Dis)connect icon then select your lonestar6 connection. It should prompt for your password. Once you've authenticated, a directory tree ending in your Home directory will be visible in the NppFTP window. You can click the the (Dis)connect icon again to Disconnect when you're done.

Rather than having to navigate around TACC's complex file system tree, it helps to use the symbolic links to those areas that we created in your Home directory (~/work or ~/scratch).

Other bash resources

• Greg's Bash Guide
  • The Guide: http://mywiki.wooledge.org/BashGuide
  • FAQ: http://mywiki.wooledge.org/BashFAQ
  • Pitfalls: http://mywiki.wooledge.org/BashPitfalls
• Ryan's Tutorials on Bash Scripting: http://ryanstutorials.net/bash-scripting-tutorial/
• CBRS short course workshop wikis (developed by Anna)
  • Introduction to Unix: https://utexas.atlassian.net/wiki/display/CbrsIntroUnix/
  • Intermediate Unix: https://utexas.atlassian.net/wiki/display/CbrsIntermUnix/
  • You won't have access to the small compute cluster they use, but you can download the example files and manipulate them in your own Linux or Unix environment.