/
Linux fundamentals

Linux fundamentals

This page should serve as a reference for the many "things Linux" we use in this course. It is by no means complete – Linux is **huge** – but offers introductions to many important topics.

See also this page, which provides lists of the most common Linux commands, by category, as well as their most useful options: Some Linux commands

Some Linux commands

  • This page provides lists of the most common Linux commands, by category, as well as their most useful options:

Terminal programs, shells and commands

You need a Terminal program in order to ssh to a remote computer.

  • Macs and Linux have a Terminal program built-in
  • Windows options:

Use ssh (secure shell) to login to a remote computers.

SSH to a remote computer
# General form:
ssh <user_name>@<full_host_name>

# For example
ssh abattenh@ls6.tacc.utexas.edu

The bash shell REPL and commands

When you type something in at a bash command-line prompt, it Reads the input, Evaluates it, then Prints the results, then does this over and over in a Loop. This behavior is called a REPL – a Read, Eval, Print Loop. The shell executes the command line input when it sees a linefeed, which happens when you press Enter after entering the command.

The input to the bash REPL is a command, which consists of:

  • The command name (any of the built-in Linux/Unix commands, or the name of a user-written script or program)
  • One or more (optional) options, usually noted with a leading dash (-) or double-dash (--).
    • short (1-character) options can be provided separately, prefixed by a single dash (-)
      • or can be combined with the combination prefixed by a single dash
    • long (multi-character or "word") options are prefixed with a double dash (--) and must be supplied separately.
    • Both long and short options can be assigned a value
  • One or more command-line arguments, which are often (but not always) file names

Some examples using the ls (list files) command:

ls               # example 1 - no options or arguments
ls -l            # example 2 - one "short" (single character) option only (-l)
ls --help        # example 3 - one "long" (word) option (--help)
ls .profile      # example 4 - one argument, a file name (.profile)
ls --width=20    # example 5 - a long option that has a value (--width is the option, 20 is the value)
ls -w 20         # example 6 - a short option w/a value, as above, where -w is the same as --width
ls -l -a -h      # example 7 - three short options entered separately (-l -a -h)
ls -lah          # example 8 - three short options that can be combined after a dash (-lah)
  • The arguments to ls are one or more file or directory names. If no arguments are provided, the contents of the current directory are listed.
    • If an argument is a directory name, the contents of that directory are listed.
  • Some handy options for ls:

    •  -l shows a long listing, including file permissions, ownership, size and last modified date.
    • -a shows all files, including dot files whose names start with a period ( . ) which are normally not listed
    • -h says to show file sizes in human readable form (e.g. 12M instead of 12201749)

A good place to start learning built-in Linux commands and their options is on the Some Linux commands page.

Getting help

How do you find out what options and arguments a command uses?

  1. In the Terminal, type in the command name then the --help long option (e.g. ls --help)
    • Works for most Linux commands; 3rd party tools may use -h or -? or even /? instead
    • May produce a lot of output, so you may need to scroll up quite a bit, or pipe the output to a pager
      • e.g. ls --help | more (a space advances the output by one screen/"page", and typing Ctrl-C will exit more)
  2. Use the built-in manual system (e.g. type man ls)
    • This system uses the less pager
    • For now, just know that a space advances the output by one screen/"page", and typing q will exit the display.
  3. Ask the Google, e.g. search for ls man page
    • Can be easier to read

Many 3rd party tools, especially bioinformatics tools, may bundle a number of different functions into one command. For these tools, just typing in the command name then Enter may provide top-level usage information. For example, the bwa tool that aligns sequencing reads to a reference genome:

Use the program name alone as a command to get help
bwa

Produces something like this:

bwa top-level help information
Program: bwa (alignment via Burrows-Wheeler transformation)
Version: 0.7.16a-r1181
Contact: Heng Li <lh3@sanger.ac.uk>

Usage:   bwa <command> [options]

Command: index         index sequences in the FASTA format
         mem           BWA-MEM algorithm
         fastmap       identify super-maximal exact matches
         pemerge       merge overlapping paired ends (EXPERIMENTAL)
         aln           gapped/ungapped alignment
         samse         generate alignment (single ended)
         sampe         generate alignment (paired ended)
         bwasw         BWA-SW for long queries

         shm           manage indices in shared memory
         fa2pac        convert FASTA to PAC format
         pac2bwt       generate BWT from PAC
         pac2bwtgen    alternative algorithm for generating BWT
         bwtupdate     update .bwt to the new format
         bwt2sa        generate SA from BWT and Occ

Note: To use BWA, you need to first index the genome with `bwa index'.
      There are three alignment algorithms in BWA: `mem', `bwasw', and
      `aln/samse/sampe'. If you are not sure which to use, try `bwa mem'
      first. Please `man ./bwa.1' for the manual.

bwa, like many bioinformatics programs, is written as a set of sub-commands. This top-level help displays the sub-commands available. You then type bwa <command> to see help for the sub-command:

Get help on bwa index
bwa index

Displays something like this:

bwa top-level help information
Usage:   bwa index [options] <in.fasta>

Options: -a STR    BWT construction algorithm: bwtsw or is [auto]
         -p STR    prefix of the index [same as fasta name]
         -b INT    block size for the bwtsw algorithm (effective with -a bwtsw) [10000000]
         -6        index files named as <in.fasta>.64.* instead of <in.fasta>.*

Warning: `-a bwtsw' does not work for short genomes, while `-a is' and

Of course Google works on 3rd party tools also (e.g. search for bwa manual)

Terminal input

Literal characters and metacharacters

In the bash shell, and in most tools and programming environment, there are two kinds of input:

  • literal characters, that just represent (and print as) themselves
    • e.g. alphanumeric characters A-Z, a-z, 0-9
  • metacharacters - these are special characters that are associated with an operation in the environment
    • e.g. the pound sign ( # ) comment character that tells the shell to ignore everything after the #

There are many metacharacters in bash# \ $ | ~ [ ]  to name a few.

Pay attention to the different metacharacters and their usages – which can depend on the context where they're used.

About command line input

You know the command line is ready for input when you see the command line prompt. It can be configured differently on different systems, but on our system it shows your account name, server name, current directory, then a dollar sign ($). Note the tilde character ( ~ ) signifies your Home directory.

The shell executes command line input when it sees a linefeed character (\n, also called a newline), which happens when you press Enter after entering the command.

 Line ending differences...

Note: The Unix linefeed (\n) line delimiter is different from Windows, where the default line ending is carriage-return + linefeed (\r\n), and some Mac text editors that just use a carriage return (\r).

More than one command can be entered on a single line – just separate the commands with a semi-colon ( ; ).

Multiple command on a line
cd; ls -lh

A single command can also be split across multiple lines by adding a backslash ( \ ) at the end of the line you want to continue, before pressing Enter.

Split a command across multiple lines
ls6:~$ ls ~/.bashrc \
> ~/.profile

Notice that the shell indicates that it is not done with command-line input by displaying a greater than sign ( > ). You just enter more text then Enter when done.

Use Ctrl-C to exit the current command input

At any time during command input, whether on the 1st command line prompt or at a > continuation, you can press Ctrl-c (Control key and the c key at the same time) to get back to the command prompt.

Text lines and the Terminal

Sometimes a line of text is longer than the width of your Terminal. In this case the text is wrapped. It can appear that the output is multiple lines, but it is not. For example, FASTQ files often have long lines:

head $CORENGS/misc/small.fq

Note that most Terminals let you increase/decrease the width/height of the Terminal window. But there will always be single lines too long for your Terminal width (and too many lines of text for its height).

So how long is a line? So how many lines of output are there really? And how long is a line? The wc (word count) command can tell us this.

  • wc -l reports the number of lines in its input
  • wc -c reports the number of characters in its input (including invisible linefeed characters)

And when you give wc -l multiple files, it reports the line count of each, then a total.

wc -l $CORENGS/misc/small.fq           # Reports the number of lines in the small.fq file
cat $CORENGS/misc/small.fq | wc -l     # Reports the number of lines on its standard input
wc -l $CORENGS/misc/*.fq               # Reports the number of lines in all matching *.fq files
tail -1 $CORENGS/misc/small.fq | wc -c # Reports the number of characters of the last small.fq line 

Command input errors

You don't always type in commands, options and arguments correctly – you can misspell a command name, forget to type a space, specify an unsupported option or a non-existent file, or make all kinds of other mistakes.

What happens? The shell attempts to guess what kind of error it is and reports an appropriate error message as best it can. Some examples:

# You mis-type a command name, or a command not installed on your system
ls6:~$ catt
catt: command not found

# You try to use an unsupported option
ls6:~$ ls -z
ls: invalid option -- 'z'
Try 'ls --help' for more information.

# You specify the name of a file that does not exist
ls6:~$ ls xxx
ls: cannot access 'xxx': No such file or directory

# You try to access a file or directory you don't have permissions for
ls6:~$ cat /etc/sudoers
cat: /etc/sudoers: Permission denied

Getting around in the shell

Type as little and as accurately as possible by using keyboard shortcuts!

Command line history and editing

Sometimes you want to repeat a command you've entered before, possibly with some changes.

  • The built-in history command lists the commands you've entered, each with a number.
    • You can re-execute any command in the history by typing an exclamation point ( ! ) then the number
    • e.g. !15 re-executes the 15th command in your history.
  • Use Up arrow to retrieve any of the last 50+ commands you've typed, going backwards through your history.
    • You can then edit the retrieved line, and hit Enter (even in the middle of the command), and the shell will use that command.
  • The Down arrow "scrolls" forward from where you are in the command history.

The command line cursor (small thick bar on the command line) marks where you are on the command line.

  • Right arrow and Left arrow move the cursor forward or backward on the current command line.
  • Use Ctrl-a (holding down the Control key and a) to jump the cursor to the start of the line.
  • Use Ctrl-e to jump the cursor to the end of the line.
  • Arrow keys are also modified by Ctrl- (Windows) or Option- (Mac)
    • Ctrl-right-arrow (Windows) or Option-right-arrow (Mac) will skip by "word" forward
    • Ctrl-left-arrow (Windows) or Option-left-arrow (Mac) will skip by "word" backward

Once the cursor is positioned where you want it:

  • Just type in any additional text you want
  • To delete text after the cursor, use:
    • Delete key on Windows
    • Function-Delete keys on Macintosh
  • To delete text before the cursor, use:
    • Backspace key on Windows
    • Delete key on Macintosh
  • Use Ctrl-k (kill) to delete everything on the line after the cursor
  • Use Ctrl-y (yank) to copy the last killed text to where the cursor is

Tab key completion

Hitting Tab when entering command line text invokes shell completion, instructing the shell to try to guess what you're doing and finish the typing for you. It's almost magic!

On most modern Linux shells you use Tab completion by pressing:

  • single Tab – completes file or directory name up to any ambiguous part
    • if nothing shows up, there is no unambiguous match
  • Tab twice – display all possible completions
    • you then decide where to go next
  • shell completion works for commands too (like bowtie)

Absolute and relative pathname syntax

An absolute pathname lists all components of the full file system hierarchy that describes a file. Absolute paths always start with the forward slash ( / ), which is the root of the file system hierarchy. Directory names are separated by the forward slash ( / ) .

You can also specify a directory relative to where you are using one of the special directory names:

  1. single period ( . ) means "the current directory"
  2. two periods ( . . ) means "directory above the current"
  3. tilde ( ~ )  means "my Home directory"

Avoid special characters in filenames

While it is possible to create file and directory names that have embedded spaces, that creates problems when manipulating them.

To avoid headaches, it is best not to create file/directory names with embedded spaces, or with special characters such as + & # ( )

Pathname wildcards

The shell has shorthand to refer to groups of files by allowing wildcards in file names.

Using these wildcards  is sometimes called filename globbing, and the pattern a glob.

  • asterisk ( * ) is the most common filename wildcard. It matches any length of any characters
  • brackets ( [ ] ) match any character between the brackets
    • and you can use a hyphen ( - ) to specify a range of characters (e.g. [A-G])
  • braces ( {  } ) enclose a list of comma-separated strings to match (e.g. {dog,pony})

For example:

  • ls *.bam – lists all files in the current directory that end in .bam
  • ls [A-Z]*.bam – does the same, but only if the first character of the file is a capital letter
  • ls [ABcd]*.bam – lists all .bam files whose 1st letter is A, B, c or d.
  • ls *.{fastq,fq}.gz – lists all .fastq.gz and .fq.gz files.

Streams and Piping

Standard streams and redirection

Most Linux commands write their results to standard output, a built-in stream that is mapped to your Terminal, but that data can be redirected to a file instead.

In fact every Linux command and program has three standard Unix streamsstandard input, standard output and standard error. Each has a number, a name, and redirection syntax:

  • standard output is stream 1
    • redirect standard output to a file with a the > or 1> redirection operator
      • a single > or 1> overwrites any existing data in the target file
      • a double >> or 1>> appends to any existing data in the target file
  • standard error is stream 2
    • redirect standard error to a file with a the 2> redirection operator
      • a single 2> overwrites any existing data in the target file
      • a double 2>> appends to any existing data in the target file

It is easy to not notice the difference between standard output and standard error when you're in an interactive Terminal session – because both outputs are sent to the Terminal window. But they are separate streams, with different meanings. In particular, programs write error and/or diagnostic messages to standard error, not to standard output.

Here's a command that shows the difference between standard error and standard output:

ls /etc/fstab xxx.txt

Produces this output in your Terminal:

ls: cannot access 'xxx.txt': No such file or directory
/etc/fstab

What is not obvious, since both streams are displayed on the Terminal, is that:

  • the diagnostic text "ls: cannot access 'xxx.txt': No such file or directory" is being written to standard error
  • the listing of the existing file ("/etc/passwd") is being written to standard output

To see this, redirect standard output and standard error to different files and look at their contents:

ls /etc/fstab xxx.txt 1> stdout.txt 2>stderr.txt
cat stdout.txt   # Displays "/etc/fstab"
cat stderr.txt   # Displays "ls: cannot access 'xxx.txt': No such file or directory"

What if you want both standard output and standard error to go to the same file? You use this somewhat odd 2>&1 redirection syntax:

# Redirect both standard output and standard error to the out.txt file
ls /etc/fstab xxx.txt > out.txt 2>&1

# Display the contents of the out.txt file
cat out.txt

# produces output like this:
ls: cannot access 'xxx.txt': No such file or directory
/etc/fstab

Two final notes.

  • When standard output is redirected to a file, the data is not displayed on the Terminal
    • If you want the data written to both standard output (the Terminal) and a file, use the tee command
    • e.g. ls -l ~ | tee home_dir_listing.log
  • There is a special Linux file called /dev/null that serves as a "global trash can" – it just throws away anything you write to it.
    • So you can direct standard output and/or standard error to /dev/null to ignore it completely.

When running batch programs and scripts you will want to manipulate standard output and standard error from programs appropriately – especially for 3rd party programs that often produce both results data and diagnostic/progress messages.

Piping

Most programs/commands read input data from some source, then write output to some destination. A data source can be a file, but can also be standard input. Similarly, a data destination can be a file but can also be a stream such as standard output.

The power of the Linux command line is due in no small part to the power of piping. The pipe operator ( | ) connects one program's standard output to the next program's standard input.

A simple example is piping uncompressed data "on the fly" to count its lines using wc -l (word count command with the lines option).

Pipe uncompressed output to a pager
# zcat is like cat, except that it understands the gz compressed format,
# and uncompresses the data before writing it to standard output.
# So, like cat, you need to be sure to pipe the output to a pager if
# the file is large.
zcat big.fq.gz | wc -l

piping a histogram

But the real power of piping comes when you stitch together a string of commands with pipes – it's incredibly flexible, and fun once you get the hang of it.

For example, here's a simple way to make a histogram of mapping quality values from a subset of BAM file records.

The power of chaining pipes
# create a histogram of mapping quality scores for the 1st 1000 mapped bam records
samtools view -F 0x4 small.bam | head -1000 | cut -f 5 | sort -n | uniq -c
  • samtools view converts the binary small.bam file to text and writes alignment record lines one at a time to standard output.
    • -F 0x4 option says to filter out any records where the 0x4 flag bit is 0 (not set)
    • since the 0x4 flag bit is set (1) for unmapped records, this says to only report records where the query sequence did map to the reference
  • | head -1000
    • the pipe connects the standard output of samtools view to the standard input of head
    • the -1000 option says to only write the first 1000 lines of input to standard output
  • | cut -f 5
    • the pipe connects the standard output of head to the standard input of cut
    • the -f 5 option says to only write the 5th field of each input line to standard output (input fields are tab-delimited by default)
      • the 5th field of an alignment record is an integer representing the alignment mapping quality
      •  the resulting output will have one integer per line (and 1000 lines)
  • | sort -n
    • the pipe connects the standard output of cut to the standard input of sort
    • the -n option says to sort input lines according to numeric sort order
    • the resulting output will be 1000 numeric values, one per line, sorted from lowest to highest
  • | uniq -c
    • the pipe connects the standard output of sort to the standard input of uniq
    • the -c option option says to just count groups of lines with the same value (that's why they must be sorted) and report the total for each group
    • the resulting output will be one line for each group that uniq sees
    • each line will have the text for the group (here the unique mapping quality values) and a count of lines in each group

More Linux concepts

Environment variables

Environment variables are just like variables in a programming language (in fact bash is a complete programming language), they are "pointers" that reference data assigned to them. In bash, you assign an environment variable as shown below:

Set an environment variable
export varname="Some value, here it's a string"

Careful – do not put spaces around the equals sign when assigning environment variable values.

Also, always surround the value with double quotes ( " " ) if it contains (or might contain) spaces.

You set environment variables using the bare name (varname above).

You then refer to or evaluate an environment variable using a dollar sign ( $ ) evaluation operator before the name:

Refer to an environment variable
echo $varname

The export keyword when you're setting ensures that any sub-processes that are invoked will inherit this value. Without the export only the current shell process will have that variable set.

Use the env command to see all the environment variables you currently have set.

Quoting in the shell

What different quote marks mean in the shell and when to use can be quite confusing.

When the shell processes a command line, it first parses the text into tokens ("words"), which are groups of characters separated by whitespace (one or more space characters). Quoting affects how this parsing happens, including how metacharacters are treated and how text is grouped.

There are three types of quoting in the shell:

  1. single quoting (e.g. 'some text') – this serves two purposes
    • It groups together all text inside the quotes into a single token
    • It tells the shell not to "look inside" the quotes to perform any evaluations
      • all metacharacters inside the single quotes are ignored
      • in particular, any environment variables in single-quoted text are not evaluated
  2. double quoting (e.g. "some text") – also serves two purposes
    • it groups together all text inside the quotes into a single token
    • it allows environment variable evaluation, but inhibits some metacharcters
      • e.g. asterisk ( * ) pathname globbing and some other metacharacters
    • double quoting also preserves any special characters in the text
      • e.g. newlines (\n) or Tabs (\t)
  3. backtick quoting (e.g. `date`)
    • evaluates the expression inside the backtick marks ( ` ` )
    • the standard output of the expression replaces the text inside the backtick marks ( ` ` )
    • the syntax $( date ) is equivalent

The quote characters themselves ( '  "  ` ) are metacharacters that tell the shell to "start a quoting process" then "end a quoting process" when the matching quote is found. Since they are part of the processing, the enclosing quotes are not included in the output.

If you see the greater than ( > ) character after pressing Enter, it can mean that your quotes are not paired, and the shell is waiting for more input to contain the missing quote of the pair (either single or double). Just use Ctrl-c to get back to the prompt.

single and double quotes

The first rule of quoting is: always enclose a command argument in quotes if it contains spaces so that the command sees the quoted text as one item. In particular, always use single ( ' ) or double ( " ) quotes when you define an environment variable whose value contains spaces.

foo='Hello world'   # correct - defines variable "foo" to have value "Hello world"
foo=Hello world     # error - no command called "world"

These two expressions using double quotes or single quotes are different because the single quotes tell the shell to treat the quoted text as a literal, and not to look inside it for metacharacter processing.

# Inside double quotes, the text "$USER" is evaluated and its value substituted
echo "my account name is $USER"  

# Inside single quotes, the text "$USER" is left as-is  
echo 'the environment variable storing my account name is $USER'

To display a metacharacter as a literal inside double quotes, use the backslash ( \ ) character to escape the following character.

# Inside double quotes, use a backslash ( \ ) to escape the dollar sign ( $ ) metacharacter 
echo "the environment variable storing my account name is \$USER"

backtick quoting and sub-shell evaluation

backtick ( ` ` ) evaluation quoting is one of the underappreciated wonders of Unix. The shell:

  • evaluates the expression/command inside the backtick marks ( ` ` )
  • the standard output of the expression replaces the text inside the backticks

An example, using the date function that just writes the current date and time to standard output, which appears on your Terminal.

date          # Calling the date command just displays date/time information
echo date     # Here "date" is treated as a literal word, and written to standard output
echo `date`   # The date command is evaluated and its standard output replaces `date`

A slightly different syntax, called sub-shell evaluation, also evaluates the expression inside $( ) and replaces it with the expression's standard output.

today=$( date );          echo $today  # environment variable "today" is assigned today's date
today="Today is: `date`"; echo $today  # "today" is assigned a string including today's date

What is text?

So what exactly is text? That is, what is stored in files that the shell interprets as text?

On standard Unix systems, each text character is stored as one byteeight binary bits – in a format called ASCII (American Standard Code for Information Interchange). Eight bits can store 2^8 = 256 values, numbered 0 - 255.

In its original form values 0 - 127 were used for standard ASCII characters. Now values 128 - 255 comprise an Extended set. See https://www.asciitable.com/

However not all ASCII "characters" are printable -- in fact the "printable" characters start at ASCII 32 (space).

ASCII values 0 - 31 have special meanings. Many were designed for use in early modem protocols, such as EOT (end of transmission) and ACK (acknowledge), or for printers, such as VT (vertical tab) and FF (form feed).

The non-printable ASCII characters we care most about are:

  • Tab (decimal 9, hexadecimal 0x9, octal 0o011)
    • backslash escape: \t
  • Linefeed/Newline (decimal 10, hexadecimal 0xA, octal 0o012)
    • backslash escape: \n
  • Carriage Return (decimal 13, hexadecimal 0xD, octal 0o015)
    • backslash escape: \r

Let's use the hexdump command (really an alias, defined in your ~/.bashrc login script) to look at the actual ASCII codes stored in a file:

tail ~/.bashrc | hexdump

This will produce output something like this:

Each line here describes 16 characters, in three display areas:

  • The numeric offset of the 16-character line, in hexadecimal (base 16)
    • 16 decimal is 0x10 hex
  • The numeric value (ASCII code) for each character, again in hexadecimal
    • each 2-digit hex number represents one 8-bit byte/character
  • The translated text, written between a greater than ( > ) and less than ( < ) sign
    • The display character associated with each ASCII code, or a period ( . ) for non-printable characters

Notice that spaces are ASCII 0x20 (decimal 32), and the newline characters appear as 0x0a (decimal 10).

Why hexadecimal? Programmers like hexadecimal (base 16) because it is easy to translate hex digits to binary, which is how everything is represented in computers. And it can sometimes be important to know which binary bits are 1s and which are 0s. (Read more about Decimal and Hexadecimal)

Writing multiple text lines

There are several ways to output multi-line text. You can:

  • Start the text with a single quote or a double quote
    • press Enter when you want to start a new line
    • keep entering text and Enter until you're satisfied
    • supply the matching single quote or a double quote then Enter
    • example:

echo 'My
name is
Anna'
  • Use echo -e
    • The -e option tells echo to replace some special backslash escapes characters that represent non-printable characters with their associated ASCII codes
      • So \n will be replaced by a newline (linefeed) character and \t will be replaced by a Tab.
    • example:

echo -e "My\nname is\nAnna"

heredoc

Another method for writing multi-line text that can be useful for composing a large block of text in a script, is the heredoc syntax, where a block of text is specified between two user-supplied block delimiters, and that text block is sent to a command. The general form of a heredoc is:

COMMAND << DELIMITER
..text...
..text...
DELIMITER

The 2nd (ending) block delimiter you specify for a heredoc must appear at the start of a new line.

For example, using the (arbitrary) delimiter EOF and the cat command:

cat << EOF
This text will be output
And this USER environment variable will be evaluated: $USER
EOF

Here the block of text provided to cat is just displayed on the Terminal. To write it to a file just use the 1> or > redirection syntax in the cat command:

cat 1> out.txt << EOF
This text will be output
And this USER environment variable will be evaluated: $USER
EOF

The out.txt file will then contain this text:

This text will be output
And this USER environment variable will be evaluated: student01

Arithemetic in bash

Arithmetic in bash is very weird:

echo $(( 50 * 2 + 1 ))

n=0
n=$(( $n + 5 ))
echo $n

And it only returns integer values, after truncation.

echo $(( 4 / 2 ))
echo $(( 5 / 2 ))

echo $(( 24 / 5 ))

As a result, if I need to do anything other than the simplest arithemetic, I use awk:

awk 'BEGIN{print 4/2}'
echo 3 2 | awk '{print ($1+$2)/2}'

You can also use the printf function in awk to control formatting. Just remember that a linefeed ( \n ) has to included in the format string:

echo 3.1415926 | awk '{ printf("%.2f\n", $1) }'

You can even use it to convert a decimal number to hexadecimal using the %x printf format specifier. Note that the convention is to denote hexadecimal numbers with an initial 0x.

echo 65 | awk '{ printf("0x%x\n", $1) }'

Bash control flow

the bash for loop

As in many programming languages, a for loop performs a series of expressions on one or more item in the for's argument list.

The bash for loop has the general structure:

for <variable_name> in <list of space-separated items>
do <something>
  
<somthing else>
done

The <items> should be (or evaluate to) for's argument list: a space-separated list of items (e.g. 1 2 3 4 or `ls -1 *.gz` ).

for loop example
for num in `seq 4`
do 
  echo $num
done

# or, since bash lets you put multiple commands on one line 
# if they are each separated by a semicolon ( ; )
for num in `seq 4`; do echo $num; done

Gory details:

  • The `seq 4` expression uses backtick evaluation to generate a set of 4 numbers: 1 2 3 4.
  • The do/done block expressions are executed once for each of the items in the list
  • Each time through the loop (the do/done block) the variable named num is assigned one of the values in the list
    • Then the value can be used by referencing the variable using $num
    • The variable name num is arbitrary – it can be any name we choose

processing multiple files in a for loop

One common use of for loops is to process multiple files, where the set of files to process is obtained by pathname wildcarding. For example, the code below counts the number of reads in a set of compressed FASTQ files:

For loop to count sequences in multiple FASTQs
for fname in *.gz; do
   echo "$fname has $((`zcat $fname | wc -l` / 4)) sequences"
done

quotes matter

We saw how double quotes allow the shell to evaluate certain metacharacters in the quoted text.

But more importantly when assigning multiple lines of text to a variable, quoting the evaluated variable preserves any special characters in the variable value's text such as Tab or newline characters.

Consider this case where a captured string contains newlines, as illustrated below.

txt=$( echo -e "aa\nbb\ncc" )
echo "$txt"   # inside double quotes, newlines preserved
echo $txt     # without double quotes, newlines are converted to spaces

This difference is very important!

  • you do want to preserve newlines when processing one line of text at a time
  • you do not want to preserve newlines when specifying the list of values a for loop processes (which must all be on one line)

See the difference:

nums=$( seq 5 )
echo $nums
echo "$nums"

echo $nums| wc -l     # newlines converted to spaces, so only one line
echo "$nums" | wc -l  # newlines preserved, so reports 5

# This loop prints a line for each of the files
for n in $nums; do
  echo "the number is: '$n'"
done

# But this loop prints only one line
for n in "$nums"; do
  echo "the number is: '$n'"
done

the if statement

The general form of an if/then/else statement in bash is:

if [ <test expression> ]
then <expression> [ expression... ]
else <expression> [ expression... ]
fi

Where

  • The <test expression> is any expression that evaluates to true or false
    • In the shell, the number 0 (or an empty value) is false
    • Anything else is true
    • There must be at least one space around the <test expression> separating it from the enclosing bracket [ ].
    • Double brackets [[  ]] can also be used to enclose the <test expression>
  • When the <test expression> is true the then expressions are evaluated.
  • When the <test expression> is false the else expressions are evaluated.

A simple example:

for val in 5 0 "27" "$emptyvar" abc '0'; do
  if [ "$val" ]
    then echo "Value '$val' is true"
    else echo "Value '$val' is false"
  fi
done

A good reference on the many built-in bash conditionals: https://www.gnu.org/software/bash/manual/html_node/Bash-Conditional-Expressions.html

reading file lines with while

The read function can be used to r