Part 1: The Bash shell and commands

What's going on in the Terminal

Once you've logged on, you're now at a Linux command line in your Home directory!

It looks as if you're running directly on the remote computer, but really there are two programs communicating:

1. your local Terminal

2. the remote Shell

There are many shell programs available in Linux, but the default is bash (Bourne-again shell).

Your Terminal is pretty "dumb" – just sending what you type over its Secure Sockets Layer (SSL) connection to the remote computer, then displaying the text sent back by the remote computer's shell. The real work is being done on the remote computer, by executable programs called by the bash shell (also called commands, since you call them on the command line).

Open

The bash REPL and commands

When you type something in at a bash command-line prompt, it Reads the input, Evaluates it, then Prints the results, then does this over and over in a Loop. This behavior is called a REPL – a Read, Eval, Print Loop.

Many programming language environments have REPLs – Python and R for example. The usual input to the bash REPL is a command, possibly with options and arguments.

Here are some examples of the ls command (list information about files) and some of its usage. Options to ls control how the information is listed, and arguments specify what is listed.

A command consists of:

• The command name – a program/tool/utility that performs a specific function

  • the command above is ls (list information about files)

  • a command can be any of the built-in Linux/Unix commands, or the name of a user-written script or program

• One or more options that control how the command performs its processing

  • usually noted with a leading dash (-) or double-dash (--).

  • -l in example 2 (long listing)

  • --help in example 3

  • options are optional – they do not have to be supplied (e.g. example 1 above)

• One or more command-line arguments that specify what the command acts on

  • arguments are often (but not always) file names

    • and are often (but not always) optional

  • e.g. haiku.txt in example 4

The shell executes the command line input when it sees a linefeed, which happens when you press Enter after entering the command.

Command options

The notes below apply to nearly all built-in Linux utilities, and to many 3rd party programs as well

• Short (1-character) options can be provided separately, prefixed by a single dash ( - )

  • or can be combined with the combination prefixed by a single dash (examples 7, 8)

• Long (multi-character/"word") options are prefixed with a double-dash ( -- ) and must be supplied separately.

• Many utilities have equivalent long and short options (e.g.  --width and -w above)

• Both long and short options can be assigned a value (examples 5, 6)

  • The short option and its value are usually separated by a space, but can also be run together (e.g. -w20)

  • Strictly speaking, the long option and its value should be separated by an equal sign (=) according to the POSIX standard (see https://en.wikipedia.org/wiki/POSIX). But many programs let you use a space as separator also.

Some handy options for ls:

•  -l shows a long listing, including file permissions, ownership, size and last modified date.

• -a shows all files, including dot files whose names start with a period ( . ) which are normally not listed

  • dot files (which can also be directories) are often configuration files, such as your .bashrc and .profile login scripts.

• -h says to show file sizes in human readable form (e.g. 12M instead of 12201749)

The arguments to ls are one or more file/directory names. If no arguments are provided, the contents of the current directory are listed. If an argument is a directory name, the contents of that directory are listed.

Exercise 1-1

What is the difference between typing just ls (then Enter), and ls -a (then Enter)?

What do you see when you enter ls -l?

How can you tell which entries are files and which are directories?

What is the file size of the file mobydick.txt in bytes? In Kilobytes (1,024s of bytes, denoted K)

Getting help

So how do you find out what options and arguments a command uses?

1. In the Terminal, type in the command name then the --help long option (e.g. ls --help)

   • Works for most Linux commands; 3rd party tools may use -h or -? or even /? instead

   • May produce a lot of output, so you either need to scroll up quite a bit, or pipe the output to a pager (e.g. ls --help | more; type q to quit or Spacebar for the next "page")

2. Use the built-in manual system (e.g. type man ls)

   • This system uses the less pager that we'll go over later.

   • For now, just know that Spacebar advances the output by one screen/"page"

     • and typing q will quit/exit the display.

3. Ask the Google, e.g. search for “ls man page”

   • Can be easier to read

Every Linux command has tons of options – most of which you'll never use! The trick is to start with the most commonly used options and build out from there. Then, if you need a command to do something special, check if there's an option already to do that.

A good place to start learning built-in Linux commands and their options is on our Some Linux commandsPreview page.

Exercise 1-2

How would you produce a long listing that is sorted by last modification time?

How would you change the sort order to show the oldest files first?

Third-party tool commands

Some 3rd party tools, especially bioinformatics tools, may bundle a number of different functions into one command. For these tools, just typing in the command name then Enter will provide top-level usage information. For example, the bwa tool that aligns sequencing reads to a reference genome:

Use the program name alone as a command to get help

Produces bwa top-level help information something like this:

bwa, like many bioinformatics programs, is written as a set of sub-commands. This top-level help displays the sub-commands available. You then type bwa <command> to see help for the sub-command:

Get help on bwa index:

Displays bwa help information on the index sub-command:

Of course Google also works on 3rd party tools (e.g. search for bwa manual). And these days ChatGPT and similar chatbots are also quite useful.

Command line history and editing

Sometimes you want to repeat a command you've entered before (since you last logged in), possibly with some changes.

• The built-in history command lists the commands you've entered, each with a number.

  • You can re-execute any command in the history by typing an exclamation point ( ! ) then the number

  • e.g. !15 re-executes the 15th command in your history.

• Use Up arrow to retrieve any of the last 50+ commands you've typed, going backwards through your history.

  • You can then edit the retrieved line, and hit Enter (even in the middle of the command), and the shell will use that command.

• The Down arrow "scrolls" forward from where you are in the command history.

The command line cursor (small thick bar on the command line) marks where you are on the command line.

• Right arrow and Left arrow move the cursor forward or backward on the current command line.

• Use Ctrl-a (holding down the Control key and a) to jump the cursor to the start of the line.

• Use Ctrl-e to jump the cursor to the end of the line.

• Arrow keys are also modified by Ctrl- (Windows) or Option- (Mac)

  • Ctrl-right-arrow (Windows) or Option-right-arrow (Mac) will skip by "words" forward

  • Ctrl-left-arrow (Windows) or Option-left-arrow (Mac) will skip by "words" backward

Once the cursor is positioned where you want it:

• Just type in any additional text you want

• To delete text after the cursor, use:

  • Ctrl-d or

    • Delete key on Windows

    • Function-Delete keys on Macintosh

• To delete text before the cursor, use:

  • Ctrl-h or

    • Backspace key on Windows

    • Delete key on Mac

• Use Ctrl-k (kill) to delete everything on the line after the cursor

Tab key completion

Hitting Tab when entering command line text invokes shell completion, instructing the shell to try to guess what you're doing and finish the typing for you. It's almost magic!

On most modern Linux shells you use Tab completion by pressing:

• single Tab – completes file or directory name up to any ambiguous part

  • if nothing shows up, there is no unambiguous match

• Tab twice – display all possible completions

  • you then decide where to go next

Let's have some fun with our friend the Tab key. Follow along if you can, as we use the Tab key to see the /stor/work/CBRS_unix/fastq path.

Tab key completion also works on commands! Type "bowtie" and Tab twice to see all the programs in the bowtie2 and bowtie tool suites.

About command line input

You know the command line is ready for input when you see the command line prompt. It can be configured differently on different systems, but on our system it shows your account name, server name, current directory, then a dollar sign ( $ ). Note the tilde character ( ~ ) signifies your Home directory.

Like everything in Unix, the command line has similarities to a text file. And in Unix, all text file "lines" are terminated by a linefeed character (\n, also called a newline).

As mentioned above, the shell executes command line input when it sees a linefeed, which happens when you press Enter after entering the command.

But you can enter more than one command on a single line – just separate the commands with a semi-colon ( ; ).

Multiple command on a line

You can also split a single command across multiple lines by adding a backslash ( \ ) at the end of the line you want to continue, before pressing Enter.

Split a command across multiple lines

Notice that the shell indicates that it is not done with command-line input by displaying a greater than sign ( > ). You just enter more text then Enter when done.

Literal characters and metacharacters

In the bash shell, and in most tools and programming environment, there are two kinds of input:

• literal characters, that just represent (and print as) themselves

  • e.g. alphanumeric characters A-Z, a-z, 0-9

• metacharacters - these are special characters that are associated with an operation in the environment

  • e.g. the Enter key that emits a linefeed character to end the current line

There are many metacharacters in bash:  # \ $ | ~ " ' [ ]  to name a few.

• Notice this list includes the backslash ( \ ) character we just used to continue command-line input on multiple Terminal lines.