The Intro Unix: The Bash shell and commands section introduced the bash REPL – a Read, Eval, Print Loop that processes lines of command input. To review, a command consists of:
The shell executes its REPL when it sees a linefeed (a.k.a newline), which happens when you press Enter after typing the command.
Types of command options:
More at: Intro Unix: The Bash shell and commands: Command options
To learn what options and arguments a command has:
More at:
Sometimes you want to repeat a command you've entered before, possibly with some changes.
To affect the cursor (small thick bar on the command line) that marks where you are on the command line.
Once the cursor is positioned where you want it:
For more on how to edit text on the command line, see: Intro Unix: The Bash shell and commands: Command line history and editing
Hitting Tab when entering command line text invokes shell completion, instructing the shell to try to guess what you're doing and finish the typing for you. It's almost magic!
On most modern Linux shells you use Tab completion by pressing:
Let's have some fun with our friend the Tab key. Follow along if you can, as we use the Tab key to see the /stor/work/CBRS_unix/fastq path.
ls /st # press Tab key - expands to /stor/ which
# is the only match
ls /stor/w # press Tab key again: expands to /stor/work/,
# again the only match
ls /stor/work/C # press Tab once - you hear a "bell" sound,
# and nothing is displayed because
# there are multiple matches
ls /stor/work/C # press Tab a 2nd time - all matching
# entries are listed
ls /stor/work/CB # press Tab key - expands to
# /stor/work/CBRS_unix
ls /stor/work/CBRS_unix/ # press Tab twice to see all completions
ls /stor/work/CBRS_unix/f # press Tab once - expands to
# /stor/work/CBRS_unix/fastq
|
Tab key completion also works on commands! Type "bowtie" and Tab twice to see all the programs in the bowtie2 and bowtie tool suites.
Like everything in Unix, the command line has similarities to a text file. And in Unix, all text file "lines" are terminated by a linefeed character (\n, also called a newline).
Note: The Unix linefeed (\n) line delimiter is different from Windows, where the default line ending is carriage-return + linefeed (\r\n), and some Mac text editors that just use a carriage return (\r). |
The shell executes command line input when it sees a linefeed, which happens when you press Enter after entering the command. But you can enter more than one command on a single line – just separate the commands with a semi-colon ( ; ).
ls -l haiku.txt; cat haiku.txt |
You can also split a single command across multiple lines by adding a backslash ( \ ) at the end of the line you want to continue, before pressing Enter. Just make sure there are no characters after the backslash.
student01@gsafcomp01:~$ ls haiku.txt \ > mobydick.txt |
The shell indicates that it is not done with command-line input by displaying a greater than sign ( > ). You can just enter more text then Enter when done. At any time during command input you can press Ctrl-c to get back to the command prompt. This is true whether you're entering a single command line prompt or at a > continuation.
For more information, see: Intro Unix: About command line input
In the bash shell, and in most tools and programming environment, there are two kinds of input:
There are many metacharacters in bash: # \ $ | ~ [ ] " ` ' { } to name a few.
We'll be emphasizing the different metacharacters and their usages – which can depend on the context where they're used – both in the bash command line and in commands/programs called from bash.
More at:
A key to text manipulation is understanding Unix streams, which represent flows of characters. Every Unix program has three "built-in" streams: standard input, standard output and standard error.
Most programs/commands read input data from some source, then write output to some destination.
The pipe operator ( | ) connects one program's standard output to the next program's standard input. The power of the Linux command line is due in no small part to the power of piping.
The key to the power of piping is that most Unix commands can accept input from standard input instead of from files. So, for example, these two expressions appear equivalent:
more jabberwocky.txt cat jabberwocky.txt | more |
The main differences between these two commands:
More at: Intro Unix: Viewing text in files: Standard streams and piping
The head and tail commands can be used to view/extract specific parts of large files.
Use the wc (word count) command to count text lines (wc -l) or characters (wc -c).
echo is the bash command to output text.
Examples:
head -n 5 haiku.txt # display the 1st 5 lines of
# "haiku.txt"
cat -n haiku.txt # display "haiku.txt" contents
# with line numbers
cat -n haiku.txt | tail -n 7 # display the last 7 lines of
# "haiku.txt"
cat -n haiku.txt | tail -n +7 # display text in "haiku.txt"
# starting at line 7
cat -n haiku.txt | tail +5 | head -3 # display the middle stanza of
# "haiku.txt" (lines 5-7)
wc -l haiku.txt # count lines in "haiku.txt" file
cat haiku.txt | wc -l # use wc -l to count lines of
# piped-in text
echo 'Hello world!' | wc -c # count characters output by echo,
# including the trailing newline
echo -n 'Hello world' ! wc -c # count characters output by echo,
# without the trailing newline |
More at:
The word grep stands for general regular expression parser.
In Unix, the grep program performs regular-expression text searching, and displays lines where the pattern text is found.
Basic usage: grep <pattern> [file] where <pattern> describes what to search for. grep can also take its input on standard input.
There are many grep regular expression metacharacters that control how the search is performed. We'll see more in Part 4: Advanced text manipulation, and at the grep command.
Because grep's metacharacters are different from metacharacters in bash, it is always a good idea to enclose the <pattern> in single quotes so that the shell treats it as literal text and passes it through as-is to grep. |
More at Intro Unix: Introducing grep
So what exactly is text? Inside of files, text isn't characters at all – it is all numbers (0's and 1's), because that's all computers know.
On standard Unix systems, each text character is stored as one byte – eight binary bits – in a format called ASCII (American Standard Code for Information Interchange). Eight bits can store 2^8 = 256 values, numbered 0 - 255. In its original form values 0 - 127 were used for standard ASCII characters. Now values 128 - 255 comprise an Extended set. See https://www.asciitable.com/
The non-printable ASCII characters we care most about are:
# display 2 lines of text using \n for newline and \t for Tab echo -e "aa z\nbb\tcc" # use the hexdump alias to view the hex values for the alphabetic # and special characters echo -e "aa z\nbb\tcc" | hexdump |
More at:
Environment variables are just like variables in a programming language (in fact bash is a complete programming language): they are names that hold a value assigned to them. As with all programming language variables, they have two operations:
In bash, you define (set/assign) an environment variable like this:
# Assign the environment variable named "varname" the value "hello world!" varname='hello world!' |
|
An environment variable can be referenced by putting the dollar sign ( $ ) metacharacter in front of the variable name (e.g $varname) or the slightly longer syntax: ${varname}.
Examples:
# The variable "$USER" is evaluated and its value substituted
foo="My USER name is $USER"; echo $foo
# Same as above using longer evaluation syntax
foo="My USER name is ${USER}; echo $foo
# Undefined environment variables just appear as empty text
bar='chess'; echo "Today's game is: $bar"
unset bar; echo "Today's game is: $bar"
# Evaluating an environment variable that contains an underscore
# may need to use the longer evaluation syntax, if the literal text
# before or after it is an underscore.
my_var="middle"
echo "File name is: foo_${my_var}_bar.txt"
|
Your built-in environment variables (e.g. $USER, $MY_GROUP, $PATH) and their values can be viewed with the env command.
More at: Intro Unix: Writing text: Environment variables
When the shell processes a command line, it first parses the text into tokens ("words"), which are groups of characters separated by whitespace (one or more space characters). Quoting affects how this parsing happens, including how metacharacters are treated and how text is grouped.
There are three types of quoting in the shell:
Note that the quote characters themselves ( ' " ` ) are metacharacters that tell the shell to "start a quoting process" then "end a quoting process" when the matching quote is found. Since they are part of the processing, the enclosing quotes are not included in the output.
Always use single ( ' ) or double ( " ) quotes when you define an environment variable whose value contains spaces so that the shell sees the quoted text as one item. |
Examples:
echo "My Unix group is $MY_GROUP" # The text "$MY_GROUP" is evaluated
# and its value substituted
echo 'My Unix group is $MY_GROUP' # The text "$MY_GROUP" is left as-is
foo="My USER name is $USER"; echo $foo # The text "$USER" is evaluated
# and its value substituted
foo='My USER name is $USER'; echo $foo # The text "$USER" is left as-is
FOO="Hello world!"
echo "The value of variable 'FOO' is \"$FOO\"" # Escape the double quotes
# inside double quotes
# Same output - the shell removes whitespace
echo Hello world!; echo Hello world!
# Echo a multi-line text variable without quotes, and the
# shell removes whitespace (here the newline)
foo='aa
bb'; echo $foo
# But enclose the multi-line variable in double quotes and whitespace
# is preserved
echo "$foo" |
If you see the greater than ( > ) character after pressing Enter, it can mean that your quotes are not paired, and the shell is waiting for more input to contain the missing quote of the pair (either single or double). Just use Ctrl-c to get back to the command prompt. |
Backtick evaluation quoting examples:
date # Calling the date command just displays
# date/time information
echo date # Here "date" is treated as a literal word, and
# written to output
echo `date` # The date command is evaluated and its output
# replaces the command
# Assign a string including today's date to variable "today"
today="Today is: `date`"; echo $today |
More at: Intro Unix: Writing text: Quoting in the shell
So far text we've been working with output to standard output, which I keep reminding you is mapped to your Terminal. But you can redirect text elsewhere.
Recall the three standard Unix streams: they each have a number, a name and redirection syntax:
If you want output to go to both the Terminal and a file, you can use the tee command (or tee -a to append).
Note that the > redirection metacharacter sends its output to a file, not to another program's standard input stream as with the | pipe metacharacter. (There are some cases where redirection involves something other than a file, but that's a topic for the Advanced Bash scripting class.)
More at: Intro Unix: Writing text: Redirection
Any time a bash command encounters an error, diagnostic error information is written to standard error, not to standard output!
When executing commands you will want to manipulate standard output and standard error appropriately – especially for 3rd party programs.
You can see that error and output streams are different by directing one or the other to the /dev/null "global trash can"
ls haiku.txt xxx.txt # displays both output and error text
# on the Terminal
ls haiku.txt xxx.txt 2>/dev/null # displays only output text on the
# Terminal
ls haiku.txt xxx.txt 1>/dev/null # displays only error text on the
# Terminal
# And this syntax (2>&1) sends both standard output and standard error
# to the same place as standard out.
ls haiku.txt xxx.txt 1>outerr.log 2>&1 |
More at:
Let's review Intro Unix: Files and File Systems. The most important takeaways are: