Part 4: Advanced text manipulation

1 Example data files
- 1.1 Exercise 3-1
2 Cut, sort, uniq
- 2.1 cut
  - 2.1.1 Exercise 3-2
- 2.2 sort
  - 2.2.1 Exercise 3-3
- 2.3 uniq
- 2.4 “Piping a histogram” with cut/sort/uniq -c
  - 2.4.1 Exercise 3-4
- 2.5 Ensuring uniqueness of field combinations
  - 2.5.1 Exercise 3-5
3 Introducing awk
- - 3.1.1 awk example 1
  - 3.1.2 awk example 2
  - 3.1.3 awk example 3
  - 3.1.4 Exercise 3-6
- 3.2 A more complicated awk script
- 3.3 Parsing field-oriented text with cut and awk
4 Regular expressions in grep, sed and perl
- 4.1 regular expressions
- 4.2 grep pattern matching
  - 4.2.1 Exercise 3-7
- 4.3 perl pattern matching
  - 4.3.1 Exercise 3-8
- 4.4 sed pattern substitution
- 4.5 perl pattern substitution
  - 4.5.1 Exercise 3-9
5 Bash control flow
- 5.1 The bash for loop
  - 5.1.1 Quotes matter
  - 5.1.2 Exercise 3-10
- 5.2 The if statement
- 5.3 Reading file lines with while
  - 5.3.1 Exercise 3-11
6 A few odds and ends
- 6.1 Arithmetic in bash
- 6.2 Removing file suffixes and prefixes
  - 6.2.1 Exercise 3-12
- 6.3 Input from a sub-shell
- 6.4 Using a heredoc for multiple text lines

Example data files

For some of the discussions below, we'll use some files in your ~/data directory.

The ~/data/walrus_sounds.tsv file () lists the types of sounds made by several well-known walruses, and the length of each occurrence. tab-delimited fields are:

column 1 - walrus name
column 2 - sound type
column 3 - length of sound

Take a look at the first few lines of this file:

cd ~/data
head walrus_sounds.tsv

The .tsv filename extension stands for tab separated values, indicating that the field separator (the character separating fields) is tab. We can verify this using the handy hexdump alias we defined for you as discussed at Intro Unix: What is text?

cd ~/data
head walrus_sounds.tsv | hexdump

The output looks like this, where the hexadecimal 0x09 character is a Tab.

We will also use two data files from the GSAF's (Genome Sequencing and Analysis Facility) automated processing that delivers sequencing data to customers. These files have information about customer Samples (libraries of DNA molecules to sequence on the machine), grouped into sets assigned as Jobs, and sequenced on GSAF's sequencing machines as part of sequencer Runs.

The files are also in your ~/data directory:

- contains job name/sample name pairs, tab-delimited, no header
- the "JAnnnnn" items in the 1st column are Jobs
- the "SAnnnnn" items in the 2nd column are Runs
  - several Jobs can be associated with the same Run
- contains information about all samples run on a particular run, along with the job each belongs to.
- columns (tab-delimited) are job_name, job_id, sample_name, sample_id, date_string
- column names are in an initial header line

Take a look at the first few lines of these files also:

cd ~/data
head joblist.txt
head sampleinfo.txt

Exercise 3-1

What field separators are used in ~/data/joblist.txt and ~/data/sampleinfo.txt?

cd ~/data head -1 joblist.txt | hexdump head -2 sampleinfo.txt | hexdump The hexdump output shows that both files use tab to separate fields.

How many lines to these sample files have?

Use the word count command with the -l (count lines) option: wc -l

cd ~/data wc -l *.txt *.tsv # or wc -l *.{txt,tsv} shows: 3841 joblist.txt 44 sampleinfo.txt 200 walrus_sounds.tsv 4085 total

Cut, sort, uniq

cut

The cut command lets you isolate ranges of data from its input lines (from files or standard input):

cut -f <field_number(s)> extracts one or more fields (-f) from each line
- the default field delimiter is tab
  - use -d <delim> to change the field delimiter
cut -c <character_number(s)> extracts one or more characters (-c) from each line
the <numbers> can be
- a comma-separated list of numbers (e.g. 1,4,7)
- a hyphen-separated range (e.g. 2-5)
- a trailing hyphen says "and all items after that" (e.g. 3,7-)
cut does not re-order fields, so cut -f 5,3,1 acts like -f 1,3,5

Examples:

cd ~/data
cut -f 2 joblist.txt | head
head joblist.txt | cut -c 9-13

# no field reordering, so these two produce the same output
cut -f1,2 walrus_sounds.tsv | head
cut -f2,1 walrus_sounds.tsv | head

Exercise 3-2

How would you extract the first 5 job_name and sample_name fields from ~/data/sampleinfo.txt without including the header? Recall that job_name and sample_name are fields 1 and 3 of ~/data/sampleinfo.txt.

Use tail -n +2 (or tail +2) to skip the header line and start at line 2. Then use cut -f to isolate the desired fields.

tail -n +2 ~/data/sampleinfo.txt | head -5 | cut -f 1,3

sort

sort sorts its input lines using an efficient algorithm

by default sorts each line lexically (as strings), low to high
- use -n sort numerically (-n)
- use -V for Version sort (numbers with surrounding text)
- use -r to reverse the sort order
use one or more -k <start_field_number>,<end_field_number> options to specify a range of "keys" (fields) to sort on
- use this option when you want to preserve all data on the input lines, but sort on part(s) of the line
- e.g. -k1,1 -2,2nr to sort field 1 lexically then field 2 as a number high-to-low
by default, fields are delimited by whitespace – one or more spaces or tabs
- use -t <delim> to change the field delimiter (e.g. -t "\t" for tab)

Examples: