Page Comparison

...

When dealing with large data files, sometimes scattered in many directories, it is often convenient to create multiple symbolic links (symlinks) to those files in a directory where you plan to work with them. A common way to make symbolic link uses ln -s, e.g.:

...

Multiple files can be linked by providing multiple file name arguments along and using the -t (targettarget) option to specify the directory where links to all the files can be created.

...

What about the case where the files you want are scattered in sub-directories? Consider a typical GSAF project directory structure, where Fastq FASTQ files are nested in subdirectories:

...

for <arg_name> in <list of whitespace-separated words>;
do
   <expression>
   <expression>
done

See https://www.gnu.org/software/bash/manual/html_node/Looping-Constructs.html.

Here's a simple example using the seq function, that returns a list of numbers.

for num in seq`seq 55`; do
  echo $num
done

Quotes matter

...

Here's a simple awk script that takes the mean average of the numbers passed to it:

...

Once a line has been read, it can be parsed, for example, using cut, as shown below. Other notes:c

• The double quotes around the text that "$line" are important to preserve special characters inside the original line (here tab characters).
  • Without the double quotes, the line fields would be separated by spaces, and the cut field delimiter would need to be changed.
• Some lines have an empty job name field; we replace job and sample names in this case.
• We assign file descriptor 4 to the file data being read (4< sampleinfo.txt after the done keyword), and read from it explicitly (read line <&4 in the while line).
  
  • This avoids conflict with any global redirection of standard output (e.g. from automatic logging).
    
    

...

join -t $'\t' -j 2 file1 file12

Viewing special characters in text

...

• Default field separators
  • Tab is the default field separator for cut
    • and the field separator can only be a single character
  • whitespace (one or more spaces or Tabs) is the default field separator for awk
    • note that some older versions of awk do not include Tab as a default delimiter
• Re-ordering
  • cut cannot re-order fields; cut -f 3,2 is the same as cut -f 2,3.
  • awk does reorder fields, based on the order you specify
• awk is a full-featured programming language while cut is just a single-purpose utility.

...

• -n tells perl to feed the input to the script one line at a time
• -e introduces the perl script
  • always encode Always enclose a command-line perl script in single quotes to protect it from shell evaluation
• $_ is a built-in variable holding the current line (including any invisible line-ending characters)
• ~ is the perl pattern matching operator (=~ says pattern must match; ! ~ says pattern not matching)
• the forward slashes ("/  /") enclose the regex pattern.

...

The sed command can be used to edit text using pattern substitution. While it is very powerful, the regex syntax for some of its more advanced features is quite different from "standard" grep or perl regular expressions. As a result, I tend to use it only for very simple substitutions, usually as a component of a multi-pipe expression.

...

perl pattern substitution

If I have a more complicated pattern, or if sed pattern substitution is not working as I expect (which happens frequently!), I again turn to perl. Here's how to invoke regex pattern substitution from a command line:

...