Page Comparison

Overview

Image Removed

After raw sequence files are generated (in FASTQ format), quality-checked, and pre-processed in some way, the next step in many NGS pipelines is mapping to a reference genome.

...

# Request a 180 minute interactive node on the normal queue using our reservation
idev -m 120 -N 1 -A OTH21164 -r CoreNGS
idev -m 120 -N 1 -A TRA23004 -r CoreNGS

# Request a 120 minute idev node on the development queue 
idev -m 120 -N 1 -A OTH21164 -p development
idev -m 120 -N 1 -A TRA23004 -p development

# Using our reservation
sbatch --reseservation=CoreNGS <batch_file>.slurm

Overview

Image Added

After raw sequence files are generated (in FASTQ format), quality-checked, and pre-processed in some way, the next step in many NGS pipelines is mapping to a reference genome.

For individual sequences it is common to use a tool like BLAST to identify genes or species of origin. However a normal NGS dataset will have tens to hundreds of millions of sequences, which BLAST and similar tools are not designed to handle. Thus a large set of computational tools have been developed to quickly align each read to its best location (if any) in a reference.

Even though many mapping tools exist, a few individual programs have a dominant "market share" of the NGS world. In this section, we will primarily focus on two of the most versatile general-purpose ones: BWA and Bowtie2 (the latter being part of the Tuxedo suite which includes the transcriptome-aware RNA-seq aligner Tophat2 as well as other downstream quantifiaction quantification tools).

Stage the alignment data

...

idev -m 180 -N 1 -A OTH21164 -r CoreNGSday4CoreNGS

Then stage the sample datasets and references we will use.

mkdir# -p $SCRATCH/core_ngs/references/fasta
Copy the FASTA files for building references
mkdir -p $SCRATCH/core_ngs/references/fasta
cp $CORENGS/references/fasta/*.fa     $SCRATCH/core_ngs/references/fasta/

# Copy the FASTQ files that will be used for alignment
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/
cd $SCRATCH/core_ngs/alignment/fastq

...

Reference Genomes

...

Here are the four reference genomes we will be using today, with some information about them. These are not necessarily the most recent versions of these references (e.g. the newest human reference genome is hg38 and the most a recent miRBase annotation is  version is v21. (See here for information about many more genomes.)

...

We've discovered a pattern (also known as a regular expression) to use in searching, and the command line tool that does regular expression matching is grep (general regular expression parser). (Read more about grep here: Advanced commands: grep.and regular expressions)

Regular expressions are so powerful that nearly every modern computer language includes a "regex" module of some sort. There are many online tutorials for regular expressions, and several slightly different "flavors" of them. But the most common is the Perl style (http://perldoc.perl.org/perlretut.html), which was one of the fist and still the most powerful (there's a reason Perl was used extensively when assembling the human genome). We're only going to use simple regular expressions here, but learning more about them will pay handsome dividends for you in the future.

...

# If you haven't stagedStage the fastaFASTA files
cds
mkdir -p core_ngs/references/fasta
cd core_ngs/references/fasta
cp $CORENGS/references/fasta/*.fa .

cd $SCRATCH/core_ngs/references/fasta
grep -P '^>' sacCer3.fa | more

...

• The -P option tells grep to Perl-style regular expression patterns. 
  
  • This makes including special characters like Tab ( \t  ), Carriage Return carriage return ( \r ) or Linefeed linefeed ( \n ) much easier that the default POSIX paterns.
  • While it is not required here, it generally doesn't hurt to include this option.

• '^>' is the regular expression describing the pattern we're looking for (described below)

• sacCer3.fa is the file to search. 
  
  • lines with text that match our pattern will be written to standard output
  • non matching lines will be omitted
• We pipe to more just in case there are a lot of contig names.

Now down to the nuts and bolts of the pattern: '^>'

First, the single quotes around the pattern – this tells the bash shell to pass the exact string contents to grep.

As part of its friendly we have seen, during command line parsing and evaluation , the shell will often look for special characters metacharacters on the command line that mean something to it (for example, the the $ in front of an environment variable name, like in $SCRATCH). Well, regular expressions treat the $ specially too – but in a completely different way! Those Those single quotes tell tell the shell "don't look inside here for special characters – treat this as a literal string and pass it to the program". The shell will obey, will strip the single quotes off the string, and will pass the actual pattern,   ^>, to the grep program. (Note that the shell does look inside double quotes ( " ) for certain special signals, such as looking for environment variable names to evaluate. Read more about  program. (Read more about Literal characters and metacharacters and Quoting in the shell.)

So what does ^> mean to grep? We know that contig name lines always start with a > character, so > is a literal for grep to use in its pattern match.

We might be able to get away with just using this literal alone as our regex, specifying '>' as as the command line pattern argument. But for for grep, the more specific the pattern, the better. So we constrain where the > can appear on the line. The special carat ( ^ )  metacharacter represents "beginning of line". So ^> means "beginning of a line followed by a > character".

Exercise: How many contigs are there in the sacCer3 reference?

# Copy the FASTA files for building references
mkdir -p $SCRATCH/core_ngs/references/fasta
cp $CORENGS/references/fasta/*.fa $SCRATCH/core_ngs/references/fasta/

Exercise: How many contigs are there in the sacCer3 reference?

cd $SCRATCH/core_ngs/references/fasta
grep -P '^>' sacCer3.fa | wc -l

grep -P -c '^>' sacCer3.fa

...

...

1. Trim the FASTQ sequences down to 50 with fastx_clipper
   • this removes most of any 5' adapter contamination without the fuss of specific adapter trimming w/cutadapt
2. Prepare the sacCer3 reference index for bwa using bwa index
   
   • this is done once, and re-used for later alignments
3. Perform a global bwa alignment on the R1 reads (bwa aln) producing a BWA-specific binary .sai intermediate file
4. Perform a global bwa alignment on the R2 reads (bwa aln) producing a BWA-specific binary .sai intermediate file
5. Perform pairing of the separately aligned reads and report the alignments in SAM format using bwa sampe
6. Convert the SAM file to a BAM file (samtools view)
7. Sort the BAM file by genomic location (samtools sort)
8. Index the BAM file (samtools index)
9. Gather simple alignment statistics (samtools flagstat and samtools idxstatidxstats)

We're going to skip the trimming step for now and see how it goes. We'll perform steps 2 - 5 now and leave , leaving samtools for a later exercise since steps 6 - 10 are common to nearly all post-alignment workflows.

...

Like other tools you've worked with so far, you first need to load bwa. Do that now, and then enter bwa with no arguments to view the top-level help page (many NGS tools will provide some help when called with no arguments). bwa is available as a BioContainers. module.

...

Make sure you're in

...

an idev session

idev -m 

180 -N 1 -A OTH21164 -r 

CoreNGS      # or -A TRA23004

idev -m 120 -N 1 -A OTH21164 -p development  # or -A TRA23004

module load biocontainers  # takes a while
module load bwa
bwa

...

mkdir -p $SCRATCH/core_ngs/alignment/fastq
mkdir -p $SCRATCH/core_ngs/references/fasta
cp $CORENGS/alignment/*fastq.gz   $SCRATCH/core_ngs/alignment/fastq/
cp $CORENGS/references/fasta/*.fa $SCRATCH/core_ngs/references/fasta/

...

mkdir -p $SCRATCH/core_ngs/references/bwa/sacCer3
cd $SCRATCH/core_ngs/references/bwa/sacCer3
ln -ssf ../../fasta/sacCer3.fa
ls -l

...

# Copy the pre-built FASTA files for building references
mkdir -p $SCRATCH/core_ngs/references
cp $CORENGS/references/fasta/*.fa $SCRATCH/core_ngs/references/fasta/

# Get the FASTQ to align Copy a pre-built bwa index for sacCer3
mkdir -p $SCRATCH/core_ngs/references/alignmentbwa/fastqsacCer3
cp $CORENGS/alignmentreferences/bwa/sacCer3/*fastq.gz* $SCRATCH/core_ngs/references/alignmentbwa/fastqsacCer3/

# Get the FASTQ to align
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/

mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
ln -ssf -f ../fastq
ln -s -fsf ../../references/bwa/sacCer3

...

bwa aln

Required arguments are a <prefix> of the bwa index files, and the input FASTQ file. There are lots of options, but here is a summary of the most important ones.

Other options control the details of how much a mismatch or gap is penalized, limits on the number of acceptable hits per read, and so on. Much more information can be found on the BWA manual page.

For a basic alignment like this, we can just go with the default alignment parameters.

Note that bwa writes its (binary) output to standard output by default, so we need to redirect that to a .sai file.

For simplicity, we will just execute these commands directly, one at a time. Each command should only take few minutes and you will see bwa's progress messages in your terminal.

...

...

Other options control the details of how much a mismatch or gap is penalized, limits on the number of acceptable hits per read, and so on. Much more information can be found on the BWA manual page.

For a basic alignment like this, we can just go with the default alignment parameters.

Note that bwa writes its (binary) output to standard output by default, so we need to redirect that to a .sai file.

For simplicity, we will just execute these commands directly, one at a time. Each command should only take few minutes and you will see bwa's progress messages in your terminal.

# If not already loaded:
module load biocontainers
module load bwa

cd $SCRATCH/core_ngs/alignment/yeast_bwa
bwa aln sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R1.cat.fastq.gz > yeast_pe_R1.sai
bwa aln sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R2.cat.fastq.gz > yeast_pe_R2.sai

When all is done you should have two .sai files: yeast_pe_R1.sai and yeast_pe_R2.sai.

Exercise: How long did it take to align the R2 file?

[bwa_aln] 17bp reads: max_diff = 2
[bwa_aln] 38bp reads: max_diff = 3
[bwa_aln] 64bp reads: max_diff = 4
[bwa_aln] 93bp reads: max_diff = 5
[bwa_aln] 124bp reads: max_diff = 6
[bwa_aln] 157bp reads: max_diff = 7
[bwa_aln] 190bp reads: max_diff = 8
[bwa_aln] 225bp reads: max_diff = 9
[bwa_aln_core] calculate SA coordinate... 50.76 sec
[bwa_aln_core] write to the disk... 0.07 sec
[bwa_aln_core] 262144 sequences have been processed.
[bwa_aln_core] calculate SA coordinate... 50.35 sec
[bwa_aln_core] write to the disk... 0.07 sec
[bwa_aln_core] 524288 sequences have been processed.
[bwa_aln_core] calculate SA coordinate... 13.64 sec
[bwa_aln_core] write to the disk... 0.01 sec
[bwa_aln_core] 592180 sequences have been processed.
[main] Version: 0.7.17-r1188
[main] CMD: /usr/local/bin/bwa aln sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R1.cat.fastq

.gz
[main] Real time: 85.584 sec; CPU: 83.825 sec

Since you have your own private compute node, you can use all its resources. It has 128 cores, so re-run the R2 alignment asking for 60 execution threads.

bwa aln -t 60 sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R2.cat.fastq.gz > yeast_R2.sai

When all is done you should have two .sai files: yeast_R1.sai and yeast_R2.sai.

pe_R2.sai

Exercise: How long did it take to align much of a speedup did you seen when aligning the R2 file with 60 threads?

[bwa_aln] 17bp reads: max_diff = 2
[bwa_aln] 38bp reads: max_diff = 3
[bwa_aln] 64bp reads: max_diff = 4
[bwa_aln] 93bp reads: max_diff = 5
[bwa_aln] 124bp reads: max_diff = 6
[bwa_aln] 157bp reads: max_diff = 7
[bwa_aln] 190bp reads: max_diff = 8
[bwa_aln] 225bp reads: max_diff = 9
[bwa_aln_core] calculate SA coordinate... 50266.7670 sec
[bwa_aln_core] write to the disk... 0.0704 sec
[bwa_aln_core] 262144 sequences have been processed.
[bwa_aln_core] calculate SA coordinate... 50268.3594 sec
[bwa_aln_core] write to the disk... 0.0703 sec
[bwa_aln_core] 524288 sequences have been processed.
[bwa_aln_core] calculate SA coordinate... 1372.6426 sec
[bwa_aln_core] write to the disk... 0.01 sec
[bwa_aln_core] 592180 sequences have been processed.
[main] Version: 0.7.17-r1188
[main] CMD: /usr/local/bin/bwa aln sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R1.cat.fastq.gz
[main] Real time: 78.185 sec; CPU: 77.598 sec

Since you have your own private compute node, you can use all its resources. It has 128 cores, so re-run the R2 alignment asking for 60 execution threads.

.17-r1188
[main] CMD: /usr/local/bin/bwa aln -t 60 sacCer3/sacCer3.fa fastq/Sample_Yeast

Exercise: How much of a speedup did you seen when aligning the R2 file with 20 threads?

[bwa_aln] 17bp reads: max_diff = 2
[bwa_aln] 38bp reads: max_diff = 3
[bwa_aln] 64bp reads: max_diff = 4
[bwa_aln] 93bp reads: max_diff = 5
[bwa_aln] 124bp reads: max_diff = 6
[bwa_aln] 157bp reads: max_diff = 7
[bwa_aln] 190bp reads: max_diff = 8
[bwa_aln] 225bp reads: max_diff = 9
[bwa_aln_core] calculate SA coordinate... 266.70 sec
[bwa_aln_core] write to the disk... 0.04 sec
[bwa_aln_core] 262144 sequences have been processed.
[bwa_aln_core] calculate SA coordinate... 268.94 sec
[bwa_aln_core] write to the disk... 0.03 sec
[bwa_aln_core] 524288 sequences have been processed.
[bwa_aln_core] calculate SA coordinate... 72.26 sec
[bwa_aln_core] write to the disk... 0.01 sec
[bwa_aln_core] 592180 sequences have been processed.
[main] Version: 0.7.17-r1188
[main] CMD: /usr/local/bin/bwa aln -t 60 sacCer3/sacCer3.fa fastq/Sample_Yeast_L005_R2.cat.fastq.gz
[main] Real time: 5.013 sec; CPU: 142.813 sec

Next we use the bwa sampe command to pair the reads and output SAM format data. Just type that command in with no arguments to see its usage.

For this command you provide the same reference index prefix as for bwa aln, along with the two .sai files and the two original FASTQ files. Also, bwa writes its output to standard output, so redirect that to a .sam file.

Here is the command line statement you need. Just execute it on the command line.

bwa sampe sacCer3/sacCer3.fa yeast_R1.sai yeast_R2.sai \
  fastq/Sample_Yeast_L005_R1.cat.fastq.gz \
  fastq/Sample_Yeast_L005_R2.cat.fastq.gz > yeast_pairedend.sam

...

_L005_R2.cat.fastq.gz
[main] Real time: 7.931 sec; CPU: 179.153 sec

Next we use the bwa sampe command to pair the reads and output SAM format data. Just type that command in with no arguments to see its usage.

For this command you provide the same reference index prefix as for bwa aln, along with the two .sai files and the two original FASTQ files. Also, bwa writes its output to standard output, so redirect that to a .sam file.

Here is the command line statement you need. Just execute it on the command line.

# Copy the FASTA files for building references
mkdir -p $SCRATCH/core_ngs/references
cp $CORENGS/references/fasta/*.fa $SCRATCH/core_ngs/references/fasta/

# Copy a pre-built bwa index for sacCer3
mkdir -p $SCRATCH/core_ngs/references/bwa/sacCer3
cp $CORENGS/references/bwa/sacCer3/*.* $SCRATCH/core_ngs/references/bwa/sacCer3/

# Get the FASTQ to align
mkdir -p $SCRATCH/core_ngs/alignment/fastq
cp $CORENGS/alignment/*fastq.gz $SCRATCH/core_ngs/alignment/fastq/

# Stage the BWA .sai files
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
ln -sf ../fastq
ln -sf ../../references/bwa/sacCer3
cp $CORENGS/catchup/yeast_bwa/*.sai .

cd $SCRATCH/core_ngs/alignment/yeast_bwa
bwa sampe sacCer3/sacCer3.fa yeast_pe_R1.sai yeast_pe_R2.sai \
  fastq/Sample_Yeast_L005_R1.cat.fastq.gz \
  fastq/Sample_Yeast_L005_R2.cat.fastq.gz > yeast_pe.sam

You should now have a SAM file (yeast_pe.sam) that contains the alignments.

Exercise: How many lines does the SAM file have? How does this compare to the number of input sequences (R1+R2)?

It's just a text file, so take a look with head, more, less, tail, or whatever you feel like. Later you'll learn additional ways to analyze the data with samtools once you create a BAM file.

...

Exercise: How many alignment records (not header records) are in the SAM file?

grep -P -c '^HWI' yeast_pairedendpe.sam

grep -P -v -c '^@' yeast_pairedend.sam

Exercise: How many sequences were in the R1 and R2 FASTQ files combined?

grep -P -v -c '^@' yeast_pe.sam

Exercises:

...

• Does the SAM file contain both mapped and un-mapped reads?
• What is the order of the alignment records in this SAM file?
  
  

...

Suppose you wanted to look only at field 3 (contig name) values in the SAM file. You can do this with the handy cut command. Below is a simple example where you're asking cut to display the 3rd column value for the last 10 alignment records.

# Stage the aligned SAM file
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pe.sam .

tail yeast_pairedendpe.sam | cut -f 3

By default cut assumes the field delimiter is Tab, which is the delimiter used in the majority of NGS file formats. You can specify a different delimiter with the -d option.

...

tail -20 yeast_pairedendpe.sam | cut -f 2-6,9

You may have noticed that some alignment records contain contig names (e.g. chrV) in field 3 while others contain an asterisk ( * ). The * means the record didn't map. We're going to use this heuristic along with cut to see about how many records represent aligned sequences. (Note this is not the strictly correct method of finding unmapped reads because not all unmapped reads have an asterisk in field 3. Later you'll see how to properly distinguish between mapped and unmapped reads using samtools.)

...

# the ^@ pattern matches lines starting with @ (only header lines), 
# and -v says output lines that don't match
grep -v -P '^@' yeast_pairedendpe.sam | head

Ok, it looks like we're seeing only alignment records. Now let's pull out only field 3 using cut:

...

grep -v -P '^@' yeast_pairedendpe.sam | cut -f 3 | grep -v '*' | head

...

grep -v -P '^@' yeast_pairedendpe.sam | cut -f 3 | grep -v '*' | wc -l

Read more at Some Linux commands: Advanced commands

Exercise: About how many records represent aligned sequences? What alignment rate does this represent?

awk 'BEGIN{print 612968/1184360}'

Exercise: What might we try in order to improve the alignment rate?

...

idev -m 120 -N 1 -A OTH21164 -r CoreNGSday4 CoreNGS     # or -A TRA23004

idev -m 90 -N 1 -A OTH21164 -p development  # or -A TRA23004

# If not already loaded
module load biocontainers  # takes a while

module load samtools
samtools

...

In this exercise, we will explore five utilities provided by samtools: view, sort, index, flagstat, and idxstats. Each of these is executed in one line for a given SAM/BAM file. In the SAMtools/BEDtools sections tomorrow we will explore samtools in capabilities more in depth.

...

mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pairedendpe.sam .

cd $SCRATCH/core_ngs/alignment/yeast_bwa
cat
yeast_pairedend.sam | samtools view -b -o yeast_pe.sam > yeast_pairedendpe.bam 

• the -b option tells the tool to output BAM format
• the -o option specifies the name of the output BAM file that will be created
• we pipe the entire SAM file to samtools view so that the header records are included (required for SAM → BAM conversion)
  • samtools view reads its input from standard input by default

...

• the tool to output BAM format

The BAM file is a binary file, not a text file, so how do you look at its contents now? Just use samtools view without the -b option. Remember to pipe output to a pager!

samtools view yeast_pairedendpe.bam | more

Notice that this does not show us the header record we saw at the start of the SAM file.Exercise: What samtools view option will include the header records in its output? Which option would show only the header records?we saw at the start of the SAM file.

Exercise: What samtools view option will include the header records in its output? Which option would show only the header records?

...

Looking at some of the alignment record information (e.g. samtools view yeast_pairedendpe.bam | cut -f 1-4 | more), you will notice that read names appear in adjacent pairs (for the R1 and R2), in the same order they appeared in the original FASTQ file. Since that means the corresponding mappings are in no particular order, searching through the file very inefficient. samtools sort re-orders entries in the SAM file either by locus (contig name + coordinate position) or by read name.

...

# Stage the aligned yeast SAM and BAM files
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pairedend.bampe.[bs]am .

To sort the paired-end yeast BAM file by position, and get a BAM file named yeast_pairedendpe.sort.bam as output, execute the following command:

cd $SCRATCH/core_ngs/alignment/yeast_bwa
samtools sort -O bam -T yeast_pairedendpe.tmp yeast_pairedendpe.bam > yeast_pairedendpe.sort.bam

• The -O options says the Output format should be BAM
• The -T options gives a prefix for Temporary files produced during sorting
  • sorting large BAMs will produce many temporary files during processingduring processing
  • make sure the temporary file prefix is different from the input BAM file prefix!
• By default sort writes its output to standard output, so we use > to redirect to a file named yeast_pairedend.sort.bam

Exercise: Compare the file sizes of the yeast_pariedend pe .sam, .bam, and .sort.bam files and explain why they are different.

ls -lh yeast_pairedendpe.*

...

samtools index yeast_pairedendpe.sort.bam

This will produce a file named yeast_pairedendpe.bam.bai.

Most of the time when an index is required, it will be automatically located as long as it is in the same directory as its BAM file and shares the same name up until the .bai extension.

...

ls -lh yeast_pairedendpe.sort.bam*

samtools flagstat

...

Here's how to run samtools flagstat and both see the output in the terminal and save it in a file – the samtools flagstat standard output is piped to tee, which both writes it to the specified file and sends it to its standard output:it to the specified file and sends it to its standard output:

# Stage the aligned yeast SAM and BAM files
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pe.sort.bam* .

samtools flagstat yeast_pairedendpe.sort.bam | tee yeast_pariedendpe.flagstat.txt

You should see something like this:

...

More information about the alignment is provided by the samtools idxstats report, which shows how many reads aligned to each contig in your reference. Note that samtools idxstats must be run on a sorted, indexed BAM file.

# Stage the aligned yeast SAM and BAM files
mkdir -p $SCRATCH/core_ngs/alignment/yeast_bwa
cd $SCRATCH/core_ngs/alignment/yeast_bwa
cp $CORENGS/catchup/yeast_bwa/yeast_pe.sort.bam* .

samtools idxstats yeast_pairedendpe.sort.bam | tee yeast_pairedendpe.idxstats.txt

Here we use the tee command which reports its standard input outputto standard output before also writing it to the specified file.

...