Content Comparison

...

We assume that you are still working in the main directory called mapping data that you created on $SCRATCH.

Load SAMtools

Check if SAMtools is loaded and if not Load the SAMtools module (if not already loaded).

module list samtools
module avail samtools
module load samtools

Can you figure out what version of samtools is loaded on TACC and where it is installed?

samtools
which samtools

Prepare your directories

From inside your main mapping directory, create a new output directory called samtools_bowtie2 or whatever makes sense to you.

LetSince the $SCRATCH directory on lonestar is effectively infinite for our purposes, we're going to copy the relevant files from our mapping tutorial into a new directory for this tutorial. This should help you identify what files came from what tutorial if you look back at it in the future. Let's copy over just the read alignment file in the SAM format and the reference genome in FASTA format to this new directory, so that we don't have so many files cluttering our space.

...

a new directory called samtools_tutorial.

cds
mkdir samtools_tutorial
cd samtools_tutorial
cp $SCRATCH/bowtie2MappingTutorial/bowtie2/SRR030257.sam 

.
cp $SCRATCH/bowtie2MappingTutorial/bowtie2/NC_012967.1.fasta .

Index the FASTA reference file

First, you need to index the reference file. (This isn't indexing it for read mapping. It's indexing it so that SAMtools can quickly jump to a certain base in the reference.)Then run this command

...

samtools faidx samtools_bowtie2/NC_012967.1.fasta

Take a look at the new *.fai file that was created by this command . Any see if you have any idea what some of the numbers mean?

...

. 

less NC_012967.1.fasta.fai

  # can exit with "q"

As you can see, the less command also works perfectly well with files that are not in danger of crashing anything without cluttering your terminal  with lines of a file.

Convert mapped reads from SAM to BAM, sort, and index

SAM is a text file, so it is slow to access information about how any given read was mapped. SAMtools and many of the commands that we will run later work on BAM files (essentially GZIP compressed binary forms of the text SAM files). These can be loaded much more quickly. Typically, they also need to be sorted, so that when the program wants to look at all reads overlapping position 4,129,888, it can easily find them all at once without having to search through the entire BAM file.

The following 3 commands are used to convert from SAM to BAM format, sort the BAM file, and index the BAM file. As each command requires the previous command to have been completed it makes more sense to run them on an iDEV node. If you want to submit them to the queue, separate them with a ";" to ensure that they run sequentially on the same node. Under no circumstances should you run this on the head node.

idev -m 60 -q development -A CCBB

Convert from SAM to BAM format.

UT-2015-05-18

samtools view -b -S -o 

SRR030257.bam 

SRR030257.sam

Sort and index the BAM file.

samtools sort 

SRR030257.bam 

SRR030257.sorted
samtools index 

SRR030257.sorted.bam

Examine the output of the previous commands to get an idea of whats going on. Here are some prompts of how to do that:

• expand

  Expand
  title What new files were created by these commands?
  Code Block	Check that	Check that	code language bash title List the contents of the output directory ls samtools_bowtie2 Code Block title Expected output NC_012967.1.fasta SRR030257.sorted.bam.bai NC_012967.1.fasta.fai SRR030257.sam SRR030257.bam SRR030257.sorted.bam

• Expand
  title Why didn't we name the

  output 

  output SRR030257.sorted.bam

   in the 

  in the samtools sort

   command

  command?
  Expand
  Answer...	Answer...	Samtools appends an extra .bam to whatever we put here, so it would have created SRR030257.sorted.bam.bam, and then we would have had to make a joke about the Flintstones.

• Expand
  title Can you guess what a *.bai file is?

  Expand
  Answer... Answer...
  Sure enough, it's the index file for the BAM file.

...

Call genome variants

Now we use the mpileup command from samtools to compile information about the bases mapped to each reference position. Output BCF The output is a BCF file. This is a binary form of the text Variant Call Format (VCF).

samtools mpileup -u -f samtools_bowtie2/NC_012967.1.fasta samtools_bowtie2/SRR030257.sorted.bam > samtools_bowtie2/SRR030257.bcf

What are all the options doing? Try calling samtools mpileup without any options to see if you can figure it out before clicking below to  

The samtools mpileup command will take a few minutes to run. You might consider As practice for a fairly common occurrence when working with the iDEV environment, once the command is running, you should try putting it in the background by pressing control-z and then typing the command bg so that you can do some other things in this terminal window at the same time – like starting to run variant calling on the BWA or bowtie mapping results. Remember, there are still many other processors available on this node!

Convert BCF to Once the mpileup command is complete, convert the BCF file to a "human-readable VCF:" VCF file using the bcftools command (the which command will tell you where this command is located and examination of that path should tell you how you have access to it).

bcftools view -v -c -g samtools_bowtie2/SRR030257.bcf > samtools_bowtie2/SRR030257.vcf

What are these options doing?

SRR030257.bcf

> SRR030257.vcf

Take a look at the samtools_bowtie2/SRR030257.vcf file using less. It has a nice header explaining what the columns mean. Below this are the rows of data describing potential genetic variants.

...

Follow the same directions to call variants in the BWA or Bowtie mapped reads.Just  Just be sure you don't write over your old files. Maybe create new directories like samtools_bwa and samtools_bowtie for the output in each case.You  You could also try running all of the commands from inside of the samtools_bwa directory, just for a change of pace.

...

VCF format has alternative Allele Frequency tags denoted by AF1. AF= Try the following command to see what values we have in our files.

grep AF1 samtools_bowtie2/SRR030257.vcf

...

...

What does the -v flag do in grep?

grep -v *something*  # The -v flag inverts the match effecitvely showing you everything that does not match your input

cat input.vcf | grep AF1=1 > output.vcf

cat input.vcf | grep -v AF1=0 > output.vcf

sed -i '/AF1=0/ d' input.vcf

Comparing the results of different mappers using bedtools

Often you want to compare the results of variant calling on different samples or using different pipelines. Bedtools is a suite of utility programs that work on a variety of file formats, one of which is conveniently VCF format. It provides many ways of slicing, dicing, and comparing the information in VCF files. For example, we can use it to find out what predictions are the same and which are different from the variant calling on reads mapped with different programs if you generated VCF files for each one.Set  Set up a new output directory and copy the respective VCF files to it, renaming them so that we know where they came from:

mkdir comparison
cp -i samtools_bowtie2/SRR030257.vcf comparison/bowtie2.vcf
cp -i samtools_bwa/SRR030257.vcf comparison/bwa.vcf
cp -i samtools_bowtie/SRR030257.vcf comparison/bowtie.vcf
cd comparison

Use the subcommands bedtools intersect and bedtools subtract we can find equal and different predictions between mappers. Try to figure out how to to do this on your own first. Hint: Remember that adding > output.vcf to the end of a command will pipe the output that is to the terminal into a file, so that you can save it.

Load Bedtools.

module load bedtools

bedtools intersect -a bowtie2.vcf -b bwa.vcf > common_bowtie2_bwa.vcf

bedtools subtract -a bowtie2.vcf -b common_bowtie2_bwa.vcf > unique_bowtie2.vcf
bedtools subtract -a bwa.vcf -b common_bowtie2_bwa.vcf > unique_bwa.vcf

Further Optional Exercises

...

• Look at how the reads supporting these variants were aligned to the reference genome in the Integrative Genomics Viewer (IGV) tutorial.
• Look into more sophisticated variant calling with GATK. We recommend starting from the GATK best practice page.