Annotating Variants: Introduction
As we've already seen, determining the presence or absence of a variant from NGS data is not trivial. It is software dependent and has inherent trade-offs between sensitivity and specificity. Inevitably, the number of putative variants in a real data set is very large; for example the first samples from the 1000 Genomes project typically found 0.1% variants (3 million variants), approximately 10% of which had never been previously observed (300,000 novel variants per individual.) False positive discovery rates are also typically very high at this stage.
...
Because these tools draw in information from may disparate sources, they can be very difficult to install, configure, use, and maintain. For example, the vcf
files from the 1000 Genomes project are arranged in a deep ftp tree by date of data generation. Large genome centers spend significant resources managing these tools.
Pre-packaged programs
Annovar - one of the most powerful yet simple to run variant annotators available
Annovar is a variant annotator. Given a vcf file from an unknown sample and a host of existing data about genes, other known SNPs, gene variants, etc., Annovar will place the discovered variants in context.
...
This next exercise will give you some idea of how Annovar works; we've taken the liberty of writing the bash script annovar_pipe.sh
around the existing summarize_annovar.pl
wrapper (a wrapper within a wrapper - a common trick) to even further simplify the process for this course.
Exercise:
First, look at the code for our annovar_pipe.sh
command.
Expand | |||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| |||||||||||||||||||
The which command is used to give the location of a program or script that is in your $PATH.
This script simply does a format conversion and then calls
change name of code block
This script is written in perl (a programing language that while powerful, is not very favored among most computational biologists who typically prefer: Python, R, or Bash shell scripts) as a method of standardizing input to call a series of external commands. Such wrappers (or wrappers within wrappers) make your life much easier. breseq itself has wrappers for using bowtie2 and samtools which you ran separately on other data. As you increase your understanding of scripts and the command line by looking at what others have done you may begin to make your own wrappers and small scripts, but such wrappers are a great example of the BioITeam and other community resources can provide you with. While we have told you no one will ever care as much about your data as you do, quality of life issues regarding repetitive treatment of similar inputs and outputs may be easily solved by someone else. |
Now let's run it on the .vcf files from the 3 individuals (NA12878, NA12891, and NA12892) from both the samtools and gatk output in the $BI/ngs_course/human_variation/ directory. (You may recognize these as the same individuals that we worked with on the Trios tutorial. Throughout the class we've been teaching you how to create a commands file using nano, but here we provide a more complex example of how you can generate a commands file. As you become more proficient with the command line, it is likely you will use various piping techniques to generate commands file. The following calls Perl to custom-create the 6 command lines needed and put them straight into a commands file
:
Code Block | ||
---|---|---|
| ||
ls $BI/ngs_course/human_variation/N*.vcf | \ perl -n -e 'chomp; $_=~/(NA\d+).*(sam|GATK)/; print "annovar_pipe.sh $_ >$1.$2.log 2>&1\n";' > commands |
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
cds
mkdir BDIB_Annovar
cd BDIB_Annovar
# Use only 1 of the following copy commands
# if you have already done the trios tutorial
cp $SCRATCH/BDIB_Human_tutorial/raw_files/N*.vcf .
# if you have not done the trios tutorial
cp $BI/ngs_course/human_variation/N*.vcf .
ls *.vcf | perl -n -e 'chomp; $_=~/(NA\d+).*(sam|GATK)/; print "annovar_pipe.sh $_ >$1.$2.log 2>&1 &\n";' > commands
|
Note | ||
---|---|---|
| ||
"commands" files are 1 way to both create a record of what it is that you are doing as well as an easy way to execute multiple commands simultaneously. In a previous tutorial (the advanced breseq tutorial, or the 2nd breseq example from the first day) we showed you how to do this by adding an & as the last character of the line to force the command to execute in the background. Tomorrow we will go over the more common method of submitting "commands" files to the que to run as jobs rather than being spoiled with interacting with everything on idev nodes. In computational biology this instructor has often found that once I learn to do something a particular way, it takes an incredible amount of inertia to change how I do something even when i know it would help to do so. In an effort to learn from me |
...
make me an expand
investigate the commands file to to determine what the piping command actually did (click for answer)
make me a code block
cat commands
...
making things harder on myself than need be, I urge you to STRONGLY consider not naming every list of commands you want to run "commands" but rather something that is actually descriptive, even "commands_DATE" would be helpful. As you start submitting your own jobs you will quickly be able to build bad habits, try to be aware of and avoid this one if possible. |
Expand | |||||||||
---|---|---|---|---|---|---|---|---|---|
| |||||||||
|
Warning | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||
|
The easiest way to check if this is working is to use ls and see an explosion of new files. This will take quite a bit of time to complete running. As such, we have ALREADY pre-computed these outputs so you can begin evaluating the results.
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
cds mkdir BDIB_Annovar cd BDIB_Annovar cp $SCRATCH/BDIB_Human_tutorial/raw_files/N*.vcf . ls *.vcf | perl -n -e 'chomp; $_=~/(NA\d+).*(sam|GATK)/; print "annovar_pipe.sh $_ >$1.$2.log 2>&1 &\n";' > commands |
change title of codeblock = MAKE SURE YOU ARE ON IDEV if you want to run
Code Block | ||
---|---|---|
| ||
chmod +x commands
./commands |
...
mkdir provided_results
cd provided_results
cp $BI/ngs_course/human_variation/*chrom20.samtools* . |
Later, when your results have completed running (likely >90 minutes as annovar not configured to use multiple processes, compare your results to these provided results).
For the scavengar hunts below you should also copy over the GATK results.
Code Block | ||||
---|---|---|---|---|
| ||||
cp $BI/ngs_course/human_variation/*chrom20.GATK* . |
ANNOVAR output
Annovar does a ton of work in assessing variants for us (though if you were going for clinical interpretation, you still have a long way to go - compare this to RUNES or CarpeNovo). It provides all these output files:
...
Everything after the "LJB_GERP++" field in exome_summary came from the original VCF file, so this file REALLY contains everything you need to go on to functional analysis! This is one of the many reasons I like Annovar.
Scavenger hunts!
Expand | |||||
---|---|---|---|---|---|
| |||||
The final answer is "DEFB126"
|
...
Expand | |||||||||
---|---|---|---|---|---|---|---|---|---|
| |||||||||
The final form of the command that you had was:grep "frameshift" NA12878.chrom20.GATK.vcf.exome_summary.csv | awk -F"," '{print $2"\t"$3}' | uniq -c | sort -r
Some questions for thought:
|
Other variant annotators:
- http://www.yandell-lab.org/software/vaast.html
- http://www.broadinstitute.org/gatk/gatkdocs/ VariantAnnotator annotations
- http://www.bioconductor.org/help/workflows/variants/
- http://vat.gersteinlab.org/
- http://code.google.com/p/mu2a/
Notes
Variants consist of single base base changes, insertions and deletions, and larger scale structural changes. "Larger scale" is usually defined relative to the capabilities of the technology; for example, a "small indel" usually means "detectable within a single sequence read". In 2009, sequence reads were about 50 bp but in 2011 they were 100 bp.