Overview

Throughout the course we have focused on samll data sets, a limited number of samples, and in some cases even purposefully capped the total number of reads you have access to. This has been done for the purpose of time and letting you see the results tick by rather than simply having you come in for 30 minutes, submit a job, and wait an hour (or 6) before it starts running, and have it take another 10 hours to run. The reality is while you will sometimes work with a test sample or a small pilot project, Big Data in Biology means LOTS of data and lots of data means needing to not just identify variants in 1 sample, but to identify commonality across different systems. here we introduce you to bedtools. A program designed to make comparisons across differnt file types generaterated from different samples or using different parameters of a given pipeline.

Learning objectives

Review another situation of the importance of knowing what version you are working with, and take note of how older versions may be better or at least have their place when you dont have access to infinite computational resources
Become familiar with how to use bedtools intersect and subtract
Understand when and how bedtools is useful

A note on version control

As mentioned several times throughout the course, different versions of software behave differently. Once again we have a situation where the BioITeam bedtools is available by default (version 2.20.1), and a different version is available on lonestar (version 2.26.0). A KEY addition to version 2.21 (aka the version after that which is available to you by default through the BioITeam as of this writing) was the ability of bedtools to simultaneously scan multiple files at once rather than having to sequentially scan pairs of files. Using the old version, to identify the common variants of files A, B, C and D, bedtools would have to be invoked a minimum of 3 times:

A & B = E
C & D = F
E & F = G

More broadly, it would have to be invoked a minimum of (number of samples - 1) times. So as you start adding more and more samples, the commands get more and more difficult to write. In some of the other tutorials you will see how we have used command line for loops to systematically deal with these types of situations though this situation would be much more complex likely requiring nested for loops and conditionals. With the new version however, such complexity isn't required though the computational requirements increase substantially. This is especially true when looking for convergence (ie common across all), or worse still thresholds (ie what variants are present in at least x% of the total samples). The larger your data set the more likely this can be a problem. Always make sure your programs are actually completing (NOT just erroring out), and remember there are multiple ways to make the program finish running correctly all be it at a slower rate (read the documents, post on forums, reach out to former instructors, etc).

Based on what was just said, what do you think the first 2 things you should do are?

Need a hint?

Different versions exist
Can be very computationally intensive

Think you know? Check your guess...

Because different versions exist, you need to make sure that you are using the right version:

Check that you remember how to check if you have access to a command, what version that command is, load a different version, and verify that is the version being used

which bedtools  # check if you have access to bedtools
bedtools --version  # determine what version of bedtools you are using
 
# This will tell you that the bedtools you are using is in the $BI/bin directory and that you are using version v2.20.1-2-g6f0be00
 
module load bedtools  # load the TACC version of bedtools, remember module spider module avail and module list can help you find exactly what module you are looking for
which bedtools
bedtools --version
 
# This will now tell you that you are using version v2.26.0 which is located in the /opt/apps/bedtools/2.26.0/bin dreictory.

Because it is computationally intensive, you need to make sure you are not on the head node

Remember to make sure you are on an idev done

For reasons discussed numerous times throughout the course already, please be sure you are on an idev done. It is unlikely that you are currently on an idev node as copying the files while on an idev node seems to be having problems as discussed. Remember the hostname command and showq -u can be used to check if you are on one of the login nodes or one of the compute nodes. If you need more information or help re-launching a new idev node, please see this tutorial.

Surely now that you have executed the above commands nothing could possibly go wrong right?

Hopefully you recognize the bold italic underlined sentence above as a warning that something can (and is likely about to go wrong), but if not ... something is about to go wrong, so try to figure out what that might be, and then click here to see if you are right

If you executed the above commands in the order they are listed from the head node (say because its the first tutorial you ran today and you are not already on a n idev node) you will find that bedtools has reverted to the BioITeam location and version because when you access an idev node it reloads your .bashrc file. Always make sure you are loading tools when and where you need them, and verify their accuracy rather than assuming it. module load bedtools

Comparing the results of different mappers using bedtools

Often you want to compare the results of variant calling on different samples or using different pipelines. Bedtools is a suite of utility programs that work on a variety of file formats, one of which is conveniently VCF format. It provides many ways of slicing, dicing, and comparing the information in VCF files. In this tutorial we will use it on .vcf files generated with samtools after mapping with each of 3 different read mappers (bowtie2, bwa and bowtie) to determine what predictions are the same and which are different from the variant calling on reads mapped with different programs.

Getting some data

For now, create a new directory named GVA_bedtools on scratch and copy the respective VCF files to it from $BI/gva_course/bedtools/:

By this point in the class you should know how to do this. Try to do it on your own and then check your work. If you were wrong, ask a question.

mkdir $SCRATCH/GVA_bedtools 
cd $SCRATCH/GVA_bedtools
cp  -i $BI/gva_course/bedtools/*.vcf .

Remember the above command is simply 1 possible solution there are multiple ways you could have done this, most commonly recursively copying the entire directly, or copying all the files rather than just the vcf files.

Common among all (intersect)

One of if not the most useful links that will be provided in this course is this link to the bedtools intersect page. Not only does it give a very nice graphical interface of what intersect is doing right at the top, but it provides amazing explanations for how to build a good analysis command. If the graphical image is say not what your ideal situation is (i.e. you are interested in seeing where 100% of the samples overlap, not where 1 sample over laps with any of the other samples) there is a command for that as well. Using the link provided, and the bedtools intersect -h command, see if you can identify the regions that were found by all 3 mappers.

Using the subcommands intersect we can find equal and different predictions between mappers. Try to figure out how to to do this on your own first.

Hint:

Remember that adding > output.vcf to the end of a command will pipe the output that is to the terminal into a file, so that you can save it.

Possible solution

bedtools intersect -a bowtie2.vcf -b bowtie.vcf bwa.vcf -f 1.0 -header > all_aligners_with_header.vcf

The above is 1 possible answer. Did you come up with something different? here is what i thought was important to go into the command line

the use of -f 1.0 ensures that only variants in 100% of all the samples are listed

the -header to keep the header information moving forward.

Evaluating the results

Considering the 4 vcf files:

How many variants did each mapper lead to?
Bowtie 220
Bowtie2 133
bwa 120
Want to see how I figured that out? Expand source
grep -c "^NC_" *.vcf
How many mutations were detected in more than 1 mapper?
124
Want to see how I figured that out? Expand source
grep -c "^NC_" *.vcf
For comparison, compare just the two bowtie mappers ... how many vaiants are the same?
34
Want to see how I figured that out? Expand source
bedtools intersect -a bowtie2.vcf -b bowtie.vcf -f 1.0 -header > bowtie_aligners_with_header.vcf grep -c "^NC_" bow*.vcf

How similar are the bowtie2 and bwa results?

90

Want to see how I figured that out?

bedtools intersect -a bowtie2.vcf -b bwa.vcf -f 1.0 -header > bowtie2_v_bwa_with_header.vcf
grep -c "^NC_" b*.vcf

See if there are other comparisons that you are interested in.

Differential comparison (subtract)

Imagine a data set comprised of "disease state" and "normal state" individuals where the disease state is caused by any of some number of dominant (yet rare) mutations. Knowing that mutation is dominant, means that variants detected in the "normal state" are not of interest. How would one remove the variants identified in the "normal state" from the "diseased state"? How about bedtools subtract? Use the bedtools subtract -h command to see if you can put together a command.

First let's simulate some diseased data (say the intersection of bowtie2.vcf and bwa). Click here for example of how to do this:

bedtools intersect -a bowtie2.vcf -b bwa.vcf -f 1.0 -header > diseased.vcf

Next let's pretend that the bowtie.vcf file represents the intersection of the normal group

bedtools subtract -a diseased.vcf -b bowtie.vcf -f 1.0 -header > causitive.vcf

Using the commands above how many potential diseased mutations are there? how many appear they might be causative?

90, 55

Want to see how I figured that out?

grep -c "^NC_" *.vcf

Given what you saw comparing the the read mappers, imagine the 'diseased' samples are bowtie and bowtie2 results and the healthy results are bwa. How many of the mutations causative?

Rather than putting the answer here, message me on zoom with what you think the answer is.

Need a hint?

#commands you need:
bedtools intersect
bedtools subract
grep

Going further

Hopefully you see how these tools can be useful (if not ask). These are but 2 of the bedtools commands, take a few minutes to look through the other bedtools subcommands that are available using bedtools -h, and bedtools other_command -h

You could go back and do this whole tutorial with the data generated data from the SRR030257 fastq files used in the mapping tutorials if you went back and did the analysis with both trimmed and untrimmed reads.

If you have done the advanced breseq tutorial, looking deeper at the 'gdtools' commands you will see you can do similar comparisons with .gd files from breseq. Would be interested to hear if you think this type of comparison would be of interest for a tutorial in the future. Do you see how you could build out a similar set of questions?

Return to GVA20 20

Bioinformatics Team (BioITeam) at the University of Texas

Bedtools tutorial -- GVA2020