/
2022 Pre-processing raw sequences

2022 Pre-processing raw sequences

Jun 01, 2023

Before you start the alignment and analysis processes, it us useful to perform some initial quality checks on your raw data. You may also need to pre-process the sequences to trim them or remove adapters. Here we will assume you have paired-end data from GSAF's Illumina HiSeq sequencer.

Reservations

Use our summer school reservation (CoreNGSday3) when submitting batch jobs to get higher priority on the ls6 normal queue today:

sbatch --reservation=CoreNGSday3 <batch_file>.slurm
idev -m 120 -N 1 -A OTH21164 -r CoreNGSday3

FASTQ Quality Assurance tools

The first order of business after receiving sequencing data should be to check your data quality. This often-overlooked step helps guide the manner in which you process the data, and can prevent many headaches.

FastQC

FastQC is a tool that produces a quality analysis report on FASTQ files.

Useful links:

First and foremost, the FastQC "Summary" should generally be ignored. Its "grading scale" (green - good, yellow - warning, red - failed) incorporates assumptions for a particular kind of experiment, and is not applicable to most real-world data. Instead, look through the individual reports and evaluate them according to your experiment type.

The FastQC reports I find most useful, and why:

  1. Should I trim low quality bases?

    • consult the Per base sequence quality report

      • based on all sequences

  2. Do I need to remove adapter sequences?

    • consult the Adapter Content report

  3. Do I have other contamination?

    • consult the Overrepresented Sequences report

      • based on the 1st 100,000 sequences, trimmed to 75bp

  4. How complex is my library?

    • consult the Sequence Duplication Levels report

    • but remember that different experiment types are expected to have vastly different duplication profiles

For many of its reports, FastQC analyzes only the first ~100,000 sequences in order to keep processing and memory requirements down. Consult the Online documentation for each FastQC report for full details.

Running FastQC

Make sure you're in an idev session. If you're in an idev session, the hostname command will display a name like c455-021.ls6.tacc.utexas.edu. But if you're on a login node the hostname will be something like login2.ls6.tacc.utexas.edu.

If you're on a login node, start an idev session like this:

Start an idev session
idev -m 120 -N 1 -A OTH21164 -r CoreNGSday3

FastQC is available as part of BioContainers on ls6. To make it available:

# Load the main BioContainers module then load the fastqc module module load biocontainers # make take a while module load fastqc

It has a number of options (see fastqc --help | more) but can be run very simply with just a FASTQ file as its argument.

# Setup (if needed) mkdir -p $SCRATCH/core_ngs/fastq_prep cd $SCRATCH/core_ngs/fastq_prep cp $CORENGS/misc/small.fq .
Running fastqc on a FASTQ file
# make sure you're in your $SCRATCH/core_ngs/fastq_prep directory cds cd core_ngs/fastq_prep fastqc small.fq

Exercise: What did FastQC create?

ls -l shows two new items.

-rw-rw-r-- 1 abattenh G-823651 676531 Jun 9 20:53 small_fastqc.html -rw-rw-r-- 1 abattenh G-823651 464403 Jun 9 20:53 small_fastqc.zip

  • small_fastqc.html is the FastQC report, in HTML format.

  • small_fastqc.zip is a zipped (compressed) directory of FastQC output files.

Let's unzip the .zip file and see what's in it.

unzip small_fastqc.zip

What was created?

ls -l shows one new item, the small_fastqc directory (note the "d" in "drwxrwxr-x")

drwxrwxr-x 4 abattenh G-823651 6 Jun 10 2022

ls -l small_fastqc shows the directory contents:

drwxrwxr-x 2 abattenh G-823651 4 Jun 10 2022 Icons drwxrwxr-x 2 abattenh G-823651 9 Jun 10 2022 Images -rw-rw-r-- 1 abattenh G-823651 77464 Jun 10 2022 fastqc.fo -rw-rw-r-- 1 abattenh G-823651 25602 Jun 10 2022 fastqc_data.txt -rw-rw-r-- 1 abattenh G-823651 676531 Jun 10 2022 fastqc_report.html -rw-rw-r-- 1 abattenh G-823651 419 Jun 10 2022 summary.txt

Looking at FastQC output

You can't run a web browser directly from your "dumb terminal" command line environment. The FastQC results have to be placed where a web browser can access them. One way to do this is to copy the results back to your laptop, for example by using scp from your computer (read more at Copying files from TACC to your laptop).

For convenience, we put an example FastQC report at this URL:
https://web.corral.tacc.utexas.edu/BioinformaticsResource/CoreNGS/yeast_stuff/Sample_Yeast_L005_R1.cat_fastqc/fastqc_report.html 

Exercise: Based on this FastQC output, should we trim this data?

The Per base sequence quality report does not look good. The data should probably be trimmed (to 40 or 50 bp) before alignment.

Newer versions of FastQC have slightly different report formats. See this example:
https://web.corral.tacc.utexas.edu/BioinformaticsResource/CoreNGS/reports/wcaar_mqc_report.html

Using MultiQC to consolidate multiple QC reports

FastQC reports are all well and good, but what if you have dozens of samples? It quickly becomes tedious to have to look through all the separate FastQC reports, including separate R1 and R2 reports for paired end datasets.

The MultiQC tool helps address this issue. Once FastQC reports have been generated, it can scan them and create a consolidated report from all the individual reports.

Whats even cooler, is that MultiQC can also consolidate reports from other bioinformatics tools (e.g. bowtie2 aligner statistics, samtools statistics, cutadapt, Picard, and may more). And if your favorite tool is not known by MultiQC, you can configure custom reports fairly easily. For more information, see this recent Byte Club tutorial on Using MultiQC.

Here we're just going to create a MultiQC report for two paried-end ATAC-seq datasets – 4 FASTQ files total. First stage the data:

mkdir -p $SCRATCH/core_ngs/multiqc/fqc.atacseq cd $SCRATCH/core_ngs/multiqc/fqc.atacseq cp $CORENGS/multiqc/fqc.atacseq/*.zip .

You should see these 4 files in your $SCRATCH/core_ngs/multiqc/fqc.atacseq directory:

50knuclei_S56_L007_R1_001_fastqc.zip 5knuclei_S77_L008_R1_001_fastqc.zip 50knuclei_S56_L007_R2_001_fastqc.zip 5knuclei_S77_L008_R2_001_fastqc.zip

Now make the BioContainers MultiQC accessible in your environment.

Make sure you're in an idev session. If you're in an idev session, the hostname command will display a name like c455-020.ls6.tacc.utexas.edu. But if you're on a login node the hostname will be something like login1.ls6.tacc.utexas.edu.

If you're on a login node, start an idev session like this:

Start an idev session
idev -m 120 -N 1 -A OTH21164 -r CoreNGSday3
# Load the main BioContainers module if you have not already module load biocontainers # may take a while # Load the multiqc module and ask for its usage information module load multiqc multiqc --help
mkdir -p $SCRATCH/core_ngs/multiqc/fqc.atacseq cd $SCRATCH/core_ngs/multiqc/fqc.atacseq cp $CORENGS/multiqc/fqc.atacseq/*.zip .

Even though multiqc has many options, it is quite easy to create a basic report by just pointing it to the directory where individual reports are located:

cd $SCRATCH/core_ngs/multiqc multiqc fqc.atacseq

Exercise: How many reports did multiqc find?

Based on its execution output, it found 4 reports

[WARNING] multiqc : MultiQC Version v1.12 now available! [INFO ] multiqc : This is MultiQC v1.7 [INFO ] multiqc : Template : default [INFO ] multiqc : Searching 'fqc.atacseq/' [INFO ] fastqc : Found 4 reports [INFO ] multiqc : Compressing plot data [INFO ] multiqc : Report : multiqc_report.html [INFO ] multiqc : Data : multiqc_data [INFO ] multiqc : MultiQC complete

Exercise: What was created by running multiqc?

One file was created (multiqc_report.html) and one directory (multiqc_data).

You can see the resulting MultiQC report here: https://web.corral.tacc.utexas.edu/BioinformaticsResource/CoreNGS/reports/atacseq/multiqc_report.html.

And an example of a MultiQC report that includes both standard and custom plots is this is the Tag-Seq post-processing MultiQC report produced by the Bioinformatics Consulting Group: https://web.corral.tacc.utexas.edu/BioinformaticsResource/CoreNGS/reports/mqc_tagseq_trim_JA21030_SA21045_mouse.html

Trimming sequences

There are two main reasons you may want to trim your sequences:

  • As a quick way to remove 3' adapter contamination, when extra bases provide little additional information

    • For example, 75+ bp ChIP-seq reads – 50 bases are more than enough for a good mapping, and trimming to 50 is easier than adapter removal, especially for paired end data.

    • You would not choose this approach for RNA-seq data, where 3' bases may map to a different exon, and that is valuable information.

      • Instead you would specifically remove adapter sequences.

  • Low quality base reads from the sequencer can affect some programs

    • This is an issue with sequencing for genome or transcriptome assembly.

    • Aligners such as bwa and bowtie2 seem to do fine with a few low quality bases, soft clipping them if necessary.

There are a number of open source tools that can trim off 3' bases and produce a FASTQ file of the trimmed reads to use as input to the alignment program.

FASTX Toolkit

The FASTX Toolkit provides a set of command line tools for manipulating both FASTA and FASTQ files. The available modules are described on their website. They include a fast fastx_trimmer utility for trimming FASTQ sequences (and quality score strings) before alignment.

Make sure you're in an idev session. If you're in an idev session, the hostname command will display a name like c455-021.ls6.tacc.utexas.edu. But if you're on a login node the hostname will be something like login3.ls6.tacc.utexas.edu.

If you're on a login node, start an idev session like this:

Start an idev session
idev -m 120 -N 1 -A OTH21164 -r CoreNGSday3

FASTX Toolkit is available as a BioContainers module.

module load biocontainers # takes a while module spider fastx module load fastxtools

Here's an example of how to run fastx_trimmer to trim all input sequences down to 50 bases. By default the program reads its input data from standard input and writes trimmed sequences to standard output:

Set up directory for working with FASTQs
# Create a $SCRATCH area to work on data for this course, # with a sub-direct[1ory for pre-processing raw fastq files mkdir -p $SCRATCH/core_ngs/fastq_prep # Make a symbolic links to the original yeast data: cd $SCRATCH/core_ngs/fastq_prep ln -s -f $CORENGS/yeast_stuff/Sample_Yeast_L005_R1.cat.fastq.gz ln -s -f $CORENGS/yeast_stuff/Sample_Yeast_L005_R2.cat.fastq.gz
Trimming FASTQ sequences to 50 bases with fastx_trimmer
# make sure you're in your $SCRATCH/core_ngs/fastq_prep directory cd $SCRATCH/core_ngs/fastq_prep zcat Sample_Yeast_L005_R1.cat.fastq.gz | fastx_trimmer -l 50 -Q 33 > trim50_R1.fq

  • The -l 50 option says that base 50 should be the last base (i.e., trim down to 50 bases)

  • The -Q 33 option specifies how base Qualities on the 4th line of each FASTQ entry are encoded.

    • The FASTX Toolkit is an older program written in the time when Illumina base qualities were encoded differently, so its default does not work for modern FASTQ files.

    • These days Illumina base qualities follow the Sanger FASTQ standard (Phred score + 33 to make an ASCII character).

Exercise: compressing fastx_trimmer output

How would you tell fastx_trimmer to compress (gzip) its output file?

Type fastx_trimmer -h (help) to see program documentation

You could supply the -z option like this:

zcat Sample_Yeast_L005_R1.cat.fastq.gz | fastx_trimmer -l 50 -Q 33 -z > trim50_R1.fq.gz

Or you could gzip the output yourself.

zcat Sample_Yeast_L005_R1.cat.fastq.gz | fastx_trimmer -l 50 -Q 33 | gzip > trim50_R1.fq.gz

Exercise: other fastx toolkit programs

What other FASTQ manipulation programs are part of the FASTX Toolkit?

Type fastx_ then tab twice (completion) to see their names.

The FASTX Toolkit also has programs that work on FASTA files. To see them, type fasta_ then tab twice (completion) to see their names.

Adapter trimming with cutadapt

Data from RNA-seq or other library prep methods that result in short fragments can cause problems with moderately long (50-100bp) reads, since the 3' end of sequence can be read through to the 3' adapter at a variable position. This 3' adapter contamination can cause the "real" insert sequence not to align because the adapter sequence does not correspond to the bases at the 3' end of the reference genome sequence.

Unlike general fixed-length trimming (e.g. trimming 100 bp sequences to 50 bp), specific adapter trimming removes differing numbers of 3' bases depending on where the adapter sequence is found.

You must tell any adapter trimming program what your R1 and R2 adapters look like.

The GSAF website describes the flavors of Illumina adapter and barcode sequences in more detail:  Illumina - all flavors (USE with Caution, this is outdated but can be useful for a basic understanding of the adapters, the GSAF primarily only uses UDI's for all projects).

The cutadapt program, available in BioContainers, is an excellent tool for removing adapter contamination.

Make sure you're in an idev session. If you're in an idev session, the hostname command will display a name like c455-021.ls6.tacc.utexas.edu. But if you're on a login node the hostname will be something like login3.ls6.tacc.utexas.edu.

If you're on a login node, start an idev session like this:

Start an idev session
idev -m 120 -N 1 -A OTH21164 -r CoreNGSday3
module load biocontainers module spider cutadapt module load cutadapt cutadapt --help

A common application of cutadapt is to remove adapter contamination from RNA library sequence data. Here we'll show that for some small RNA libraries sequenced by GSAF, using their documented small RNA library adapters.

When you run cutadapt you give it the adapter sequence to trim, and this is different for R1 and R2 reads. Here's what the options look like (without running it on our files yet).

cutadapt command for R1 sequences (GSAF RNA library)
cutadapt -m 22 -O 4 -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC <fastq_file>
cutadapt command for R2 sequences (GSAF RNA library)
cutadapt -m 22 -O 4 -a TGATCGTCGGACTGTAGAACTCTGAACGTGTAGA <fastq_file>

Notes:

  • The -m 22 option says to discard any sequence that is smaller than 22 bases (minimum) after trimming.

    • This avoids problems trying to map very short, highly ambiguous sequences.

  • the -O 4 (Overlap) option says not to trim 3' adapter sequences unless at least the first 4 bases of the adapter are seen at the 3' end of the read.

    • This prevents trimming short 3' sequences that just happen by chance to match the first few adapter sequence bases.

Figuring out which adapter sequence to use when can be tricky. Your sequencing provider can tell you what adapters they used to prep your libraries. For GSAF's adapter layout, please refer to Illumina - all flavors (USE with Caution, this is outdated but can be useful for a basic understanding of the adapters, the GSAF primarily only uses UDI's for all projects) (you may want to read all the "gory details" below later).

The top strand, 5' to 3', of a read sequence looks like this.

Illumina library read layout
<P5 capture> <indexRead2> <Read 1 primer> [insert] <Read 2 primer> <indexRead1> <P7 capture>

The -a argument to cutadapt is documented as the "sequence of adapter that was ligated to the 3' end". So we care about the <Read 2 primer> for R1 reads, and the <Read 1 primer> for R2 reads.

The "contaminent" for adapter trimming will be the <Read 2 primer> for R1 reads. There is only one Read 2 primer:

Read 2 primer, 5' to 3', used as R1 sequence adapter
AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC

The "contaminent" for adapter trimming will be the <Read 1 primer> for R2 reads. However, there are three different Read 1 primers, depending on library construction:

Read 1 primer depends on library construction
TCTACACGTTCAGAGTTCTACAGTCCGACGATCA # small RNA sequencing primer site CAGGTTCAGAGTTCTACAGTCCGACGATCA # "other" TCTACACTCTTTCCCTACACGACGCTCTTCCGATCT # TruSeq Read 1 primer site. This is the RC of the R2 adapter

Since R2 reads are the reverse complement of R1 reads, the R2 adapter contaminent will be the RC of the Read 1 primer used.

For ChIP-seq libraries where reads come from both DNA strands, the TruSeq Read 1 primer is always used.
Since it is the RC of the Read 2 primer, its RC is just the Read 1 primer back.
Therefore, for ChIP-seq libraries the same cutadapt adapter option can be used for both R1 and R2 reads:

Cutadapt adapter sequence for ChIP-seq lib
cutadapt -a GATCGGAAGAGCACACGTCTGAACTCCAGTCAC

For RNAseq libraries in this class, we use the small RNA sequencing primer as the Read 1 primer.
The contaminent is then the RC of this, minus the 1st and last bases:

Small RNA library Read 1 primer, 5' to 3', used as R2 sequence adapter
TCTACACGTTCAGAGTTCTACAGTCCGACGATCA # R1 primer - small RNA sequencing Read 1 primer site, 5' to 3' TGATCGTCGGACTGTAGAACTCTGAACGTGTAGA # R2 adapter contaminent (RC of R1 small RNA sequencing Read 1 primer)

Exercise: other cutadapt options

The cutadapt program has many options. Let's explore a few.

How would you tell cutadapt to trim trailing N's?

cutadapt --help | less

Then, in the less pager, type /trim <enter> to look for the first occurrence of the string "trim", then n to look for subsequent occurrences.

The relevant option is --trim-n

How would you control the accuracy (error rate) of cutadapt's matching between the adapter sequences and the FASTQ sequences?

Use the less pager to search for terms like "error" or "accuracy".

cutadapt --help | less

Then, in the less pager, type /error <enter> to look for the first occurrence of the string "error", then n to look for subsequent occurrences.

The relevant option is  -e <floating point error rate> or --error-rate=<floating point error rate>:

-e ERROR_RATE, --error-rate=ERROR_RATE Maximum allowed error rate (no. of errors divided by the length of the matching region) (default: 0.1)

Suppose you are processing 100 bp reads with 30 bp adapters. By default, how many mismatches between the adapter and a sequence will be tolerated?

cutadapt's default error rate is 0.1 (10%)

Up to three mismatches will be tolerated when the whole 30 bp adapter is found (10% of 30).

If only 20 of the 30 adapter bases are found, up to two mismatches will be tolerated (10% of 20).

How would you require a more stringent matching (i.e., allowing fewer mismatches)?

Providing --error-rate=0.05 (or -e 0.05) as an option, for example, would specify a 5% error rate, or no more than 1 mismatching base in 20.

cutadapt example

Let's run cutadapt on some real human miRNA data.

First, stage the data we want to use. This data is from a small RNA library where the expected insert size is around 15-25 bp.

Setup for cutadapt on miRNA FASTQ
mkdir -p $SCRATCH/core_ngs/fastq_prep cd $SCRATCH/core_ngs/fastq_prep cp $CORENGS/human_stuff/Sample_H54_miRNA_L004_R1.cat.fastq.gz . cp $CORENGS/human_stuff/Sample_H54_miRNA_L005_R1.cat.fastq.gz .


Exercise: How many reads are in these files? Is it single end or paired end data?

echo $(( `zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | wc -l` / 4 )) # or zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | wc -l | awk '{print $1 / 4}'

Looking at the FASTQ file names, we see this is two lanes of single-end data (L004 and L005).

The data from lane 4 has 2,001,337 reads, the data from lane 5 has 2,022,237 reads.

Exercise: How long are the reads?

You could just Look at the size of the actual sequence on the 2nd line of any FASTQ entry and count the characters....

But you're experts now! So challenge yourself.

Use a combination of tail and head to extract the 2nd line of the .gz file.

Then use the wc program, but not with the -l option (check wc --help).

zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | head -2 | tail -1 | wc -c

These are 101-base reads. wc -c counts the "invisible" newline character, so subtract 1 from the character count it returns for each line (here there is only 1 line).

Adapter trimming is a rather slow process, and these are large files. So to start with we're going to create a smaller FASTQ file to work with.

zcat Sample_H54_miRNA_L004_R1.cat.fastq.gz | head -2000 > miRNA_test.fq

Now execute cutadapt like this:

Setup for cutadapt on miRNA FASTQ
mkdir -p $SCRATCH/core_ngs/fastq_prep cd $SCRATCH/core_ngs/fastq_prep cp $CORENGS/human_stuff/miRNA_test.fq .
Cutadapt batch command for R1 FASTQ
cutadapt -m 20 -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC miRNA_test.fq 2> miRNA_test.cuta.log | gzip > miRNA_test.cutadapt.fq.gz

Notes:

  • Here's one of those cases where you have to be careful about separating standard output and standard error. 

    • cutadapt writes its FASTQ output to standard output, and writes summary information to standard error.

  • In this command we first redirect standard error to a log file named miRNA_test.cuta.log using the 2> syntax, in order to capture cutadapt diagnostics.

    • Then standard output is piped to gzip, whose output is the redirected to a new compressed FASTQ file.

    • (Read more about Standard I/O streams)

You should see a miRNA_test.cuta.log log file when the command completes. How many lines does it have?

wc -l miRNA*.log

 

Take a look at the first 15 lines.

head -15 miRNA_test.cuta.log

 

It will look something like this:

cutadapt log file
This is cutadapt 1.18 with Python 3.7.1 Command line parameters: -m 20 -a AGATCGGAAGAGCACACGTCTGAACTCCAGTCAC miRNA_test.fq Processing reads on 1 core in single-end mode ... Finished in 0.06 s (113 us/read; 0.53 M reads/minute). === Summary === Total reads processed: 500 Reads with adapters: 492 (98.4%) Reads that were too short: 64 (12.8%) Reads written (passing filters): 436 (87.2%) Total basepairs processed: 50,500 bp Total written (filtered): 10,909 bp (21.6%)

Notes:

  • The Total reads processed line tells you how many sequences were in the original FASTQ file.

  • Reads with adapters tells you how many of the reads you gave it had at least part of an adapter sequence that was trimmed.

    • Here adapter was found in nearly all (98.4%) of the reads. This makes sense given this is a short (15-25 bp) RNA library.

  • The Reads that were too short line tells you how may sequences were filtered out because they were shorter than our minimum length (20) after adapter removal (these may have ben primer dimers).

    • Here ~13% of the original sequences were removed, which is reasonable.

  • Reads written (passing filters) tells you the total number of reads that were written out by cutadapt

    • These are reads that were at least 20 bases long after adapters were removed

paired-end data considerations