Bastien
Chevreux
bach@chevreux.org

> Solomon Short

> New problems demand new solutions. New solutions create new problems.

# Assembly of Pacific Biosciences data with MIRA3

## MIRA 3.4.0: currently only CCS or error-corrected-CLR supported

This is here to set the stage: at the moment, MIRA can only make use of PacBio reads which have an error rate of roundabout 5%. This means you will have to use either:

1. CCS (Circular Consensus Sequence) reads with at least 3 to 4 passes.

2. CLR (Continuous Long Reads) which were error corrected either with PacBio CCS reads or some other high-quality sequencing technology (Illumina comes to mind)

Impatients can directly jump to which contains walkthroughs using data made publicly available by PacBio.

## WARNING

When I developed the routines for PacBio, I had no access to their data and it shows. Now that first numbers regarding their reads get published and the first data sets available publicly, I realise a couple of preconditions were not met. Especially the fact that raw PacBio CLR reads seem to have an error rate between 1-in-5 (80% correct) and 1-in-7 (85% correct) means that MIRA will probably not operate very well with those. One should have something between 1-in-12 to 1-in-20 as error rate (92% to 95% correct) to get MIRA working happily.

During the course of 2011, PacBio has made available on their DevNet site quite a number of documents and introductory videos. A must read for everyone interested in this sequencing technology.

As of MIRA 3.4.0, large parts of this documentation are still from a time where things like terminology, sequencing specifics etc.pp were not available publicly. It'll take same time for me to convert the guide.

## Introduction

Pacific Biosciences looks like the new kid on the block of sequencing technologies. They seem to have, for the first time since Sanger sequencing, something which is able to produce sequences which are actually longer than Sanger. They also have something new: strobed sequencing. That technique alone was reason enough for me to see whether it could be of any use. After a couple of modifications to the MIRA assembly engine, I think I can say that "yes, it very well can be."

One could feed strobed PacBio sequences to MIRA 3.0.0 and the 2.9.x line before and get some results out of it by faking them to be Sanger, though the results were not always pretty.

The first version of MIRA to officially support sequences from Pacific Biosciences is MIRA 3.2. Versions in the 3.0.1 to 3.0.5 range and 3.1.x had different degrees of support, but were never advertised having it.

### Some reading requirements

This guide assumes that you have basic working knowledge of Unix systems, know the basic principles of sequencing (and sequence assembly) and what assemblers do.

While there are step by step walk-throughs on how to setup your data for Sanger, 454 and Solexa in other MIRA guides, this guide is (currently) a bit more terse. You are expected to read at some point in time

- the mira_reference help file to look up some command line options.

- for hybrid assemblies of PacBio data with Sanger, 454, Solexa the corresponding mira_usage, mira_454 or mira_solexa help files to look up how to prepare the different data sets.

### Probable highlights and lowlights of PacBio sequencing data

#### Lowlights

Sequence quality, indels, maybe chimeras.

#### Highlights

Long reads, maybe strobe sequencing

## Assembly of PacBio data with MIRA

### Preparing data

MIRA will happily read data in several different formats (FASTA, FASTQ, etc.). For the sake of simplicity, this guide will use FASTQ as demonstration format, but most of the time not add the quality line.

#### Simple, unstrobed, non-paired reads

This is actually quite simple. Just put your reads as FASTQ in a file and you are done. No need to bother about read naming conventions or similar things. Like so:

    @readname_001
    ACGTTGCAGGGTCATGCAGT...
    @readname_002
    ...

#### Strobed reads with two strobes, simulating paired ends

It is probably preferable to keep both parts physically separated in different reads. In this case, read naming becomes important for an assembler. Pick any paired-end naming scheme you want and name your reads accordingly. The following example shows the same data as above, split in two reads and using the Solexa naming scheme to denote the first read of a pair by appending `/1` to the read name and the second part by appending `/2`:

    @readname_001/1
    ACGTTGCAGG
    @readname_001/2
    GTCATGCAGT
    @readname_002/1
    ...

The example above used Solexa naming scheme to denote paired-end partner reads. You can use any naming scheme you want as long as MIRA knows it. E.g.: the forward/reverse or Sanger or TIGR naming schemes, you will just need to tell MIRA about it with the segmentnaming parameter in the manifest file.

Also note you might want to set segmentplacement in the manifest, as well as templatesize.

Should all your reads approximately have the same total length of first part (/1) + insert + second part (/2), then you don't need to create an additional file with information about expected distance between the parts, you can use the template and segment information in the manifest to tell MIRA about it. In case you have different sizes because, e.g. you have sequenced different libraries, then you will need to tell MIRA which reads have which distance from each other. You can do this in an XML file in TRACEFORMAT (as defined by the NCBI).

#### Strobed reads with multiple strobes

You should split the parts separated by these long inserts into different reads. E.g., if your strobes are 500 bases long, but separated by dark inserts > 1Kb, split them. You are free to split them however you like, in sub-pairs, in single strobes or whatever. For the example given above, this could be done like this:

    @readname_001a/1
    ACGTTGCAGG
    @readname_001a/2
    GTCATGCAGT
    @readname_001b/1
    TATGCACTGAC
    @readname_001b/2
    TAGCTGA
    @readname_002
    ...

which would then be two read-pairs: the first and second strobes are paired, as well as the third and fourth. Here too, you can use any combination strobes to pair to each other (or to use without pair information).

Combining first and fourth strobe as well as second and fourth would look like this:

    @readname_001a/1
    ACGTTGCAGG
    @readname_001b/1
    TAGCTGA
    @readname_001b/2
    GTCATGCAGT
    @readname_001a/2
    TATGCACTGAC
    @readname_002
    ...

Note that in this case you probably need to provide paired-end information in a NCBI TRACEARCHIVE XML file to tell MIRA about the different insert sizes.

Finally, you can put the reads all in one template like this:

    @readname_001.f1
    ACGTTGCAGG
    @readname_001.f2
    GTCATGCAGT
    @readname_001.f3
    TATGCACTGAC
    @readname_001.f4
    TAGCTGA
    @readname_002
    ...

Note the subtle change in the naming of reads where I changed to a different postfix naming. This is because the Solexa naming scheme currently does not (officially) allow for more than two reads per DNA template (well, /1 and /2). The forward/reverse naming scheme like implemented by MIRA however does allow this.

This has just one drawback: currently MIRA will not be able to store the distances between the strobes when they are all in one template. This is being worked on and will be possible in a future version.

### Setting up files and directories

### Launching MIRA

In the most basic incantation, you will need to tell MIRA just five things:

1. the name of your project.

2. whether you want a "genome" or "EST" assembly

3. whether it is a denovo or mapping assembly

4. which quality level (draft, normal or accurate)

5. the data you want to use

The above needs to be written together in a manifest file, which is a file listing the MIRA configuration and the data it should use. Have a look at the walkthrough below and at the MIRA reference to get an idea on how to write such a file (it's simple, really). Then start MIRA like this:

    mira manifest.conf >&log_assembly.txt

MIRA has -- at the last count -- more than 150 parameters one can use to fine tune almost every aspect of an assembly, from data loading options to results saving, from data preprocessing to results interpretation, from simple alignment parameters to parametrisation of internal misassembly decision rules ... and much more. Many of these parameters can be even set individually for each sequencing technology they apply to. Example given: in an assembly with Solexa, Sanger, 454 and PacBio data, the minimum read length for Solexa could be set to 30, while for 454 it could be 80, Sanger 100 and PacBio 150. Please refer to the reference manual for a full overview on how to use quick switches and extended switches in manifest files.

## Walkthroughs: real data sets from PacBio

We'll use some data provided by PacBio for the E. coli O104:H4 outbreak in 2011, see for more info.

### Error corrected CLR for E. coli C227-11$

That data set is quite interesting: PacBio took CLR reads (the reads with only ~85% accuracy) and mapped CCS reads (presumably >00% accuracy) to them to correct errors of the CLR reads. The resulting error corrected CLR data is of pretty good quality, not only from the quality values but when assembled, the number of sequencing errors in the reads which can be spotted in the aligments is obviously quite low.

#### Preparing a directory structure

Note: this is how I set up a project, feel free to implement whatever structure suits your needs.

    $ mkdir c227-11-clrc
    $ cd c227-11-clrc
    arcadia:c227-11-clrc$ mkdir origdata data assemblies

Your directory should now look like this:

    arcadia:c227-11-clrc$ ls -l
    drwxr-xr-x 2 bach users 48 2011-08-19 20:21 assemblies
    drwxr-xr-x 2 bach users 48 2011-08-19 20:21 data
    drwxr-xr-x 2 bach users 48 2011-08-19 20:21 origdata

"c227-11-clrc" is an arbitrary name I just chose by concatenating the name of the bug and "-clrc" to indicate that this project has CLR sequences which were Corected. But you can name this whatever you want: foobar, blafurbsel, ...

Explanation of the structure:

- the `origdata` directory will contain the 'raw' result files that one might get from sequencing. In our case it will be the `.tar.gz` file with the data in FASTQ format from the Devnet site.

- the `data` directory will contain the preprocessed sequences for the assembly, ready to be used by MIRA

- the `assemblies` directory will contain assemblies we make with our data (we might want to make more than one).

#### Getting the data and preparing it

Head over to PacBio DevNet and fetch the data set for the E. coli C227-11 CLR corrected data set. Put it into the `origdata` directory created a few moments ago.

Now, let's extract the data to the `data` directory:

    arcadia:c227-11-clrc$ cd data
    arcadia:data$ tar xvzf ../origdata/e-coli-c227-11-corrected-fastq-1.2.2beta.tgz
    e-coli-c227-11-corrected-fastq-1.2.2beta/
    e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq

One thing you would quickly find out but which I tell now to save time: at the moment, PacBio seems to love ultra long read names. Here are the first 10 from the current data set:

    arcadia:data$ grep ^@m e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq | head -10
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10040/0_5174/c0
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10040/0_5174/c1
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1017/0_1636/c0
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/0_4073/c0
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/0_4073/c1
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/1054/4121_4891/c0
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10548/0_5766/c0
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10548/0_5766/c1
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/10640/0_2393/c0
    @m110618_035655_42142_c100158802555500000315044108071130_s1_p0/11000/0_3285/c0

How sweet! Read names with 80 and more characters AND having the "/" character as component, the later being a recipe for desaster sooner or later in some post-processing pipelines.

MIRA has absolutely no problem with the above: neither with long read names nor with the "/" character in the name. However, long read names are a problem for example for `gap4` (an assembly viewer) and the "/" character might lead to confusion with the standard UNIX directory separator, many programs which can be used to post-process assemblies simply disallow that character.

For the sake of simplicity and compatibility, let's rename all sequences. For this we'll use `convert_project`, which is a binary of the MIRA program package:

    arcadia:data$ convert_project
      -f fastq -t fastq -R c227-11-clrc
      e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq
      c227-11-clrc
    Loading from fastq, saving to: fastq
    Loading data from FASTQ ...
    Counting sequences in FASTQ file: found 73496 sequences.
    Localtime: Sat Aug 20 20:36:26 2011
    Unusual offset of 34, guessing this file to be a Sanger-type FASTQ format.
    Using calculated FASTQ quality offset: 33
    Localtime: Sat Aug 20 20:36:26 2011
    Loading data from FASTQ file:
     [0%] ....|.... [10%] ....|.... [20%] ....|.... [30%] ....|.... [40%] ....|....
    [50%] ....|.... [60%] ....|.... [70%] ....|.... [80%] ....|.... [90%] ....|.... [100%]

    Done.
    Loaded 73496 reads, Localtime: Sat Aug 20 20:36:35 2011
     done.
    Data conversion process finished, no obvious errors encountered.

The above command has been split in multiple lines for better overview but should be entered in one line.

The parameters to `convert_project` say

- -f fastq: the "from" type (type of the input file) is FASTQ

- -t fastq: the "to" type (type of the output file) should be FASTQ

- -R c227-11-clrc: sequences should be renamed, all starting with "c227-11-clrc" and to which `convert_project` will append an underscore and a counter.

- e-coli-c227-11-corrected-fastq-1.2.2beta/e-coli-c227-11-corrected.fastq: that's the full name of our input file.

- c227-11-clrc: that's the partial name of our output file. Partial because `convert_project` will automatically add the postfix of the target format to the name, in this case `.fastq`.

  Just in case you wonder why this works like this: `convert_project` can convert to multiple formats at once, e.g., like this: convert_project -f fastq -t fasta -t fastq -t maf ... and then you will get nicely named files.

Your directory should now look like this ...

    arcadia:data$ ls -l
    -rw-r--r-- 1 bach bach 257844076 2011-08-20 21:04 c227-11-clrc.fastq
    drwxr-x--- 2 bach bach      4096 2011-07-22 04:49 e-coli-c227-11-corrected-fastq-1.2.2beta

... and as we do not need the subdirectory with the extracted data from PacBio anymore, let's get rid of it:

    arcadia:data$ rm -rf e-coli-c227-11-corrected-fastq-1.2.2beta
    arcadia:data$ ls -l
    -rw-r--r-- 1 bach bach 257844076 2011-08-20 21:04 c227-11-clrc.fastq

Perfect, we're done here.

#### Starting the assembly

Good, we're almost there. Let's switch to the `assembly` directory and create a subdirectory for our first assembly test.

    arcadia:data$ cd ../assemblies/
    arcadia:assemblies$ mkdir 1sttest
    arcadia:assemblies$ cd 1sttest

This directory is quite empty and the PacBio data is not present. And it does not need to be as we'll tell MIRA in the manifest configuration file where to find the data and what to do with it.

    # A manifest file can contain comment lines, these start with the #-character

    # First part of a manifest: defining some basic things

    # In this example, we just give a name to the assembly
    #  and tell MIRA it should assemble a genome de-novo in accurate mode
    # As special parameter, we want to use 4 threads in parallel instead
    #  of letting MIRA decide how many CPU cores it should use

    project = c227-11-clrc
    job = genome,denovo,accurate
    parameters = -GE:not=4

    # The second part defines the sequencing data MIRA should load and assemble
    # The data is logically divided into "readgroups", for more information
    #  please consult the MIRA manual, chapter "Reference"

    readgroup = MyPacBioCLRCReads
    technology = pcbiohq
    data = ../../data/c227-11-clrc.fastq

Starting the assembly is now just a matter of one line:

    arcadia:1sttest$ mira manifest.conf >&log_assembly.txt

Some 3 to 4 hours later, you should have a nice and shiny assembly of your data.

Now, that was easy, wasn't it? In the above example - for assemblies having only PacBio data and if you followed the walkthrough on how to prepare the data - everything you might want to adapt in the first time are the following options:

- project= (for naming your assembly project)

- job= (perhaps to change the quality of the assembly to 'draft'

Of course, you are free to change any of the > options via the extended parameters MIRA has ... but this is covered in the MIRA main reference manual.

#### Working with the results of the assembly

There is a whole chapter in the manual dedicated to this, you are expected to read it :-)

However, for the impatient, here's a quick rundown on what I am going to show as example in this section: filtering of results and loading results into an assembly viewer.

##### Filtering results

This assembly project has an average coverage of roundabout 23 to 24x. Due to sequencing errors, MIRA will have also created a few small contigs with just a few reads and lot less coverage. These contigs are, well, most of the time not interesting as they contain junk most of the time. I want to get rid of them.

Let's say that only contigs ≥ 500 base pairs and with an average coverage ≥ 8 (which is 1/3 of the average coverage of 24 of the whole project) are interesting. Let's filter them out of the MIRA results and create a CAF file which can then be imported into either `gap4` or `gap5` (assembly viewers and finishing tools from the Staden package).

Let's take a quick look at the main directories and files of the assembly:

    arcadia:1sttest$ ls -l
    drwxr-xr-x 6 bach bach    4096 2011-08-20 16:57 c227-11-clrc_assembly
    -rw-r--r-- 1 bach bach 2524406 2011-08-20 19:46 log_assembly.txt
    arcadia:1sttest$ cd c227-11-clrc_assembly
    arcadia:c227-11-clrc_assembly$ ls -l
    drwxr-xr-x 2 bach bach  4096 2011-08-20 17:01 c227-11-clrc_d_chkpt
    drwxr-xr-x 2 bach bach  4096 2011-08-20 19:46 c227-11-clrc_d_info
    drwxr-xr-x 2 bach bach  4096 2011-08-20 20:10 c227-11-clrc_d_results
    drwxr-xr-x 2 bach bach 36864 2011-08-20 19:46 c227-11-clrc_d_tmp
    arcadia:c227-11-clrc_assembly$ ls -l c227-11-clrc_d_info
    -rw-r--r-- 1 bach bach     2320 2011-08-20 19:46 c227-11-clrc_info_assembly.txt
    -rw-r--r-- 1 bach bach       88 2011-08-20 16:57 c227-11-clrc_info_callparameters.txt
    -rw-r--r-- 1 bach bach   168049 2011-08-20 19:46 c227-11-clrc_info_consensustaglist.txt
    -rw-r--r-- 1 bach bach  1599427 2011-08-20 19:46 c227-11-clrc_info_contigreadlist.txt
    -rw-r--r-- 1 bach bach     6718 2011-08-20 19:46 c227-11-clrc_info_contigstats.txt
    -rw-r--r-- 1 bach bach     7401 2011-08-20 19:46 c227-11-clrc_info_debrislist.txt
    -rw-r--r-- 1 bach bach    10572 2011-08-20 19:15 c227-11-clrc_info_readrepeats.lst
    -rw-r--r-- 1 bach bach 42697892 2011-08-20 19:46 c227-11-clrc_info_readtaglist.txt
    arcadia:c227-11-clrc_assembly$ cd c227-11-clrc_d_results
    arcadia:c227-11-clrc_results$ ls -l
    -rw-r--r-- 1 bach bach 192615652 2011-08-20 19:46 c227-11-clrc_out.ace
    -rw-r--r-- 1 bach bach 597368574 2011-08-20 19:46 c227-11-clrc_out.caf
    -rw-r--r-- 1 bach bach 306918692 2011-08-20 19:46 c227-11-clrc_out.maf
    -rw-r--r-- 1 bach bach   6333500 2011-08-20 19:46 c227-11-clrc_out.padded.fasta
    -rw-r--r-- 1 bach bach  18987968 2011-08-20 19:46 c227-11-clrc_out.padded.fasta.qual
    -rw-r--r-- 1 bach bach      7240 2011-08-20 19:46 c227-11-clrc_out.tcs
    -rw-r--r-- 1 bach bach   6297565 2011-08-20 19:46 c227-11-clrc_out.unpadded.fasta
    -rw-r--r-- 1 bach bach  18881604 2011-08-20 19:46 c227-11-clrc_out.unpadded.fasta.qual
    -rw-r--r-- 1 bach bach   4442567 2011-08-20 19:46 c227-11-clrc_out.wig

OK, we're at the right spot for filtering. While we are at it, tell `convert_project` to not only convert to CAF, but also to write a new file with tabular data on contig statistics of the filtered contigs:

    arcadia:c227-11-clrc_results$ convert_project
      -f maf -t caf -t cstats -x 500 -y 8
      c227-11-clrc_out.maf
      c227-11-clrc_filteredx500y8
    Loading from maf, saving to: caf cstats
    First counting reads:
     [0%] ....|.... [10%] ....|.... [20%] ....|.... [30%] ....|.... [40%] ....|.... [50%] ....|.... [60%] ....|.... [70%] ....|.... [80%] ....|.... [90%] ....|.... [100%]
    Now loading and processing data:

... lots of lines omitted ...

    Data conversion process finished, no obvious errors encountered.
    arcadia:c227-11-clrc_results$ ls -l *filtered*
    -rw-r--r-- 1 bach bach 584042411 2011-08-20 22:12 c227-11-clrc_filteredx500y8.caf
    -rw-r--r-- 1 bach bach      1518 2011-08-20 22:12 c227-11-clrc_filteredx500y8_info_contigstats.txt

    arcadia:c227-11-clrc_results$ cat c227-11-clrc_filteredx500y8_info_contigstats.txt
    # name                  length  av.qual #-reads mx.cov. av.cov  GC%     CnIUPAC CnFunny CnN     CnX     CnGap   CnNoCov
    c227-11-clrc_c1         335375  90      4081    35      21.27   49.73   2       0       0       0       1975    0
    c227-11-clrc_c2         651370  90      9224    41      23.88   51.23   3       0       0       0       4535    0
    c227-11-clrc_c3         356318  90      4962    40      24.18   50.81   0       0       0       0       2208    0
    c227-11-clrc_c4         386288  90      5178    39      23.58   51.49   3       0       0       0       2367    0
    c227-11-clrc_c5         908271  90      12277   40      23.41   50.73   3       0       0       0       5912    0
    ...

Once filtered, how many contigs are there? Well, 22: it's the number of lines minus one of the file `c227-11-clrc_filteredx500y8_info_contigstats.txt`:

    arcadia:c227-11-clrc_results$ wc -l c227-11-clrc_filteredx500y8_info_contigstats.txt
    23

##### Looking at the assembly in gap4

I'm very fond of gap4, so I'll use it to show ho the assembly looks like:

    arcadia:c227-11-clrc_results$ caf2gap -project c227-11 -ace c227-11-clrc_filteredx500y8.caf >&/dev/null
    arcadia:c227-11-clrc_results$ ls -l C*
    -rw-r--r-- 1 bach bach 543539856 2011-08-20 22:35 C227-11.0
    -rw-r--r-- 1 bach bach  39512896 2011-08-20 22:35 C227-11.0.aux
    arcadia:c227-11-clrc_results$ gap4 C227-11.0

And while it's difficult to judge an assembly only from a screenshot, I made one: 22 contigs, none smaller than 4kb and pretty good certainty your bases are correct. Pray tell, isn't that beautiful?

###### Result of the assembly of E. coli C227-11 with error corrected CLR reads.

### CCS reads for E. coli C227-11

Head over to PacBio DevNet and fetch the data set for the E. coli C227-11 CCS data set.

For the rest ... well, it's pretty much the same as for the CLR data set. Just one little difference: in the `.tgz` you downloaded, PacBio has split the data set into multiple FASTQ files (for whatever reason). You will need to concatenate them into one file before starting to work with that. Yep, and that's it.

## Hybrid assemblies: PacBio plus Sanger/454/Solexa/IonTorrent

TO BE EXPANDED: no real walkthrough yest, just a few hints.

- Prepare your PacBio data like explained in this guide.

- Prepare your other data (Sanger, 454, Solexa, IonTorrent or any combination of it) like explained in the respective MIRA guides.

- Write a manifest which includes all data files as readgroups and start MIRA.

## Known bugs in 3.4.0 with PacBio data

For error-corrected CLR data, MIRA does not get the average coverage of a project correct: it underestimates it, sometimes by a factor of 10. This in turn leads to too many "large" contigs and subsequently to a N50 number which is way off the thruth.

Will be fixed asap.