Trinity Performance - Memory Usage

Memory Usage Over Time

The Trinity pipeline consists of a series of executables, many of which run simultaneously. The charts below show the amount of memory consumed by the individual processes over the course of a test run. When running on a single node, the number of processes that can be run at the same time may be limited by the amount of available RAM. This is particularly noticeable when running the final Trinity phase, Butterfly.

Butterfly

The final phase of a Trinity run consists of running multiple instances of a Java application called Butterfly. It is easy to set parameters such that Butterfly requires a huge amount of memory. By default, Trinity runs two simultaneous instances, each one requesting 10GB of memory. Thus, the system must have 20 GB available to it. It is common, however, to set the --CPU to a much larger value.

The --CPU parameter specifies the number of CPU's available. Trinity passes this value on to the Bowtie application to increase its running efficiency; the value also is used to determine the number of Butterfly instances that can be run simultaneously. However, the system must also have sufficient RAM to run this number of instances. If 32 copies of Butterfly run at the same time, 320 GB of memory must be available.

However, some systems do not allocate all requested memory until required. In this case, many instances of Butterfly may use just one or five GB of memory and more instances may be run at once. But the safest option is to make sure that available memory is sufficient to hold 10GB per instance.

Trinity makes various options available to minimize this issue. The --bflyCalculateCPU option is the simplest, allowing Trinity to use 80% of available memory to run 10GB Butterfly instances, so a 200GB system would run 16 instances. For more precise control, the --bflyCPU option is available. Combining this option with the CPU option allows both Bowtie and Butterfly to run at maximum efficiency while keeping memory pressure under control.

In addition, the --bflyHeapSpaceMax is available. If you are confident that no instances of Butterfly will use all 10GB of memory, setting this to a smaller value may allow more Butterfly processes to run.

The flag --normalize_reads allows the use of in silico normalization. The normalization will be more memory intensive because it uses Jellyfish to count the kmers, but it may reduce the memory footprint required later. (However, also note that at this time quality trimming and the in silico normalization are currently not compatible). Normalization requires that the reads in the left.fq file are identically ordered as those in the right.fq file.

The --max_memory parameter gives the largest amount of memory of which Trinity may take advantage. This is mostly seen in sorting. Note that unless the --bflyHeapSpaceMax parameter is given, --max_memory does not affect the number of Phase II sessions that may run.

Drosophila Data Set

Command line: Trinity --seqType fa --max_memory 120G --SS_lib_type RF --bflyCalculateCPU --output trinity_out_Droso_2.1.1 --CPU 16 --monitoring --left reads.left.fa --right reads.right.fa

Mouse Data Set

If there is sufficient memory available, the --bflyCPU parameter may be left off. In this case, Trinity will run simultaneous Butterfly instances up to the --CPU parameter. The Mouse data set was run using 400GB of memory, which allows a time savings of around 25%. Note the Butterfly memory spike over the final 1.5 hours of runtime. The test ran to completion in slightly more than eight hours of wall time.

Command line: Trinity --seqType fq --max_memory 120G --SS_lib_type RF --bflyCalculateCPU --output trinity_out_Mouse_2.1.1 --CPU 16 --monitoring --left reads.left.fq --right reads.right.fq

Pombe Data Set

The S. Pombe test consists of paired-end reads, including quality scores, with a length of 68. This test also completed in slightly more than eight hours.

Command line: Trinity --seqType fq --max_memory 120G --SS_lib_type RF --bflyCalculateCPU --output trinity_out_Schizo_2.1.1 --CPU 16 --monitoring --left reads.left.fq --right reads.right.fq