Repository navigation
Keep annotations and gene lookups compact: memory stops growing megabytes per genome - #21
Merged
Merged
Conversation
…ing megabytes per genome Each genome's GFF was held as a String per name in up to six maps, and its gene-to-cluster lookup as a map of every name, copied again for the corrected matrix: 3.4 MB per genome with minimal GFFs and 6.2 MB with PGAP-style ones. Names now live once per genome in one buffer with a sorted hash table, the lookup is one cluster per name corrected in place, matrix cells are one string per cluster, and the block and context tables are built in chunks bounded by genome-cluster pairs, the contexts now in parallel. On 800 synthetic genomes peak memory goes from 2.8 to 0.56 GB (5.5 to 1.0 GB with PGAP-style GFFs) and run time from 33 to 22 seconds, with every output byte for byte the same.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Memory grew by 3.4 MB per genome with minimal GFFs and 6.2 MB with PGAP-style ones (six names per CDS), so 5,000 genomes would have needed about 17 and 32 GB. Measured at 200 genomes with PGAP-style GFFs, almost all of it was annotation: label maps 428 MB, the ID index 228 MB, alias maps 224 MB, the gene-to-cluster maps 228 MB (held twice once the corrected copy was made) and the genes 69 MB, every name a String of its own in up to six places.
What changed
GffIndexkeeps each name once, back to back in one buffer per genome, found through a sorted table of 32-bit hashes; a CDS is a 32-byte record. Products and notes are read once, when the GFF is loaded, to recognise mobile elements, and only the few mobile genes keep their labels.GffGeneis now a view into the index andGffRecordthe owned input.GeneClustersreplaces the per-genome map of every name to its cluster. The map is built for a moment, in parallel, read into one cluster per name of the index (4 bytes) and dropped. The corrections the contexts and blocks need are applied in place, after the multistate matrix (the last reader of the input matrix's clusters), instead of in a copy of every map.cluster_synteny_blocks.tsvis built in chunks bounded by genome-cluster pairs instead of 256 clusters, with rows formatted straight into bytes. A block is a genome's neighbourhood, so where annotations share no gene IDs every genome is a block of its own, and a 256-cluster chunk held 200 MB at 400 genomes, more with every genome.cluster_locus_context.tsvis built in parallel with the same chunks; in one thread it was a sixth of a run at 400 genomes.Measured (one uncached run each, 8 threads, synthetic MTBC-sized genomes: 4.9 Mb, 4,000 genes, 80 gaps per genome)
Memory per genome goes from 3.4 to 0.45 MB with minimal GFFs and from 6.2 to 1.0 MB with PGAP-style ones, so 5,000 such genomes would need about 2.5 and 5.5 GB instead of 17 and 32.
benchmarks/search_scaling.pygains--rich-gfffor the PGAP-style annotations, and both metric tables are regenerated;docs/how-it-works/limits.mdhas the new numbers.Checked by running it
-D warnings, all tests with minimap2 required,mkdocs build --strict, and a check with Rust 1.85 (the MSRV).