Skip to content

Keep annotations and gene lookups compact: memory stops growing megabytes per genome - #21

Merged
Paururo merged 1 commit into
mainfrom
perf/compact-annotations
Oct 17, 2026
Merged

Paururo merged 1 commit into
mainfrom
perf/compact-annotations

Conversation

@Paururo

@Paururo Paururo commented Oct 17, 2026

Copy link
Copy Markdown
Member

Memory grew by 3.4 MB per genome with minimal GFFs and 6.2 MB with PGAP-style ones (six names per CDS), so 5,000 genomes would have needed about 17 and 32 GB. Measured at 200 genomes with PGAP-style GFFs, almost all of it was annotation: label maps 428 MB, the ID index 228 MB, alias maps 224 MB, the gene-to-cluster maps 228 MB (held twice once the corrected copy was made) and the genes 69 MB, every name a String of its own in up to six places.

What changed

  • GffIndex keeps each name once, back to back in one buffer per genome, found through a sorted table of 32-bit hashes; a CDS is a 32-byte record. Products and notes are read once, when the GFF is loaded, to recognise mobile elements, and only the few mobile genes keep their labels. GffGene is now a view into the index and GffRecord the owned input.
  • GeneClusters replaces the per-genome map of every name to its cluster. The map is built for a moment, in parallel, read into one cluster per name of the index (4 bytes) and dropped. The corrections the contexts and blocks need are applied in place, after the multistate matrix (the last reader of the input matrix's clusters), instead of in a copy of every map.
  • Matrix cells are one string per cluster instead of a String per cell.
  • cluster_synteny_blocks.tsv is built in chunks bounded by genome-cluster pairs instead of 256 clusters, with rows formatted straight into bytes. A block is a genome's neighbourhood, so where annotations share no gene IDs every genome is a block of its own, and a 256-cluster chunk held 200 MB at 400 genomes, more with every genome.
  • cluster_locus_context.tsv is built in parallel with the same chunks; in one thread it was a sixth of a run at 400 genomes.
  • One behaviour fix: when two corrections in one genome named the same gene ID, the cluster their neighbours took in the context and block tables depended on hash-map order and could change from run to run. The first cluster in matrix order now wins.

Measured (one uncached run each, 8 threads, synthetic MTBC-sized genomes: 4.9 Mb, 4,000 genes, 80 gaps per genome)

Genomes GFFs main this branch
100 minimal 3.6 s, 448 MB 2.5 s, 238 MB
100 PGAP-style 3.1 s, 1,114 MB 2.2 s, 330 MB
200 minimal 7.1 s, 798 MB 4.6 s, 292 MB
200 PGAP-style 6.5 s, 1,951 MB 4.3 s, 436 MB
400 minimal 14.8 s, 1,460 MB 9.5 s, 367 MB
400 PGAP-style 14.9 s, 3,716 MB 9.7 s, 637 MB
800 minimal 32.9 s, 2,804 MB 21.8 s, 550 MB
800 PGAP-style 35.0 s, 5,467 MB 21.1 s, 1,028 MB

Memory per genome goes from 3.4 to 0.45 MB with minimal GFFs and from 6.2 to 1.0 MB with PGAP-style ones, so 5,000 such genomes would need about 2.5 and 5.5 GB instead of 17 and 32.

benchmarks/search_scaling.py gains --rich-gff for the PGAP-style annotations, and both metric tables are regenerated; docs/how-it-works/limits.md has the new numbers.

Checked by running it

  • After every step, all outputs byte for byte identical to main: the tutorial (20 files, with the reference audit and corrected GFFs) and 200 synthetic genomes with minimal and with PGAP-style GFFs (15 files each).
  • A model test keeps the index as it was on main and compares it with the compact one on 400 random annotations with duplicate gene IDs, aliases that are another gene's ID, genes without an ID, nested genes and repeated corrections: every lookup, alias list, mobile label, cluster, overlap, flank and context window agrees. Broken four ways on purpose (alias claims, alias owner, corrections overwriting, no alias fallback), it failed each time.
  • A size guard (under 250 bytes per CDS with six names) and a parser test that reads every cell back, also broken on purpose first.
  • fmt, clippy -D warnings, all tests with minimap2 required, mkdocs build --strict, and a check with Rust 1.85 (the MSRV).

…ing megabytes per genome

Each genome's GFF was held as a String per name in up to six maps, and its gene-to-cluster lookup as a map of every name, copied again for the corrected matrix: 3.4 MB per genome with minimal GFFs and 6.2 MB with PGAP-style ones. Names now live once per genome in one buffer with a sorted hash table, the lookup is one cluster per name corrected in place, matrix cells are one string per cluster, and the block and context tables are built in chunks bounded by genome-cluster pairs, the contexts now in parallel. On 800 synthetic genomes peak memory goes from 2.8 to 0.56 GB (5.5 to 1.0 GB with PGAP-style GFFs) and run time from 33 to 22 seconds, with every output byte for byte the same.
@Paururo Paururo added documentation Improvements or additions to documentation rescue Rescue classification and the minimap2 search behind it matrix The presence/absence matrix: parsing, clustering, relationships pipeline Run orchestration, evolution calls, the quality guard, reference audit report The HTML run report and what the writer emits tests Test suites and fixtures benchmarks The Python benchmark harness and its checked-in metrics labels Oct 17, 2026
@Paururo
Paururo merged commit 3df291b into main Oct 17, 2026
8 checks passed
@Paururo
Paururo deleted the perf/compact-annotations branch October 18, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

benchmarks The Python benchmark harness and its checked-in metrics documentation Improvements or additions to documentation matrix The presence/absence matrix: parsing, clustering, relationships pipeline Run orchestration, evolution calls, the quality guard, reference audit report The HTML run report and what the writer emits rescue Rescue classification and the minimap2 search behind it tests Test suites and fixtures

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant