[Paururo]

Two items from the review of what Recore still lacks: fitting into what comes after it, and iterating at scale.

Downstream

• gene_presence_absence_recore.Rtab: the corrected matrix as 1/0 in Panaroo's Rtab layout, written in the same pass as the CSV, so they cannot disagree. This is what pyseer and most R code read.
• --write-rescued-fasta: rescued_genes.fna and .faa hold every cell the matrix gained, read in the gene's direction and translated with table 11 from the phase the alignment fixes, under the corrected GFF's feature IDs. The proteins match an independent translation, and the minus strand, phase and unknown frame have tests.
• Input from PPanGGOLiN, PanTA and PIRATE, each run for real (2.3.2, 1.0.1, 1.0.5) on the tutorial data:
  • PPanGGOLiN's matrix.csv and family FASTA already worked.
  • PanTA's cells are sample-contig-gene, so no cell matched a GFF: synteny validated nothing and no rescue reached the matrix. They match now (0 to 2 corrections, the two planted).
  • PIRATE's own PIRATE.gene_families.tsv is read into Panaroo's layout, its representative headers (family;key=...) are resolved, and its fission notation (a:b) (taken from its link_clusters.pl) is understood. PIRATE_to_roary.pl output names rows allele__family and cannot be used.
  • On all three, Recore rescues exactly the planted missing CDSs; on PIRATE it also finds the planted pseudogenes and keeps them out.

Scale

• --search-cache DIR keeps minimap2's final hits per genome. They are keyed by the genome's bytes, the query sequence's hash, the settings that change a hit (preset, base-level alignment, secondary budget, --min-identity) and the minimap2 and recore versions.
  • Another policy, coverage, synteny or prevalence setting aligns only what is new.
  • Hits are handed on in the order an uncached search gives them, so outputs are byte-identical.
  • A cut-short or foreign cache file is searched again.
• Measuring showed the search is the smaller part of a large run (10 of 48 seconds at 800 genomes). So:
  • cluster_synteny_blocks.tsv, which held every cluster's blocks in memory in one thread, is built in parallel chunks and streamed, byte for byte the same file.
  • Synteny validation runs per cluster in parallel; a rescue reads only its own cluster's reference contexts.
  • GFF loading and the internal-stop scan run per genome in parallel.
• benchmarks/search_scaling.py and its table: MTBC-sized synthetic genomes (4.9 Mb, 4,000 genes, 80 gaps each), 8 threads:

Genomes	main	this branch	warm cache	peak memory, main → branch
100	7.4 s	3.4 s	2.4 s	1.6 → 0.5 GB
200	16.6 s	7.1 s	6.1 s	3.3 → 0.8 GB
400	36.4 s	18.0 s	13.0 s	4.7 → 1.5 GB
800		48.1 s	38.1 s	2.6 GB

Checked by running it

• fmt, clippy -D warnings, 196 library tests, 4 integration tests with minimap2 required, mkdocs build --strict.
• Every new test was broken on purpose first and failed.
• Every output compared with main's, file by file, on the tutorial and on 200 synthetic genomes: identical, except the new .Rtab.
• Tutorial: Rescued 11 and Matrix-corrected 8 unchanged.