fix(dataset): decide seq_encoding from the whole corpus, and record it

[jayhesselberth]

Closes #230.

What main does today

I reproduced all three defects before touching anything, and one detail changes the picture from the issue:

corpus	main
no maps at all	dataset falls back, model is still built for 36 channels → RuntimeError: expected input[4, 4, 11] to have 36 channels at the first forward pass
chunk 0 fine, later chunks empty	passes the check, and those chunks encode to all-zero sequence channels — silent, per chunk
chunk 0 empty, rest fine	whole corpus flips to base_onehot → same channel-count crash

The silent-train from 0.6.7 is now a confusing crash instead — except for the ragged corpus, which is still silently wrong, and except for sequence-blind arms, which train fine on the wrong input. The middle state is the worst of the three: the dataset half falls back and the model half does not, so the run dies several steps later on a message naming neither the corpus nor the encoding.

Changes

The decision is a whole-corpus count, never chunk 0. _signal_kmer_coverage is one vectorised pass over the CSR offsets and the context column on the streaming path — data already in memory — and the message names the affected fraction. A warning that does not say how much of the corpus it covers is the one that got read past for four releases.

A corpus carries the maps for every chunk or for none, and partial coverage raises. None is the version-skew case the fallback exists for. Anything in between is damage, where both available answers are the same silent representation change: encoding the uncovered rows gives them the all-zero channels above, and switching a 6.7M-chunk corpus to base_onehot on the strength of a few bad rows discards the encoding for every good one. data merge already refuses to merge a mapped corpus with an unmapped one (_validate_member_sets folds the three CSR spellings into one logical member), so ragged means a damaged file rather than skew. The escape hatch is --seq-encoding base_onehot, which is explicit.

An encoding named on the command line is not substituted. --seq-encoding signal_kmer over a corpus that cannot supply it stops the run; taking the default still falls back, which is the case the fallback exists for. click's get_parameter_source separates the two, and --encoding-fallback / --no-encoding-fallback overrides either way (LeechDataset(allow_encoding_fallback=...) in code, default True, so nothing outside training changes behaviour).

The model and config.json both come from LeechDataset.effective_seq_encoding. That is the issue's suggested minimum, and it is what makes a fallen-back arm auditable after the fact — one field, where before there was no way to find out at all. It also keeps #217's ONNX contract honest: the contract is derived from the same config and exists precisely so a non-Python consumer can trust the input spec.

This half has to land with the policy fix rather than without it. On its own it would make the whole-corpus fallback train coherently to completion — restoring exactly the silent success the issue is about.

Three other sites where a fallback can never work, because the trained model's sequence branch has a fixed channel count and any fallback guarantees a shape error several steps later: eval test and both calibration paths pass allow_encoding_fallback=False; model benchmark builds its model from the effective encoding.

Tests

16 new tests, all failing on main (run against the main checkout's src to confirm) except the two healthy-corpus controls:

• test_signal_kmer.py::TestEncodingFallbackPolicy — ragged stops the run in both flag positions; one empty row is reported as one row, not as the whole corpus; a ragged corpus still reads as base_onehot when asked for by name; a corpus with no maps at all falls back and the warning carries the fraction; the CSR count and the per-dict count agree.
• test_config_propagation.py::TestSeqEncodingProvenance — the config records the effective encoding; the checkpoint's weights load into a model rebuilt from the saved config alone; explicit --seq-encoding errors on the CLI while the default does not; --encoding-fallback overrides.
• test_onnx_export.py::test_a_config_that_misstates_its_encoding_cannot_be_exported — the issue's open question ("I have not tested whether that fails loudly at export"). It does: refused at load_state_dict with a size mismatch, not published as a 36-channel contract. Unreachable from leech model train now, but it keeps a hand-edited or older config from being trusted.

Full suite: 1510 passed, 44 skipped (3:01, in a Slurm allocation). ruff format / ruff check clean. ty check src/leech/ reports the same 2 pre-existing diagnostics as main (crf/evaluate.py, signal_refine.py), neither in a touched file.

CHANGELOG, CLAUDE.md (new "The sequence encoding is decided once, from the whole corpus, and recorded" section) and docs/reference/cli.md updated.

🤖 Generated with Claude Code