Repository navigation
8. Evaluation package: train, score, evaluate, ablate - #8
Merged
Merged
Conversation
The training and evaluation code behind the project's results lived in a private analysis repository, mixed with cohort-specific paths and column names. This ports the reusable part into evaluation/, with every path and column name a setting: - config.py: Columns (names of the id, date, visit, time, event and optional categorical columns, loadable from JSON) and Paths (data root, runs root, output dir, file-name templates), each defaulting to an environment variable. - data.py: the three views a per-visit table is scored under -- one sample per patient (last visit), one per visit, one per fixed landmark day -- and build_landmark_table, which adds forward-filled snapshot rows for patients still at risk on each landmark day. - model.py: build / train / save / load a split's model with its full architecture and objective in metadata.json, and predictions conditioned on the landmark, F_k(s + h | T > s). - metrics.py: Aalen-Johansen incidence, Graf IPCW weights, the IPCW cumulative/dynamic AUC with a competing event (cases by h weighted by 1/G(T-), controls event-free after h), Brier/IPA, calibration slope and intercept, top-decile O:E, patient bootstrap, Wilcoxon, BH. The functions are ports, not rewrites: on the original cohort they reproduce that repository's saved predictions to within 1e-16 and its metric tables to within floating-point rounding. tests/test_evaluation_metrics.py checks the metrics on hand-computed cases, Aalen-Johansen against scikit-survival, and the AUC against R timeROC (identical apart from timeROC excluding events that fall exactly on day h). The slope fit returns NaN instead of failing when every prediction is equal (statsmodels drops a constant regressor), which a coarse comparator score can produce.
evaluation.train trains one configuration on every split and writes, per split, the checkpoint, metadata and three prediction files: each patient at their last visit, at every visit, and at every fixed landmark day they are still at risk. Its flags mirror the library's: --all-landmark-nll, --hazard/--proportional, --landmark-row-weighting, --gamma and the rest, with defaults that reproduce the original loss. evaluation.apply_external loads each split's model and scores an external cohort (for a temporal validation: every visit up to each patient's anchor) at the anchor and at every retained visit. Retraining the reported configuration on one split of the original cohort with this script gives identical weights and predictions to the original run.
evaluation.evaluate scores several arms against a reference arm, raw (no recalibration), on three populations: - landmarks: internal test patients still at risk at days 0/90/180/270; per-split metrics, mean and SD, and each arm's per-split difference from the reference (Wilcoxon signed-rank, BH within horizon and metric); - visits: internal test patients at every visit, by days since the first visit; - external: the external cohort at the anchor, split models averaged into one ensemble, overall and by history; patient bootstrap for each arm and for its paired difference from the reference. Arms are scored on identical rows: a row any arm cannot score is dropped for all of them. That is also how an existing clinical score is compared: its risks are written in the same file layout and passed as one more arm. On the original cohort this reproduces the project's raw-calibration tables row for row, including the bootstrap intervals and p-values.
Whether the model uses earlier visits cannot be read off a comparison of
patients with long and short records: they are different patients.
evaluation.history_ablation re-scores the SAME patients with the SAME
models on a changed record:
full the record up to the anchor (must equal the saved predictions)
masked the anchor visit alone, landmark kept at the anchor day
reanchored the anchor visit alone, landmark 0, gap column reset as for a
first visit (with --gap-col)
and reports each variant's metrics and the paired bootstrap contrasts,
plus how far the masked record moves the ranking (Spearman, share of
patients changing risk decile).
On the original cohort its predictions for all three variants are identical
to the project's own ablation.
evaluation.make_synthetic writes a cohort in the expected layout, with no real patient involved, so the workflow can be run, read and tested without access to the data it was built for. It keeps the two properties that matter here: visits stop at the event, so patients with an early event have few visits (what biases 1/V_i weighting), and a lab value rises before the event of interest (what a dynamic model can use). tests/test_evaluation_pipeline.py runs make_synthetic -> train (two arms) -> apply_external -> evaluate -> history_ablation on a tiny cohort, and CI now installs the eval extra so it runs on every push.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The training and evaluation protocol, rebuilt from the original analysis code with every path and column name a setting (
evaluation/config.py, environment variables or flags).train: one configuration on every split; checkpoints, full metadata, and predictions at the last visit, every visit, and every fixed landmark.apply_external: scores an external or temporal cohort with each split's model.evaluate: raw discrimination and calibration on fixed landmarks, every visit and the external cohort, on identical rows for every arm. Paired per-split tests or a patient bootstrap against a reference arm. An existing clinical score can be compared as one more arm.history_ablation: re-scores the same patients with earlier visits removed, to test whether the model uses history at all.make_synthetic: a synthetic cohort with informative visit timing, so everything runs, and is tested in CI, with no patient data.Checked. On the original cohort, which is not included here, these scripts reproduce the original analysis exactly: