From a84d22996ebce9768f9f98450e88b8b1235aa627 Mon Sep 17 00:00:00 2001 From: dadudu13 Date: Mon, 5 Oct 2026 14:50:17 -0500 Subject: [PATCH 1/2] Document every modification, and the experiment that was not adopted docs/modifications.md walks through the changes in the order they were made: what was wrong or missing, what changed, which argument switches it and what its default does, and how it was checked. For the landmark weighting it also records how the weighting was chosen, with a synthetic reproduction of the first-visit effect that anyone can rerun. It ends with the one experiment kept on an experiments/* branch and why it was not merged. LICENSE keeps the original MIT notice and adds a copyright line for the modifications. --- LICENSE | 1 + docs/modifications.md | 212 ++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 213 insertions(+) create mode 100644 docs/modifications.md diff --git a/LICENSE b/LICENSE index aa5fbb3..de9f517 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,7 @@ MIT License Copyright (c) 2026 Vincent Jeanselme +Copyright (c) 2026 Dulin Wang (modifications listed in docs/modifications.md) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/docs/modifications.md b/docs/modifications.md new file mode 100644 index 0000000..cfbb043 --- /dev/null +++ b/docs/modifications.md @@ -0,0 +1,212 @@ +# Modifications to Dynamic-DeepHit + +This fork starts from [Jeanselme/DynamicDeepHit](https://github.com/Jeanselme/DynamicDeepHit) +(PyTorch port of Lee et al., *Dynamic-DeepHit*, IEEE TBME 2020) and adapts it to +**landmark prediction from irregular clinic visits with a competing risk**: at any visit, +predict the risk of the event of interest within *h* days, given everything seen so far +and given that the patient is still event-free. + +Every change below is one commit, and every commit in the history passes the test +suite that exists at that point. The changes come in two groups: + +* **Foundation** (sections 1–2): fixes and infrastructure. Several of these change the + model's default behaviour on purpose, because the upstream behaviour was a bug. +* **Modifications** (sections 3–7): new training objectives and output layers. Each + one is behind an argument whose default leaves the model exactly as it was. This + was checked at every commit by training a small model with default arguments, with + and without landmarks, and comparing the predictions, which stayed bit-identical + across all thirteen commits. + +Notation: patient *i* has visits at days *si1 < … < siVi* +(day 0 = first visit). Cause 1 is the event of interest and cause 2 the competing +event. *Fk(t)* is the cumulative incidence of cause *k*, and +*S(t) = 1 − Σk Fk(t)*. + +--- + +## 1. Foundation: making the port correct and reproducible + +| Change | Why | Effect on defaults | +|---|---|---| +| Import the DSM helpers from **auton-survival** through `ddh/dsm_compat.py` | The `DeepSurvivalMachines` package the code imported from was renamed and repackaged; its old import paths no longer resolve | none | +| **Clip discretised times** to the bins training can reach | A time equal to the first bin edge got index −1, which torch reads as the *last* bin, so the earliest events were scored as the latest | yes (bug fix) | +| **Training loop**: reshuffle each epoch, validate on every row, ReduceLROnPlateau, gradient clipping, configurable patience | Validation counted batches with one batch size and sliced with another, so early stopping watched only a fixed fraction of the validation set | yes | +| **Seed every RNG** in `fit` | `random_state` reached only the train/validation shuffle; weight initialisation and batch order were unseeded | yes | +| **Restore RNN bias terms** | With `bias=False` no gate can learn an offset, and an all-mean input from a zero state is stuck at a zero hidden state | yes | +| Optional **learned embedding** for one integer-coded categorical column (`cancer_vocab_size`, `cancer_embedding_dim`, `cancer_id_col`) | An integer-coded category fed as a number is treated as ordinal | none (off unless all three are set) | +| **σ inside the exponential** of the ranking loss | Upstream computed exp(Δ)/σ instead of the paper's exp(Δ/σ), so σ only rescaled the term | yes (bug fix) | +| Ranking loss **summed over pairs**, with **per-cause weights** αk (`cause_weights`) | The paper sums over comparable pairs with a weight per cause; the code averaged and fixed αk = 1/K | yes | + +## 2. Foundation: conditioning on the landmark + +A prediction made at day *s* must condition on survival to *s*: + + P(T = t, K = k | T > s, history up to s) = P(T = t, K = k | history) / S(s) + +`fit(s=...)` takes one landmark per patient. The likelihood subtracts log *S(si)*. The +ranking term compares patients on remaining time *t − s* and scores each comparator's +conditional CIF *ri* days after its own landmark. With `s=None` every function takes +its original path. + +A self-review of this change found six problems, each fixed in its own commit: + +1. Landmarks were clipped to a wider bin range than event times. +2. `compute_nll` ignored landmarks. +3. The embedded categorical column was still a reconstruction target. +4. The conditioning term's survival floor (1e-10) allowed gradients of order 1e10; it is now 1e-3. +5. Validation loss depended on the batch size; the whole validation set is now scored in one pass. +6. auton-survival was found only through one hard-coded relative path; it can now come from an environment variable, a nearby checkout, or an installed package. + +--- + +## 3. Train the likelihood at every landmark (`all_landmark_nll=True`) + +**Problem.** The model was trained with one row per patient: full history, +conditioned on the last visit. Evaluation, though, truncates *every* patient to each +landmark. Short-history inputs were therefore seen in training almost only for +patients whose follow-up really was short, a group with early events. So the model +learned "short history → high risk", and over-predicted at early landmarks. + +**Change.** Every (patient, visit) prefix contributes a conditional likelihood term: + + L = (1/K) Σi Σv wiv · −log P(Ti, Ki | Ti > siv, Xi,≤v), with wiv = 1/Vi by default + +The expanded rows are back-propagated in chunks (`landmark_chunk_rows`). This is exact, +since the objective is a weighted sum. Validation uses the same objective. With +*Vi = 1* the objective reduces to the previous loss, which the tests assert. + +Before this, a cheaper variant was tried and kept: `landmark_augment_prob` truncates a +random subset of patients to a random earlier visit each epoch. The deterministic +objective above covers every landmark instead of a sample. + +Also in this group: a batch in which no patient has a second visit no longer makes +the reconstruction loss NaN. + +## 4. Discrete-time cause-specific hazard head (`hazard=True`, `proportional=True`) + +**Problem.** The DeepHit output is one softmax over all (cause, bin) cells, so the +total event probability inside the time grid is forced to 1. Patients who reach the +end of follow-up event-free have nowhere to go, and the model piles probability into +the last bin. + +**Change.** `hazard=True` emits per-bin competing-risks hazards: K cause logits plus a +"survive this bin" logit, softmaxed within each bin. These are converted back to the +same (cause, bin) probability interface, so every downstream consumer is unchanged, +but P(no event by the last bin) is now representable. The survive logit's bias is +initialised in closed form so initial survival is not ≈ 0 on long grids. +`proportional=True` additionally factorises logit λk(b) = αk(b) + βk·h: a per-bin +baseline plus one shared patient score. This makes the risk ordering identical in +every bin. The tests assert that, and also assert it does *not* hold for the free head. + +## 5. Re-weighting the all-landmark objective over time (`landmark_decay_days`, `entry_weight`) + +* `landmark_decay_days=T` weights each landmark by exp(−day/T), renormalised within + each patient. It is keyed on **day**, not visit index: visit index mixes up clinical + timing with how often a patient is seen. +* `entry_weight` scales a patient's *total* contribution. It is applied outside the + per-patient normalisation, because a per-patient constant folded inside would divide + straight back out. A test pins that placement. + +## 6. Horizon-specific likelihood (`gamma`, `horizon_days`, `horizon_ipcw_clip`) + +**Problem.** The bin-level likelihood scores the bin an event fell in. Nothing in it +constrains the *cumulative* risk at the horizons the model is used at. The model +over-predicted the 90-day level even in sample, so this was the objective not asking, +rather than a failure to generalise. + +**Change.** For every landmark row and every horizon *h*, add the IPCW-weighted binary +log-likelihood of the observed status at *s + h* under +p = (F1(s+h) − F1(s)) / S(s). + +* Labels and weights follow the evaluation exactly (Graf et al.): a competing event + before *h* is a true 0, and a row censored before *h* carries no weight. +* The censoring weights are conditioned on the landmark (G(·)/G(s)). Without this, + the term inflated late landmarks. +* `gamma` is added on top of α and β, so `gamma=0` is bit-identical to the previous + objective. + +## 7. Who counts how much: landmark row weighting (`landmark_row_weighting`) + +**Problem.** Under 1/Vi weighting every patient contributes a total weight of 1, spread +over their visits. But visits stop at the event, so patients with an early event have +few visits, and each of their early rows carries a large weight. On the development +cohort this put the first-visit rows' 90-day event rate well above the true rate, while a +static model trained on first visits only, one row per patient, did not over-predict. + +**Change.** Three alternatives to `"patient"` (1/Vi): + +* `"visit"`: every row weighs the same. +* `"given"`: per-row weights from outside the model (`row_weights`), e.g. inverse-intensity weights from a model of the visit process. +* `"band"`: each patient counts **once per landmark-day band** (`landmark_band_edges`). This is how the model is evaluated, with each patient still at risk scored once per landmark. A single band reproduces 1/Vi, and bands holding one visit each reproduce equal rows; the tests assert both limits. + +All three keep the batch's total weight equal to the per-patient scheme's. Only the +distribution changes, which matters because gradients are clipped to a fixed norm. + +### How the weighting was chosen + +The weighting was settled by a sequence of single-change comparisons on the development +cohort, every arm scored raw (no recalibration): + +1. **1/Vi** (section 3) fixed the last-visit-only training, but over-predicted at + the first visits. +2. **Decay by landmark day** (section 5) was the deployed arm for a while, and needed a + post-hoc recalibration to be calibrated. +3. **A static model** trained on first visits only, one row per patient, with the same + output layer, was calibrated at the first visit. So the over-prediction came from the + weighting, not from the hazard head. +4. **Equal weight per visit** removed the early over-prediction, as a one-factor change. +5. **Inverse-intensity weights** from a model of the visit process landed on the per-visit + result: modelling the visit process said what the simple rule already assumed. +6. **Band weighting** weights rows the way the model is evaluated, each patient still at + risk counted once per landmark. It was the best calibrated across landmarks without any + recalibration, and replaced decay plus recalibration as the reported configuration. + +No weighting changed discrimination; the choice is about calibration. An under-prediction +after the first year was a separate problem that none of the weightings addressed. + +The synthetic cohort (`evaluation/make_synthetic.py`) is built with the property behind +step 1: visits stop at the event, so patients with an early event have few visits. It +reproduces the first-visit effect with the published code: + +| arm, 90-day risk | O:E at the first visit | O:E at day 90 | AUC at the first visit | +|---|---|---|---| +| 1/Vi (`patient`) | 0.53 | 0.69 | 0.82 | +| decay, T = 180 d | 0.85 | 0.98 | 0.81 | +| equal per visit (`visit`) | 1.35 | 1.10 | 0.79 | +| band (`band`) | 1.31 | 1.03 | 0.80 | +| static, first visit only | 1.21 | 1.11 | 0.82 | + +O:E is observed (Aalen-Johansen) over mean predicted risk; below 1 is over-prediction. +1/Vi over-predicts about twofold at the first visit; giving each patient equal +weight there removes it (on this small cohort the correction overshoots a little), and AUC +barely moves. The simulation is not built to mimic the real cohort beyond the first visits, +where every arm over-predicts, so read it for the first-visit contrast only. Means over three +splits, with the proportional hazard head (see [`evaluation/`](../evaluation/README.md)); +reproduce with: + +```sh +python -m evaluation.make_synthetic --out $DDH_DATA_ROOT +export DDH_COLUMNS=$DDH_DATA_ROOT/columns.json +COMMON="--splits 0 1 2 --iters 40 --alpha 0 --hazard --proportional" +python -m evaluation.train --arm patient $COMMON --all-landmark-nll +python -m evaluation.train --arm visit $COMMON --all-landmark-nll --landmark-row-weighting visit +python -m evaluation.train --arm band $COMMON --all-landmark-nll --landmark-row-weighting band +python -m evaluation.train --arm decay $COMMON --all-landmark-nll --landmark-decay-days 180 +python -m evaluation.train --arm static $COMMON --first-visit-only +python -m evaluation.evaluate --arm band --arm "1/V_i=patient" --arm "per visit=visit" \ + --arm decay --arm "static=static" --splits 0 1 2 --stage landmarks +``` + +**Configuration the project reports:** `hazard=True`, `all_landmark_nll=True`, +`alpha=0` (no ranking term), `landmark_row_weighting="band"`. See +[`evaluation/`](../evaluation/README.md) for how it is trained and evaluated. + +--- + +## Tried and not adopted + +One change was tried and not merged; it is kept on its own branch. + +| Branch | Idea | Why it was not adopted | +|---|---|---| +| `experiments/all-landmark-rank` | Move the ranking term onto the same (patient, visit) rows as the all-landmark likelihood, pairs weighted *wiwj*, within-patient pairs dropped | A three-split pilot against the all-landmark likelihood alone showed no discrimination gain (the interval included zero and was narrow next to the gain the all-landmark likelihood itself had bought). Calibration moved slightly the wrong way, and each epoch took 50% longer. Stopped after the pilot. | From 31ba9a68afaf05b7622a4515e9d6175c22be5a12 Mon Sep 17 00:00:00 2001 From: dadudu13 Date: Mon, 5 Oct 2026 14:50:17 -0500 Subject: [PATCH 2/2] Rewrite the README around what this fork changes and how to run it The README now says what the fork is for (landmark prediction from irregular visits with a competing risk), lists the foundation fixes and the opt-in modifications with their switches, gives the install recipe CI uses and a minimal API example, and explains how the history is organised. evaluation/README.md documents the data layout and column mapping, the train -> apply -> evaluate -> ablation workflow on the synthetic cohort, the exact metric definitions, how to compare against an existing score, and which outputs are patient-level and must stay with the data. --- README.md | 91 +++++++++++++++++++++++++--- evaluation/README.md | 140 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 222 insertions(+), 9 deletions(-) create mode 100644 evaluation/README.md diff --git a/README.md b/README.md index 1c0b628..878cd46 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,88 @@ -# DynamicDeepHit -This repository is a Pytorch implementation of the paper : [Dynamic Deep Hit](https://ieeexplore.ieee.org/document/8681104). Original implementation in Tensorflow can be found at [chl8856/Dynamic-DeepHit](https://github.com/chl8856/Dynamic-DeepHit). +# Dynamic-DeepHit for landmark prediction +A fork of [Jeanselme/DynamicDeepHit](https://github.com/Jeanselme/DynamicDeepHit), the PyTorch +implementation of [Dynamic-DeepHit](https://ieeexplore.ieee.org/document/8681104) (Lee et al., +IEEE TBME 2020; original TensorFlow code at +[chl8856/Dynamic-DeepHit](https://github.com/chl8856/Dynamic-DeepHit)). -The code is based on the [Deep Survival Machines repository](https://github.com/autonlab/DeepSurvivalMachines) and use a similar module structure. +The fork adapts the model to **landmark prediction from irregular clinic visits with a +competing risk**. At any visit, it predicts the risk of the event of interest within +*h* days, given everything seen so far and given that the patient is still event-free. +## What changed -### Code Description -All the code related to Dynamic Deep Hit is loacted in `ddh`. Only API should be used for application. +Each row is one or a few commits; [`docs/modifications.md`](docs/modifications.md) gives +the problem, the change and how it was tested. -An example is available in the `examples` folder. +**Foundation.** These change upstream defaults where the upstream behaviour was a bug. -## TODOs -- Full documentation -- Further testing to ensure the model is reproducing the original paper's performance +| change | where | +|---|---| +| Port to auton-survival (the maintained successor of DeepSurvivalMachines) | `ddh/dsm_compat.py` | +| Discretisation index fix: the earliest events were scored in the last bin | `discretize` | +| Training loop: per-epoch shuffle, full validation, LR schedule, gradient clipping, seeding | `utils.train_ddh`, `fit` | +| Ranking loss as in the paper: σ inside the exponential, summed over pairs, per-cause weights | `losses.ranking_loss` | +| Likelihood and ranking conditioned on survival to a landmark, plus six fixes from a self-review | `fit(s=...)`, `losses` | + +**Modifications.** Every one is opt-in; with default arguments, predictions are +bit-identical to the foundation. + +| change | switch | problem it addresses | +|---|---|---| +| Likelihood at every landmark | `all_landmark_nll=True` | trained on last visits only, evaluated at every landmark | +| Discrete-time cause-specific hazard head | `hazard=True`, `proportional=True` | the softmax head must spend all probability inside the time grid | +| Landmark-day decay and between-patient entry weights | `landmark_decay_days`, `entry_weight` | where in follow-up the objective puts its weight | +| Horizon-specific likelihood | `gamma`, `horizon_days` | nothing in the bin-level likelihood constrains cumulative risk at *h* | +| Landmark row weighting | `landmark_row_weighting="visit"/"given"/"band"` | patients with early events have few visits, so 1/Vi weighting over-weights them early | + +The configuration the project reports is `hazard=True, all_landmark_nll=True, alpha=0, +landmark_row_weighting="band"`. One experiment that was tried and not adopted (the ranking +term at every landmark) is kept on the `experiments/all-landmark-rank` branch; +[`docs/modifications.md`](docs/modifications.md#tried-and-not-adopted) says why it was dropped. + +## Install + +```sh +git clone https://github.com/dadudu13/DynamicDeepHit && cd DynamicDeepHit +pip install torch +# auton-survival 0.1.0; --no-deps because its release metadata pins torch<2.0 +pip install --no-deps "auton-survival @ git+https://github.com/autonlab/auton-survival@5dde465f7223601717abddc1d075e837707c403b" +pip install -e ".[test,eval]" +python -m pytest +``` + +Tested locally with Python 3.11 and torch 2.11 and 2.14; CI runs Python 3.11 and 3.12. + +## Use + +```python +import numpy as np +from ddh import DynamicDeepHit + +# x: per-patient arrays (visits x features); t, e: per-visit outcome time (days from the +# first visit) and event (0 censored, 1 event of interest, 2 competing); +# visit_days: per-patient arrays of each visit's day; s: each patient's last visit day. +model = DynamicDeepHit(split=50, hidden_rnn=32, typ="GRU", alpha=0, beta=0.75, hazard=True) +model.fit(x, t, e, iters=50, s=s, visit_days=visit_days, + all_landmark_nll=True, landmark_row_weighting="band", + landmark_band_edges=[0, 30, 90, 180, 365, 730]) +cif_event = 1 - model.predict_survival(x, [90.0, 180.0], risk=1) # unconditional, from day 0 +``` + +To get risks conditioned on a landmark, and the full train / external-cohort / +evaluation workflow, see [`evaluation/`](evaluation/README.md). It runs end to end on +the synthetic cohort it generates. + +## How the history is organised + +* One change per commit, each explaining the problem it solves. Every commit passes + the tests that exist at that point. +* Upstream's history is kept intact underneath. The work is merged through one pull + request per theme: foundation, landmark training, hazard head, landmark-time + weighting, horizon term, row weighting, packaging, evaluation, documentation. +* Experiments that were not adopted stay on their own branches. + +## Credit and license + +The original PyTorch implementation is by Vincent Jeanselme (MIT); the +modifications are by Dulin Wang, under the same license ([LICENSE](LICENSE)). diff --git a/evaluation/README.md b/evaluation/README.md new file mode 100644 index 0000000..a9e8cb8 --- /dev/null +++ b/evaluation/README.md @@ -0,0 +1,140 @@ +# Training and evaluating landmark Dynamic-DeepHit models + +These scripts are the training and evaluation protocol used with this fork, with +every file path and column name turned into a setting. No data ships with them: +they read a cohort you keep wherever you keep it, and `make_synthetic` builds a +stand-in cohort so the whole workflow can be run and tested without patient data. + +On the cohort they were written for, these scripts reproduce the project's +original analysis code exactly. Checked: + +* a retrained model's weights; +* the predictions at fixed landmarks, at every visit and on the external cohort; +* every metric and bootstrap interval in its calibration and discrimination tables; +* the within-patient ablation. + +## The question being evaluated + +At a visit (or at a fixed day after the first visit), a patient who is still +event-free gets a predicted risk of the event of interest within *h* days. A +competing event can pre-empt it. Each evaluation row is one patient at one +landmark. Its outcome is the time from that landmark to the event or to +censoring, and which event it was. + +## Data layout + +``` +$DDH_DATA_ROOT/ + split_0/ train.csv val.csv test.csv one row per (patient, visit) + train_landmark.csv test_landmark.csv fixed-landmark tables (see below) + external.csv optional: external / later-period cohort + split_1/ ... +``` + +Each `split_k` is one random patient-level split. Every file in it, including +`external.csv`, has been preprocessed with transforms fitted on that split's +training patients. + +**Per-visit tables.** Column names are set with a JSON file (`--columns` or +`DDH_COLUMNS`); the defaults are in brackets. + +| column | meaning | +|---|---| +| patient id [`patient_id`] | | +| visit date [`visit_date`] | day 0 is the patient's first visit | +| visit number [`visit`] | order within the patient; not a predictor | +| time [`time_to_event`] | days from the **first** visit to event or censoring, the same on every row | +| event [`event`] | 0 censored, 1 event of interest, 2 competing event, the same on every row | +| event date [`event_date`] | optional; derived from date + time if absent | +| categorical [none] | optional integer-coded column given a learned embedding | +| meta [none] | any other non-predictor columns | + +Every other column is a predictor. Predictors must be numeric and already +imputed and scaled. **They must not contain NaN**: the model reads NaN in the first +predictor as padding. + +**Fixed-landmark tables.** `data.build_landmark_table` adds, for each patient +still at risk on day *L* (*L* in the landmark days), a snapshot row dated *L* days +after the first visit. It is forward-filled from their latest visit. "Days since" +columns are advanced to the landmark date, and the gap since the last visit is +recomputed. Build these from the unscaled table, then apply the same scaling. + +**External cohort.** For a temporal validation, keep each patient's visits up to +and including an anchor visit (for example their first visit inside the validation +window) and nothing after it. The outcome is still measured from the first visit. + +## Workflow + +```sh +export DDH_DATA_ROOT=~/ddh/data DDH_RUNS_ROOT=~/ddh/runs DDH_OUTPUT_DIR=~/ddh/results + +# 0. a synthetic cohort (skip with real data); it writes the column mapping it uses +python -m evaluation.make_synthetic --out $DDH_DATA_ROOT +export DDH_COLUMNS=$DDH_DATA_ROOT/columns.json + +# 1. train two configurations on every split +python -m evaluation.train --arm patient --splits 0 1 2 --all-landmark-nll --alpha 0 --hazard --proportional +python -m evaluation.train --arm band --splits 0 1 2 --all-landmark-nll --alpha 0 --hazard --proportional \ + --landmark-row-weighting band + +# 2. score the external cohort with each split's model +python -m evaluation.apply_external --arm patient --splits 0 1 2 +python -m evaluation.apply_external --arm band --splits 0 1 2 + +# 3. compare them; the first --arm (or --reference) is the reference +python -m evaluation.evaluate --arm band --arm "1/V_i=patient" --splits 0 1 2 + +# 4. does the model use earlier visits? +python -m evaluation.history_ablation --arm band --splits 0 1 2 --gap-col days_since_last_obs +``` + +The demo uses the proportional hazard head. On a cohort this small the free hazard +head (drop `--proportional`) over-fits within the first epochs: training loss falls +while validation loss rises from the start, because each of its ~100 time bins has its +own output weights and the synthetic training set has only a few events per bin. The +proportional head shares one patient score across bins. The configuration the project +reports uses the free head on a larger cohort, where validation loss kept improving for +several epochs. + +If your files use other names, set `DDH_FILE_TEMPLATE` (for example +`'cohort_{part}.csv'`) and `DDH_EXTERNAL_TEMPLATE`. If your run folders use another +layout, set `DDH_SPLIT_DIR_TEMPLATE`. Every flag has `--help`. + +## What `evaluate` reports + +All models are scored **raw**, with no recalibration. Every arm is scored on the +same rows, because a row any arm cannot score is dropped for all of them. + +| stage | population | uncertainty | files | +|---|---|---|---| +| `landmarks` | internal test patients still at risk at days 0, 90, 180, 270 | per-split metrics; mean and SD over splits; each arm's per-split difference from the reference, Wilcoxon signed-rank, Benjamini-Hochberg within horizon and metric | `landmarks_per_split.csv`, `landmarks_summary.csv` | +| `visits` | internal test patients at every visit, by days since the first visit | as above | `visits_per_split.csv`, `visits_summary.csv` | +| `external` | external cohort at the anchor; the split models averaged into one ensemble; all / first visit / history ≤ 365 d / > 365 d | patient bootstrap (1,000 resamples) for each arm, and for each arm's paired difference from the reference | `external.csv` | + +The metrics (definitions in [`metrics.py`](metrics.py)): + +| metric | definition | +|---|---| +| `obs` | Aalen-Johansen cumulative incidence of the event of interest at *h*, with the competing event as a competing event | +| `op` (O:E) | `obs` / mean predicted risk. Below 1 means the model over-predicts. | +| `auc` | IPCW cumulative/dynamic AUC with a competing event: cases (event of interest by *h*), each weighted by the inverse probability of being uncensored at their event time, against controls (still event-free after *h*). Rows censored, or with a competing event, before *h* are in neither group. Matches R timeROC's `AUC_1`, except that an event on day *h* counts as a case here. | +| `brier`, `ipa` | IPCW Brier score of the status at *h*, with Graf weights from the Kaplan-Meier censoring distribution; IPA = 1 − Brier / Brier of predicting `obs` for everyone | +| `slope`, `citl` | weighted logistic recalibration of the status at *h* on logit(risk): its slope, and its intercept with logit(risk) as an offset | +| `top_op` | O:E among the 10% highest-risk rows | + +**Comparing with an existing score.** Write the score's predicted risks +(`risk_d` columns) into the same files an arm writes: + +* `split_k/test_fixed_landmark_predictions.csv` and `split_k/test_every_visit_predictions.csv` for the internal stages; +* `split_k/external_anchor_predictions.csv` for the external stage. + +Then pass that folder with `--arm`. Rows the score does not cover drop out for every +arm, so the comparison stays paired. + +## Keeping patient data private + +Predictions and `ablation_predictions.csv` are **patient-level**. Keep +`DDH_RUNS_ROOT` and `DDH_OUTPUT_DIR` next to your data, not inside a repository. +The tables `evaluate` writes are aggregates. This repository's `.gitignore` refuses +`*.csv`, `*.parquet`, `*.pkl` and `*.pt` files, as a guard against committing any of +them.