diff --git a/LICENSE b/LICENSE index aa5fbb3..de9f517 100644 --- a/LICENSE +++ b/LICENSE @@ -1,6 +1,7 @@ MIT License Copyright (c) 2026 Vincent Jeanselme +Copyright (c) 2026 Dulin Wang (modifications listed in docs/modifications.md) Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal diff --git a/README.md b/README.md index 1c0b628..878cd46 100644 --- a/README.md +++ b/README.md @@ -1,15 +1,88 @@ -# DynamicDeepHit -This repository is a Pytorch implementation of the paper : [Dynamic Deep Hit](https://ieeexplore.ieee.org/document/8681104). Original implementation in Tensorflow can be found at [chl8856/Dynamic-DeepHit](https://github.com/chl8856/Dynamic-DeepHit). +# Dynamic-DeepHit for landmark prediction +A fork of [Jeanselme/DynamicDeepHit](https://github.com/Jeanselme/DynamicDeepHit), the PyTorch +implementation of [Dynamic-DeepHit](https://ieeexplore.ieee.org/document/8681104) (Lee et al., +IEEE TBME 2020; original TensorFlow code at +[chl8856/Dynamic-DeepHit](https://github.com/chl8856/Dynamic-DeepHit)). -The code is based on the [Deep Survival Machines repository](https://github.com/autonlab/DeepSurvivalMachines) and use a similar module structure. +The fork adapts the model to **landmark prediction from irregular clinic visits with a +competing risk**. At any visit, it predicts the risk of the event of interest within +*h* days, given everything seen so far and given that the patient is still event-free. +## What changed -### Code Description -All the code related to Dynamic Deep Hit is loacted in `ddh`. Only API should be used for application. +Each row is one or a few commits; [`docs/modifications.md`](docs/modifications.md) gives +the problem, the change and how it was tested. -An example is available in the `examples` folder. +**Foundation.** These change upstream defaults where the upstream behaviour was a bug. -## TODOs -- Full documentation -- Further testing to ensure the model is reproducing the original paper's performance +| change | where | +|---|---| +| Port to auton-survival (the maintained successor of DeepSurvivalMachines) | `ddh/dsm_compat.py` | +| Discretisation index fix: the earliest events were scored in the last bin | `discretize` | +| Training loop: per-epoch shuffle, full validation, LR schedule, gradient clipping, seeding | `utils.train_ddh`, `fit` | +| Ranking loss as in the paper: σ inside the exponential, summed over pairs, per-cause weights | `losses.ranking_loss` | +| Likelihood and ranking conditioned on survival to a landmark, plus six fixes from a self-review | `fit(s=...)`, `losses` | + +**Modifications.** Every one is opt-in; with default arguments, predictions are +bit-identical to the foundation. + +| change | switch | problem it addresses | +|---|---|---| +| Likelihood at every landmark | `all_landmark_nll=True` | trained on last visits only, evaluated at every landmark | +| Discrete-time cause-specific hazard head | `hazard=True`, `proportional=True` | the softmax head must spend all probability inside the time grid | +| Landmark-day decay and between-patient entry weights | `landmark_decay_days`, `entry_weight` | where in follow-up the objective puts its weight | +| Horizon-specific likelihood | `gamma`, `horizon_days` | nothing in the bin-level likelihood constrains cumulative risk at *h* | +| Landmark row weighting | `landmark_row_weighting="visit"/"given"/"band"` | patients with early events have few visits, so 1/Vi weighting over-weights them early | + +The configuration the project reports is `hazard=True, all_landmark_nll=True, alpha=0, +landmark_row_weighting="band"`. One experiment that was tried and not adopted (the ranking +term at every landmark) is kept on the `experiments/all-landmark-rank` branch; +[`docs/modifications.md`](docs/modifications.md#tried-and-not-adopted) says why it was dropped. + +## Install + +```sh +git clone https://github.com/dadudu13/DynamicDeepHit && cd DynamicDeepHit +pip install torch +# auton-survival 0.1.0; --no-deps because its release metadata pins torch<2.0 +pip install --no-deps "auton-survival @ git+https://github.com/autonlab/auton-survival@5dde465f7223601717abddc1d075e837707c403b" +pip install -e ".[test,eval]" +python -m pytest +``` + +Tested locally with Python 3.11 and torch 2.11 and 2.14; CI runs Python 3.11 and 3.12. + +## Use + +```python +import numpy as np +from ddh import DynamicDeepHit + +# x: per-patient arrays (visits x features); t, e: per-visit outcome time (days from the +# first visit) and event (0 censored, 1 event of interest, 2 competing); +# visit_days: per-patient arrays of each visit's day; s: each patient's last visit day. +model = DynamicDeepHit(split=50, hidden_rnn=32, typ="GRU", alpha=0, beta=0.75, hazard=True) +model.fit(x, t, e, iters=50, s=s, visit_days=visit_days, + all_landmark_nll=True, landmark_row_weighting="band", + landmark_band_edges=[0, 30, 90, 180, 365, 730]) +cif_event = 1 - model.predict_survival(x, [90.0, 180.0], risk=1) # unconditional, from day 0 +``` + +To get risks conditioned on a landmark, and the full train / external-cohort / +evaluation workflow, see [`evaluation/`](evaluation/README.md). It runs end to end on +the synthetic cohort it generates. + +## How the history is organised + +* One change per commit, each explaining the problem it solves. Every commit passes + the tests that exist at that point. +* Upstream's history is kept intact underneath. The work is merged through one pull + request per theme: foundation, landmark training, hazard head, landmark-time + weighting, horizon term, row weighting, packaging, evaluation, documentation. +* Experiments that were not adopted stay on their own branches. + +## Credit and license + +The original PyTorch implementation is by Vincent Jeanselme (MIT); the +modifications are by Dulin Wang, under the same license ([LICENSE](LICENSE)). diff --git a/docs/modifications.md b/docs/modifications.md new file mode 100644 index 0000000..cfbb043 --- /dev/null +++ b/docs/modifications.md @@ -0,0 +1,212 @@ +# Modifications to Dynamic-DeepHit + +This fork starts from [Jeanselme/DynamicDeepHit](https://github.com/Jeanselme/DynamicDeepHit) +(PyTorch port of Lee et al., *Dynamic-DeepHit*, IEEE TBME 2020) and adapts it to +**landmark prediction from irregular clinic visits with a competing risk**: at any visit, +predict the risk of the event of interest within *h* days, given everything seen so far +and given that the patient is still event-free. + +Every change below is one commit, and every commit in the history passes the test +suite that exists at that point. The changes come in two groups: + +* **Foundation** (sections 1–2): fixes and infrastructure. Several of these change the + model's default behaviour on purpose, because the upstream behaviour was a bug. +* **Modifications** (sections 3–7): new training objectives and output layers. Each + one is behind an argument whose default leaves the model exactly as it was. This + was checked at every commit by training a small model with default arguments, with + and without landmarks, and comparing the predictions, which stayed bit-identical + across all thirteen commits. + +Notation: patient *i* has visits at days *si1 < … < siVi* +(day 0 = first visit). Cause 1 is the event of interest and cause 2 the competing +event. *Fk(t)* is the cumulative incidence of cause *k*, and +*S(t) = 1 − Σk Fk(t)*. + +--- + +## 1. Foundation: making the port correct and reproducible + +| Change | Why | Effect on defaults | +|---|---|---| +| Import the DSM helpers from **auton-survival** through `ddh/dsm_compat.py` | The `DeepSurvivalMachines` package the code imported from was renamed and repackaged; its old import paths no longer resolve | none | +| **Clip discretised times** to the bins training can reach | A time equal to the first bin edge got index −1, which torch reads as the *last* bin, so the earliest events were scored as the latest | yes (bug fix) | +| **Training loop**: reshuffle each epoch, validate on every row, ReduceLROnPlateau, gradient clipping, configurable patience | Validation counted batches with one batch size and sliced with another, so early stopping watched only a fixed fraction of the validation set | yes | +| **Seed every RNG** in `fit` | `random_state` reached only the train/validation shuffle; weight initialisation and batch order were unseeded | yes | +| **Restore RNN bias terms** | With `bias=False` no gate can learn an offset, and an all-mean input from a zero state is stuck at a zero hidden state | yes | +| Optional **learned embedding** for one integer-coded categorical column (`cancer_vocab_size`, `cancer_embedding_dim`, `cancer_id_col`) | An integer-coded category fed as a number is treated as ordinal | none (off unless all three are set) | +| **σ inside the exponential** of the ranking loss | Upstream computed exp(Δ)/σ instead of the paper's exp(Δ/σ), so σ only rescaled the term | yes (bug fix) | +| Ranking loss **summed over pairs**, with **per-cause weights** αk (`cause_weights`) | The paper sums over comparable pairs with a weight per cause; the code averaged and fixed αk = 1/K | yes | + +## 2. Foundation: conditioning on the landmark + +A prediction made at day *s* must condition on survival to *s*: + + P(T = t, K = k | T > s, history up to s) = P(T = t, K = k | history) / S(s) + +`fit(s=...)` takes one landmark per patient. The likelihood subtracts log *S(si)*. The +ranking term compares patients on remaining time *t − s* and scores each comparator's +conditional CIF *ri* days after its own landmark. With `s=None` every function takes +its original path. + +A self-review of this change found six problems, each fixed in its own commit: + +1. Landmarks were clipped to a wider bin range than event times. +2. `compute_nll` ignored landmarks. +3. The embedded categorical column was still a reconstruction target. +4. The conditioning term's survival floor (1e-10) allowed gradients of order 1e10; it is now 1e-3. +5. Validation loss depended on the batch size; the whole validation set is now scored in one pass. +6. auton-survival was found only through one hard-coded relative path; it can now come from an environment variable, a nearby checkout, or an installed package. + +--- + +## 3. Train the likelihood at every landmark (`all_landmark_nll=True`) + +**Problem.** The model was trained with one row per patient: full history, +conditioned on the last visit. Evaluation, though, truncates *every* patient to each +landmark. Short-history inputs were therefore seen in training almost only for +patients whose follow-up really was short, a group with early events. So the model +learned "short history → high risk", and over-predicted at early landmarks. + +**Change.** Every (patient, visit) prefix contributes a conditional likelihood term: + + L = (1/K) Σi Σv wiv · −log P(Ti, Ki | Ti > siv, Xi,≤v), with wiv = 1/Vi by default + +The expanded rows are back-propagated in chunks (`landmark_chunk_rows`). This is exact, +since the objective is a weighted sum. Validation uses the same objective. With +*Vi = 1* the objective reduces to the previous loss, which the tests assert. + +Before this, a cheaper variant was tried and kept: `landmark_augment_prob` truncates a +random subset of patients to a random earlier visit each epoch. The deterministic +objective above covers every landmark instead of a sample. + +Also in this group: a batch in which no patient has a second visit no longer makes +the reconstruction loss NaN. + +## 4. Discrete-time cause-specific hazard head (`hazard=True`, `proportional=True`) + +**Problem.** The DeepHit output is one softmax over all (cause, bin) cells, so the +total event probability inside the time grid is forced to 1. Patients who reach the +end of follow-up event-free have nowhere to go, and the model piles probability into +the last bin. + +**Change.** `hazard=True` emits per-bin competing-risks hazards: K cause logits plus a +"survive this bin" logit, softmaxed within each bin. These are converted back to the +same (cause, bin) probability interface, so every downstream consumer is unchanged, +but P(no event by the last bin) is now representable. The survive logit's bias is +initialised in closed form so initial survival is not ≈ 0 on long grids. +`proportional=True` additionally factorises logit λk(b) = αk(b) + βk·h: a per-bin +baseline plus one shared patient score. This makes the risk ordering identical in +every bin. The tests assert that, and also assert it does *not* hold for the free head. + +## 5. Re-weighting the all-landmark objective over time (`landmark_decay_days`, `entry_weight`) + +* `landmark_decay_days=T` weights each landmark by exp(−day/T), renormalised within + each patient. It is keyed on **day**, not visit index: visit index mixes up clinical + timing with how often a patient is seen. +* `entry_weight` scales a patient's *total* contribution. It is applied outside the + per-patient normalisation, because a per-patient constant folded inside would divide + straight back out. A test pins that placement. + +## 6. Horizon-specific likelihood (`gamma`, `horizon_days`, `horizon_ipcw_clip`) + +**Problem.** The bin-level likelihood scores the bin an event fell in. Nothing in it +constrains the *cumulative* risk at the horizons the model is used at. The model +over-predicted the 90-day level even in sample, so this was the objective not asking, +rather than a failure to generalise. + +**Change.** For every landmark row and every horizon *h*, add the IPCW-weighted binary +log-likelihood of the observed status at *s + h* under +p = (F1(s+h) − F1(s)) / S(s). + +* Labels and weights follow the evaluation exactly (Graf et al.): a competing event + before *h* is a true 0, and a row censored before *h* carries no weight. +* The censoring weights are conditioned on the landmark (G(·)/G(s)). Without this, + the term inflated late landmarks. +* `gamma` is added on top of α and β, so `gamma=0` is bit-identical to the previous + objective. + +## 7. Who counts how much: landmark row weighting (`landmark_row_weighting`) + +**Problem.** Under 1/Vi weighting every patient contributes a total weight of 1, spread +over their visits. But visits stop at the event, so patients with an early event have +few visits, and each of their early rows carries a large weight. On the development +cohort this put the first-visit rows' 90-day event rate well above the true rate, while a +static model trained on first visits only, one row per patient, did not over-predict. + +**Change.** Three alternatives to `"patient"` (1/Vi): + +* `"visit"`: every row weighs the same. +* `"given"`: per-row weights from outside the model (`row_weights`), e.g. inverse-intensity weights from a model of the visit process. +* `"band"`: each patient counts **once per landmark-day band** (`landmark_band_edges`). This is how the model is evaluated, with each patient still at risk scored once per landmark. A single band reproduces 1/Vi, and bands holding one visit each reproduce equal rows; the tests assert both limits. + +All three keep the batch's total weight equal to the per-patient scheme's. Only the +distribution changes, which matters because gradients are clipped to a fixed norm. + +### How the weighting was chosen + +The weighting was settled by a sequence of single-change comparisons on the development +cohort, every arm scored raw (no recalibration): + +1. **1/Vi** (section 3) fixed the last-visit-only training, but over-predicted at + the first visits. +2. **Decay by landmark day** (section 5) was the deployed arm for a while, and needed a + post-hoc recalibration to be calibrated. +3. **A static model** trained on first visits only, one row per patient, with the same + output layer, was calibrated at the first visit. So the over-prediction came from the + weighting, not from the hazard head. +4. **Equal weight per visit** removed the early over-prediction, as a one-factor change. +5. **Inverse-intensity weights** from a model of the visit process landed on the per-visit + result: modelling the visit process said what the simple rule already assumed. +6. **Band weighting** weights rows the way the model is evaluated, each patient still at + risk counted once per landmark. It was the best calibrated across landmarks without any + recalibration, and replaced decay plus recalibration as the reported configuration. + +No weighting changed discrimination; the choice is about calibration. An under-prediction +after the first year was a separate problem that none of the weightings addressed. + +The synthetic cohort (`evaluation/make_synthetic.py`) is built with the property behind +step 1: visits stop at the event, so patients with an early event have few visits. It +reproduces the first-visit effect with the published code: + +| arm, 90-day risk | O:E at the first visit | O:E at day 90 | AUC at the first visit | +|---|---|---|---| +| 1/Vi (`patient`) | 0.53 | 0.69 | 0.82 | +| decay, T = 180 d | 0.85 | 0.98 | 0.81 | +| equal per visit (`visit`) | 1.35 | 1.10 | 0.79 | +| band (`band`) | 1.31 | 1.03 | 0.80 | +| static, first visit only | 1.21 | 1.11 | 0.82 | + +O:E is observed (Aalen-Johansen) over mean predicted risk; below 1 is over-prediction. +1/Vi over-predicts about twofold at the first visit; giving each patient equal +weight there removes it (on this small cohort the correction overshoots a little), and AUC +barely moves. The simulation is not built to mimic the real cohort beyond the first visits, +where every arm over-predicts, so read it for the first-visit contrast only. Means over three +splits, with the proportional hazard head (see [`evaluation/`](../evaluation/README.md)); +reproduce with: + +```sh +python -m evaluation.make_synthetic --out $DDH_DATA_ROOT +export DDH_COLUMNS=$DDH_DATA_ROOT/columns.json +COMMON="--splits 0 1 2 --iters 40 --alpha 0 --hazard --proportional" +python -m evaluation.train --arm patient $COMMON --all-landmark-nll +python -m evaluation.train --arm visit $COMMON --all-landmark-nll --landmark-row-weighting visit +python -m evaluation.train --arm band $COMMON --all-landmark-nll --landmark-row-weighting band +python -m evaluation.train --arm decay $COMMON --all-landmark-nll --landmark-decay-days 180 +python -m evaluation.train --arm static $COMMON --first-visit-only +python -m evaluation.evaluate --arm band --arm "1/V_i=patient" --arm "per visit=visit" \ + --arm decay --arm "static=static" --splits 0 1 2 --stage landmarks +``` + +**Configuration the project reports:** `hazard=True`, `all_landmark_nll=True`, +`alpha=0` (no ranking term), `landmark_row_weighting="band"`. See +[`evaluation/`](../evaluation/README.md) for how it is trained and evaluated. + +--- + +## Tried and not adopted + +One change was tried and not merged; it is kept on its own branch. + +| Branch | Idea | Why it was not adopted | +|---|---|---| +| `experiments/all-landmark-rank` | Move the ranking term onto the same (patient, visit) rows as the all-landmark likelihood, pairs weighted *wiwj*, within-patient pairs dropped | A three-split pilot against the all-landmark likelihood alone showed no discrimination gain (the interval included zero and was narrow next to the gain the all-landmark likelihood itself had bought). Calibration moved slightly the wrong way, and each epoch took 50% longer. Stopped after the pilot. | diff --git a/evaluation/README.md b/evaluation/README.md new file mode 100644 index 0000000..a9e8cb8 --- /dev/null +++ b/evaluation/README.md @@ -0,0 +1,140 @@ +# Training and evaluating landmark Dynamic-DeepHit models + +These scripts are the training and evaluation protocol used with this fork, with +every file path and column name turned into a setting. No data ships with them: +they read a cohort you keep wherever you keep it, and `make_synthetic` builds a +stand-in cohort so the whole workflow can be run and tested without patient data. + +On the cohort they were written for, these scripts reproduce the project's +original analysis code exactly. Checked: + +* a retrained model's weights; +* the predictions at fixed landmarks, at every visit and on the external cohort; +* every metric and bootstrap interval in its calibration and discrimination tables; +* the within-patient ablation. + +## The question being evaluated + +At a visit (or at a fixed day after the first visit), a patient who is still +event-free gets a predicted risk of the event of interest within *h* days. A +competing event can pre-empt it. Each evaluation row is one patient at one +landmark. Its outcome is the time from that landmark to the event or to +censoring, and which event it was. + +## Data layout + +``` +$DDH_DATA_ROOT/ + split_0/ train.csv val.csv test.csv one row per (patient, visit) + train_landmark.csv test_landmark.csv fixed-landmark tables (see below) + external.csv optional: external / later-period cohort + split_1/ ... +``` + +Each `split_k` is one random patient-level split. Every file in it, including +`external.csv`, has been preprocessed with transforms fitted on that split's +training patients. + +**Per-visit tables.** Column names are set with a JSON file (`--columns` or +`DDH_COLUMNS`); the defaults are in brackets. + +| column | meaning | +|---|---| +| patient id [`patient_id`] | | +| visit date [`visit_date`] | day 0 is the patient's first visit | +| visit number [`visit`] | order within the patient; not a predictor | +| time [`time_to_event`] | days from the **first** visit to event or censoring, the same on every row | +| event [`event`] | 0 censored, 1 event of interest, 2 competing event, the same on every row | +| event date [`event_date`] | optional; derived from date + time if absent | +| categorical [none] | optional integer-coded column given a learned embedding | +| meta [none] | any other non-predictor columns | + +Every other column is a predictor. Predictors must be numeric and already +imputed and scaled. **They must not contain NaN**: the model reads NaN in the first +predictor as padding. + +**Fixed-landmark tables.** `data.build_landmark_table` adds, for each patient +still at risk on day *L* (*L* in the landmark days), a snapshot row dated *L* days +after the first visit. It is forward-filled from their latest visit. "Days since" +columns are advanced to the landmark date, and the gap since the last visit is +recomputed. Build these from the unscaled table, then apply the same scaling. + +**External cohort.** For a temporal validation, keep each patient's visits up to +and including an anchor visit (for example their first visit inside the validation +window) and nothing after it. The outcome is still measured from the first visit. + +## Workflow + +```sh +export DDH_DATA_ROOT=~/ddh/data DDH_RUNS_ROOT=~/ddh/runs DDH_OUTPUT_DIR=~/ddh/results + +# 0. a synthetic cohort (skip with real data); it writes the column mapping it uses +python -m evaluation.make_synthetic --out $DDH_DATA_ROOT +export DDH_COLUMNS=$DDH_DATA_ROOT/columns.json + +# 1. train two configurations on every split +python -m evaluation.train --arm patient --splits 0 1 2 --all-landmark-nll --alpha 0 --hazard --proportional +python -m evaluation.train --arm band --splits 0 1 2 --all-landmark-nll --alpha 0 --hazard --proportional \ + --landmark-row-weighting band + +# 2. score the external cohort with each split's model +python -m evaluation.apply_external --arm patient --splits 0 1 2 +python -m evaluation.apply_external --arm band --splits 0 1 2 + +# 3. compare them; the first --arm (or --reference) is the reference +python -m evaluation.evaluate --arm band --arm "1/V_i=patient" --splits 0 1 2 + +# 4. does the model use earlier visits? +python -m evaluation.history_ablation --arm band --splits 0 1 2 --gap-col days_since_last_obs +``` + +The demo uses the proportional hazard head. On a cohort this small the free hazard +head (drop `--proportional`) over-fits within the first epochs: training loss falls +while validation loss rises from the start, because each of its ~100 time bins has its +own output weights and the synthetic training set has only a few events per bin. The +proportional head shares one patient score across bins. The configuration the project +reports uses the free head on a larger cohort, where validation loss kept improving for +several epochs. + +If your files use other names, set `DDH_FILE_TEMPLATE` (for example +`'cohort_{part}.csv'`) and `DDH_EXTERNAL_TEMPLATE`. If your run folders use another +layout, set `DDH_SPLIT_DIR_TEMPLATE`. Every flag has `--help`. + +## What `evaluate` reports + +All models are scored **raw**, with no recalibration. Every arm is scored on the +same rows, because a row any arm cannot score is dropped for all of them. + +| stage | population | uncertainty | files | +|---|---|---|---| +| `landmarks` | internal test patients still at risk at days 0, 90, 180, 270 | per-split metrics; mean and SD over splits; each arm's per-split difference from the reference, Wilcoxon signed-rank, Benjamini-Hochberg within horizon and metric | `landmarks_per_split.csv`, `landmarks_summary.csv` | +| `visits` | internal test patients at every visit, by days since the first visit | as above | `visits_per_split.csv`, `visits_summary.csv` | +| `external` | external cohort at the anchor; the split models averaged into one ensemble; all / first visit / history ≤ 365 d / > 365 d | patient bootstrap (1,000 resamples) for each arm, and for each arm's paired difference from the reference | `external.csv` | + +The metrics (definitions in [`metrics.py`](metrics.py)): + +| metric | definition | +|---|---| +| `obs` | Aalen-Johansen cumulative incidence of the event of interest at *h*, with the competing event as a competing event | +| `op` (O:E) | `obs` / mean predicted risk. Below 1 means the model over-predicts. | +| `auc` | IPCW cumulative/dynamic AUC with a competing event: cases (event of interest by *h*), each weighted by the inverse probability of being uncensored at their event time, against controls (still event-free after *h*). Rows censored, or with a competing event, before *h* are in neither group. Matches R timeROC's `AUC_1`, except that an event on day *h* counts as a case here. | +| `brier`, `ipa` | IPCW Brier score of the status at *h*, with Graf weights from the Kaplan-Meier censoring distribution; IPA = 1 − Brier / Brier of predicting `obs` for everyone | +| `slope`, `citl` | weighted logistic recalibration of the status at *h* on logit(risk): its slope, and its intercept with logit(risk) as an offset | +| `top_op` | O:E among the 10% highest-risk rows | + +**Comparing with an existing score.** Write the score's predicted risks +(`risk_d` columns) into the same files an arm writes: + +* `split_k/test_fixed_landmark_predictions.csv` and `split_k/test_every_visit_predictions.csv` for the internal stages; +* `split_k/external_anchor_predictions.csv` for the external stage. + +Then pass that folder with `--arm`. Rows the score does not cover drop out for every +arm, so the comparison stays paired. + +## Keeping patient data private + +Predictions and `ablation_predictions.csv` are **patient-level**. Keep +`DDH_RUNS_ROOT` and `DDH_OUTPUT_DIR` next to your data, not inside a repository. +The tables `evaluate` writes are aggregates. This repository's `.gitignore` refuses +`*.csv`, `*.parquet`, `*.pkl` and `*.pt` files, as a guard against committing any of +them.