From 682768b238883796b673b6ece9152ac5b14e1371 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Tue, 22 Sep 2026 22:08:56 -0400 Subject: [PATCH 01/16] Add proposed v0.5 survival-dates ADR (#40) Design for generating survival dates inside create_mock_data() from metadata, agreed with the maintainer: full parity with create_wide_survival_data() (entry, event, death, LTFU, admin censoring and both consistency rules), exact port of the legacy statistics, and a post-baseline anchored stage modelled on the #39 formula stage rather than a multi-column generator. Metadata gains `anchor` and `censored_by` on variables.csv; the seed contract gains an appended `survival` sub-stream. Two decisions await ratification: applying garbage and missing codes after the consistency rules (D5), and adding is.na to the formula allow-list so status and follow-up time can be derived with mockFormula (D8). Records six concerns found while preparing the design, to be filed as issues. Most consequential: with the packaged Gompertz parameters, 388 of 400 legacy death dates land exactly at followup_min, so the minimal example's competing-risk rule never fires. Context claims were verified against the running code on 2026-09-22. --- development/adr/v05-survival-dates.md | 179 ++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 development/adr/v05-survival-dates.md diff --git a/development/adr/v05-survival-dates.md b/development/adr/v05-survival-dates.md new file mode 100644 index 0000000..0921a5a --- /dev/null +++ b/development/adr/v05-survival-dates.md @@ -0,0 +1,179 @@ +# ADR: v0.5 Metadata-Driven Survival Dates + +**Status**: PROPOSED 2026-09-22. Design agreed with the maintainer in conversation (scope, fidelity policy, architecture, metadata convention, semantics). Two points need explicit ratification before implementation: D5 (contamination after the consistency rules) and D8 (adding `is.na` to the formula allow-list). +**Date**: 2026-09-22 +**Decision owner**: MockData maintainers +**Issue**: #40 (also advances #23, #17) +**Depends on**: #38 (seed contract), #39 (formula stage; dependency-ordering machinery reused here) + +## Decision summary + +| # | Question | Decision | +|---|---|---| +| D1 | Architecture | Survival dates are a new variable type computed in a **post-baseline stage** from their already-generated anchor date. One column per spec variable; the assembly contract is unchanged | +| D2 | Scope | **Full parity** with `create_wide_survival_data()`: entry, event, death, loss to follow-up, administrative censoring, and both consistency rules | +| D3 | Metadata convention | Two columns on `variables.csv` beside `followup_min`/`followup_max`/`event_prop`: `anchor` (required) and `censored_by` (optional) | +| D4 | Fidelity | **Exact port** of the legacy statistics and rules; concerns are filed as issues, not fixed in #40 | +| D5 | Contamination order | Consistency rules run on clean dates; missing codes and garbage are applied afterwards by postprocess (**ratify**) | +| D6 | Stage and seed | Pipeline becomes baseline, survival, formulas, postprocess. `.MOCK_STAGES` gains `survival = 4L` (appended; no existing index moves) | +| D7 | Dependencies | Survival dates record `depends_on = c(anchor, censored_by)`, the field formulas already use; one ordering utility serves every derived stage | +| D8 | Formula allow-list | Add `is.na`, so survival status and time can be derived with `mockFormula` (**ratify**; amends the #39 ADR's D3) | +| D9 | Legacy path | `create_wide_survival_data()` warns once as deprecated and is kept as the characterization reference | + +## Context + +Survival data has been generated outside `create_mock_data()` since v0.2.0. The NEWS known-issue entry reads: "Cannot be used in `create_mock_data()` batch generation (requires paired variables)". Users call `create_wide_survival_data()` by hand. + +The legacy engine is a loop. It generates the entry date as a plain calendar date. It then generates each other date by calling `create_date_var()` with the entry date passed in a `df_mock` column named `anchor_date` (`R/create_wide_survival_data.R`, step 1). Finally it applies two rules: any date earlier than entry becomes `NA` (`:347`), and an event date later than the death date becomes `NA` (`:363`). The per-date statistics live in `create_date_var()` (`R/create_date_var.R:209-350`): a date is a survival date when `followup_min`, `followup_max` and `event_prop` are all present; `floor(n_valid * event_prop)` rows receive an event, shuffled; follow-up times are uniform, exponential or Gompertz within the window; non-events are `NA`. + +The v0.4 pipeline handles these dates incorrectly today, in both of its paths (verified 2026-09-22 on the minimal example): + +- A Gompertz or exponential survival date fails the native support check (`R/create_mock_data.R:36`), and the fallback is all-or-nothing for the spec. The legacy dispatcher then calls `create_date_var()` without an `anchor_date` column, so every survival date in the spec, uniform ones included, warns and is dropped. For the minimal example, `create_mock_data()` returns `interview_date` and none of the four survival dates. +- In a spec whose survival dates are all uniform, nothing forces the fallback, and the native backend generates them as plain calendar dates over their `recStart` range. The adapter reads `followup_min`, `followup_max` and `event_prop` onto the spec (`R/mock_spec_recodeflow.R:418`), but nothing downstream uses them. With `interview_date` and `ltfu_date` alone (n = 2,000), `ltfu_date` is 100 per cent non-missing although `event_prop` is 0.1, and 128 dates fall before entry. + +So #40 fixes a correctness defect as well as a usability gap. + +The design follows from #39. A formula variable is a column computed from other generated columns, evaluated in dependency order after baseline generation. A survival date has the same structure: a column computed from its anchor. The only difference is that it draws random numbers. Treating survival dates as a derived stage reuses #39's skip-in-baseline pattern, dependency ordering, strict validation at stage entry, and diagnostics marking. + +### Constraints + +- **One seeded-output break in v0.5**, already spent by #38. Specs without survival variables must produce byte-identical output; the #48 pinned-value tests enforce this. +- **No backward-compatibility obligation** for 0.x metadata (maintainer decision, 2026-09-22). Metadata that relied on the old silent behaviour may now fail validation, provided the error names the fix. +- **recodeflow semantics untouched.** The new columns are MockData extensions; cchsflow and chmsflow need no change. +- **Keep causal modelling open.** The maintainer expects mock data to be used to study causal structure (DAG-based simulation) in future. That is out of scope here, but no decision should make a later structural-equation stage harder. D1 and D7 are chosen with this in mind. + +## Decision points + +### D1: Architecture + +**(a) Post-baseline anchored stage (chosen).** A new type `survival` (rtype `date`). Both backends skip it in baseline generation, as they skip `formula` (`R/mock_spec_native.R:87,405`; `R/mock_spec_simstudy.R:137,260`). A new exported `generate_survival_dates(data, spec, seed = NULL)` computes the survival dates from the anchor columns in `data`, then applies the consistency rules. Postprocess then treats them like any other column. + +**(b) Multi-column generator (rejected).** One spec entry returning a data frame of two or more columns. This changes the one-column-per-variable contract in the native backend, the simstudy backend, postprocess and diagnostics. It also forces an awkward ownership question for a five-date family, since the entry date would be both a plain variable and a member of the group. + +**(c) Delegate to `create_wide_survival_data()` (rejected).** This bypasses spec validation, diagnostics, the seed contract and postprocess, and makes a deprecated function part of the v0.4 pipeline. + +### D2: Scope + +Full parity with the five legacy roles and both rules. No time-to-event or status columns are generated; the legacy engine produces none. D8 lets users derive them with `mockFormula`. + +### D3: Metadata convention + +| Column | On | Meaning | +|---|---|---| +| `anchor` | `variables.csv` | Name of the date variable this date is generated relative to. A non-blank `anchor` makes the variable type `survival` | +| `censored_by` | `variables.csv` | Optional. Name of a competing survival date. Where that date is earlier than this one, this one becomes `NA` | + +The minimal example becomes: + +``` +variable anchor censored_by followup_min followup_max event_prop distribution +interview_date uniform +primary_event_date interview_date death_date 0 5475 0.3 gompertz +death_date interview_date 365 7300 0.2 gompertz +ltfu_date interview_date 365 7300 0.1 uniform +admin_censor_date interview_date 365 7300 1.0 +``` + +A single `survival_role` column (`entry`, `event`, `death`, `ltfu`, `admin_censor`) was considered and rejected. It reproduces the legacy function's five argument slots, which fix one entry date and one competing-risk rule. `anchor` and `censored_by` express the same case and extend to a second cohort or a second competing risk without a new vocabulary. `censored_by` also makes the competing-risk rule visible in metadata. + +**Placement.** The columns are unprefixed and sit on `variables.csv`, beside the existing survival parameters. The rule, to be written into `reference-config.qmd`, is: MockData extension columns on `variables.csv` are unprefixed; MockData columns on `variable_details.csv`, where they sit among recodeflow's semantic columns, carry the `mock` prefix (as `mockFormula` does). A MockData-owned sidecar that keeps recodeflow files free of extension columns is a separate v0.6 decision (see Future directions); the spec layer does not depend on where columns are read from, so that change will not affect this design. + +**Validation of the convention.** A date variable with any of `followup_min`, `followup_max` or `event_prop` but no `anchor` is an error at spec construction, naming the fix (`anchor = ""`). A variable with `anchor` but without all three parameters is also an error. + +### D4: Fidelity + +The legacy statistics and rules are ported verbatim: + +1. `n_events = floor(n * event_prop)`; the event indicator is shuffled with `sample()`. +2. Follow-up days for events, by `distribution`. **Uniform:** `runif(n_events, followup_min, followup_max)`. **Exponential:** `rexp(n_events, rate = 1 / ((followup_max - followup_min) / 3)) + followup_min`, capped at `followup_max`. **Gompertz:** `(1/shape) * log(1 - (shape/rate) * log(1 - u))` with `u ~ runif`, clamped to the window; `shape` defaults to 0.1 and `rate` to 0.0001. +3. Date = anchor + follow-up days for events; `NA` otherwise. +4. After all survival dates are drawn: any date earlier than its anchor becomes `NA`; then, for each variable with `censored_by`, rows where the censoring date is earlier become `NA`. + +Survival dates are drawn in spec `position` order. No ordering among them is needed, because every anchor is a baseline date. + +Output cannot be bit-identical to `create_wide_survival_data()`, because the RNG streams differ (#38). Parity is therefore tested structurally and distributionally (Test strategy). Two further differences are inherent in the pipeline: + +| Legacy | New pipeline | Reason | +|---|---|---| +| Missing codes fill the last `n - n_valid` rows; events are split across the first `n_valid` rows (`R/create_date_var.R:295,417`) | Postprocess assigns missing codes at random rows | Missingness belongs to postprocess in v0.4. Proportions match in expectation | +| Garbage is applied inside `create_date_var()`, before the rules, so before-entry garbage is erased | Rules run first; garbage afterwards (D5) | Stage order | + +### D5: Contamination after the consistency rules (ratify) + +The consistency rules produce clean, analysis-ready dates. Missing codes and garbage are applied afterwards, by postprocess, as for every other variable. This matches #39, where formulas are computed on clean baseline values before contamination. + +It changes one behaviour. In the legacy engine, garbage that places a date before entry is set to `NA` by the before-entry rule; `test-survival-garbage-deprecation.R:147` acknowledges this. The garbage tutorial states that survival garbage exists to create temporal violations such as death before entry, for testing cleaning pipelines. Under D5 those violations survive, so the pipeline does what the tutorial describes. This is a change of stage order, not of any rule, but it alters legacy output and so needs explicit ratification under D4. + +### D6: Stage and seed + +`create_mock_data()` runs baseline, survival, formulas, then postprocess. Survival precedes formulas so that a `mockFormula` can use survival dates. `.MOCK_STAGES` (`R/mock_spec_native.R:12`) gains `survival = 4L`. Appending leaves indices 0 to 3 in place, so every existing pinned value holds; the frozen-stage test (`test-seed-contract.R:99`) is extended, not renumbered. Survival draws run inside `.with_mock_seed(seed, stage = "survival")`. + +### D7: Dependencies + +Survival variables set `depends_on = c(anchor, censored_by)`. `.order_formula_variables()` (`R/mock_spec_formula.R:88`) is generalized to order any derived type by `depends_on`, and the formula stage uses the generalized version. The purpose is forward compatibility: a later structural-equation stage, whether in MockData or a separate package, would read a single dependency field from the spec and add a type rather than a mechanism. + +### D8: Add `is.na` to the formula allow-list (ratify) + +The #39 allow-list has no missingness test, so a formula cannot derive an event indicator from a survival date. With `is.na` added, users can write, for example, `as.integer(!is.na(death_date))` for status and `as.numeric(pmin(death_date, admin_censor_date, na.rm = TRUE) - interview_date)` for follow-up time in days. The #39 ADR permits widening the list on concrete need; this is one. `is.na` has no side effects and cannot reach outside the sandbox. + +### D9: Legacy path + +`create_wide_survival_data()` is already tagged `@keywords deprecated`. v0.5 adds a one-time deprecation warning pointing to metadata-driven generation. The function is retained unchanged: it is the reference for the characterization tests that pin the rules before the port. + +## Validation, direct API and diagnostics + +**Validator.** A `survival` branch in `validate_mock_spec()` beside the `date` and `formula` branches (`R/mock_spec.R:968`): + +- `anchor` names a spec variable of type `date` (not `survival` or `formula`; chained anchors are not supported) +- `censored_by`, if set, names a `survival` variable with the same `anchor` +- `followup_min` and `followup_max` finite, non-negative, and `followup_min <= followup_max` +- `event_prop` in [0, 1] +- `distribution` one of `uniform`, `exponential`, `gompertz`; `shape` and `rate` positive when supplied +- `rtype` is `date` + +`generate_survival_dates()` strict-validates at entry, like `evaluate_mock_formulas()`, and stops if an anchor column is absent from `data`. + +**Adapter.** `.recodeflow_variable_kind()` (`R/mock_spec_recodeflow.R:117`) returns `survival` for a date row with a non-blank `anchor`. Construction reads `anchor` and `censored_by` beside the existing `followup_*`, `event_prop`, `rate` and `shape` reads. The `recStart` range of a survival date is not used for generation, as in the legacy engine. + +**Direct API.** `mock_survival()` returns a one-variable `mock_spec`; `mock_spec_survival()` returns a composable variable. Arguments: `name`, `anchor`, `followup_min`, `followup_max`, `event_prop`, `distribution = "uniform"`, `censored_by = NULL`, `shape = NULL`, `rate = NULL`, plus the usual missing-code, garbage, provenance and model-hint arguments. + +**Diagnostics.** Following #39's D5 (`R/mock_spec_postprocess.R:35`): `derived = TRUE`, `anchor`, `censored_by`, `depends_on`, and `n_events`, the number of non-`NA` dates after the consistency rules and before postprocess. + +## Concerns to file as issues (not fixed in #40) + +Found while preparing this design. Each is reproduced exactly by the port. + +1. **Gompertz output is degenerate with the packaged parameters.** With `shape = 0.1` and `rate = 1e-04`, time in days, the inverse-CDF never exceeds about 94 days (median 65). In the minimal example every non-garbage `death_date` therefore clamps to `followup_min`, exactly 365 days after entry (388 of 400 deaths in a 2,000-row legacy run; the other 12 are the 3 per cent high garbage). Every non-garbage `primary_event_date` falls within about 94 days despite a 15-year window. No death precedes an event, so the competing-risk rule never fires. The survival tutorial is built on this fixture. Whether the fault lies in the parameterization, the time scale, or the fixture values needs a methodological decision before the tutorial is rewritten around `create_mock_data()`. +2. **Administrative censoring is drawn per person.** `admin_censor_date` has `event_prop = 1.0` and a follow-up window, so each person gets a random date between one and 20 years after entry (1,762 distinct values in 2,000 rows). Its `variable_details` row specifies a fixed date, 2024-12-31, which is ignored. Administrative censoring is usually a fixed study end date. +3. **Declared date missingness is ignored.** Every missing row for the minimal example's dates is `else` with `NA::b` and a proportion (0.05 for `death_date`). Neither path applies it: the v0.4 adapter skips `else` rows (`R/mock_spec_recodeflow.R:232`), and the legacy `death_date` is exactly 20.0 per cent non-missing, its `event_prop`. Belongs with #15. +4. **The documentation overstates censoring.** `create_wide_survival_data()` documents that "observation ends at min(event, death, ltfu, admin_censor)" (`:61`), but the code applies only death before event. Under D3, users encode additional rules with `censored_by`. +5. **Positional missing-code placement** in `create_date_var()`: the last rows are the missing ones, so row order carries meaning. +6. **Inert range rows.** The `recStart` range on a survival date's `variable_details` row (for example `[2002-01-01,2021-01-01]`) never bounds anything, which invites the reader to think it does. + +## Consequences + +- `create_mock_data()` generates complete survival data from metadata, closing the NEWS known issue from v0.2.0. +- Survival metadata that previously generated silently wrong output now either generates correctly (with `anchor`) or fails validation with a message naming the fix. +- New exports: `generate_survival_dates()`, `mock_survival()`, `mock_spec_survival()`. Additions to `_pkgdown.yml` in the same PR. +- The formula allow-list grows by one function (D8). +- Specs without survival variables: output unchanged. + +## Test strategy + +1. **Characterization first.** Pin the legacy engine's two rules and its structural invariants on `create_wide_survival_data()` before any port code, as the reference. +2. **Validator and adapter.** One failing case, with its message, for each rule in "Validation"; the missing-anchor error. +3. **Generation.** Event proportion within two-sided binomial bounds for each distribution (#23's direction); every non-`NA` date at or after its anchor; `censored_by` sets `NA` exactly where the censoring date is earlier and nowhere else; typed zero-row output for `n = 0`; skipping the stage makes postprocess fail on the missing column. +4. **Contract.** Frozen-stage test extended with `survival = 4L`; pinned reference values for one survival spec; all existing pins untouched. +5. **End to end.** `create_mock_data()` on the amended minimal example produces all five dates with no fallback; the simstudy backend produces the same survival columns; a `mockFormula` derives status and follow-up time from them (D8); D5 is pinned by a test showing before-entry garbage survives. + +## Migration + +- Metadata: add `anchor` (and `censored_by` where a competing risk applies) to each survival date's row. Validation names the missing column. +- Code calling `create_wide_survival_data()` keeps working, with a deprecation warning. Replace it with `create_mock_data()` on metadata carrying `anchor`. +- Seeded survival output differs from the legacy function's; regenerate pinned expected values once. +- NEWS: feature entry; removal of the v0.2.0 known-issue entry with a migration note; the D5 behaviour change stated explicitly. + +## Future directions + +- **Sidecar overlay (v0.6, separate ADR).** A MockData-owned table keyed by variable, joined onto `variables` before spec construction, so all extension columns can live outside recodeflow files. The legacy `mock_data_config.csv` readers from v0.2 are the starting point to reconcile or retire. +- **Structural (causal) generation.** Out of scope. D1 and D7 keep one dependency field and one ordering mechanism for all derived variables, which is the base a DAG-driven stage would need. From c9b5ebabc2150c41a69fa4b5d371ff61d7476a43 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Thu, 24 Sep 2026 06:08:20 -0400 Subject: [PATCH 02/16] Accept the v0.5 survival-dates ADR with D5 and D8 ratified (#40) Maintainer ratified D5 (contamination after the consistency rules) and D8 (is.na in the formula allow-list) on 2026-09-24. Two adjustments keep future causal extensions open at no cost: survival dates are drawn in dependency order with position tie-breaking (identical to position order for every v0.5 spec), so chained dates later need only a validator change; and the ADR now assesses exposure-dependent hazards against the design, recording the one contained structural change they would need (merging the survival and formula stages into one dependency-ordered pass when a hazard depends on a derived exposure). Also records a verified semantic of the #39 stage order that D8 makes visible: derived columns describe true values, while postprocess contaminates each column independently, so a derived status can sit beside a date shown as missing. Documented as deliberate and pinned by tests. --- development/adr/v05-survival-dates.md | 36 +++++++++++++++++++-------- 1 file changed, 26 insertions(+), 10 deletions(-) diff --git a/development/adr/v05-survival-dates.md b/development/adr/v05-survival-dates.md index 0921a5a..4f3571a 100644 --- a/development/adr/v05-survival-dates.md +++ b/development/adr/v05-survival-dates.md @@ -1,6 +1,6 @@ # ADR: v0.5 Metadata-Driven Survival Dates -**Status**: PROPOSED 2026-09-22. Design agreed with the maintainer in conversation (scope, fidelity policy, architecture, metadata convention, semantics). Two points need explicit ratification before implementation: D5 (contamination after the consistency rules) and D8 (adding `is.na` to the formula allow-list). +**Status**: ACCEPTED 2026-09-24. Design agreed with the maintainer in conversation on 2026-09-22 (scope, fidelity policy, architecture, metadata convention, semantics); D5 and D8 ratified on 2026-09-24. **Date**: 2026-09-22 **Decision owner**: MockData maintainers **Issue**: #40 (also advances #23, #17) @@ -14,10 +14,10 @@ | D2 | Scope | **Full parity** with `create_wide_survival_data()`: entry, event, death, loss to follow-up, administrative censoring, and both consistency rules | | D3 | Metadata convention | Two columns on `variables.csv` beside `followup_min`/`followup_max`/`event_prop`: `anchor` (required) and `censored_by` (optional) | | D4 | Fidelity | **Exact port** of the legacy statistics and rules; concerns are filed as issues, not fixed in #40 | -| D5 | Contamination order | Consistency rules run on clean dates; missing codes and garbage are applied afterwards by postprocess (**ratify**) | +| D5 | Contamination order | Consistency rules run on clean dates; missing codes and garbage are applied afterwards by postprocess | | D6 | Stage and seed | Pipeline becomes baseline, survival, formulas, postprocess. `.MOCK_STAGES` gains `survival = 4L` (appended; no existing index moves) | | D7 | Dependencies | Survival dates record `depends_on = c(anchor, censored_by)`, the field formulas already use; one ordering utility serves every derived stage | -| D8 | Formula allow-list | Add `is.na`, so survival status and time can be derived with `mockFormula` (**ratify**; amends the #39 ADR's D3) | +| D8 | Formula allow-list | Add `is.na`, so survival status and time can be derived with `mockFormula` (amends the #39 ADR's D3) | | D9 | Legacy path | `create_wide_survival_data()` warns once as deprecated and is kept as the characterization reference | ## Context @@ -89,7 +89,7 @@ The legacy statistics and rules are ported verbatim: 3. Date = anchor + follow-up days for events; `NA` otherwise. 4. After all survival dates are drawn: any date earlier than its anchor becomes `NA`; then, for each variable with `censored_by`, rows where the censoring date is earlier become `NA`. -Survival dates are drawn in spec `position` order. No ordering among them is needed, because every anchor is a baseline date. +Survival dates are drawn in dependency order (D7), ties broken by spec `position`. In v0.5 every anchor is a baseline date, so this equals position order and changes no output. It keeps chained dates (a survival date anchored on another, such as an event whose hazard changes at an exposure-change date) a validator-only change later. Output cannot be bit-identical to `create_wide_survival_data()`, because the RNG streams differ (#38). Parity is therefore tested structurally and distributionally (Test strategy). Two further differences are inherent in the pipeline: @@ -98,11 +98,13 @@ Output cannot be bit-identical to `create_wide_survival_data()`, because the RNG | Missing codes fill the last `n - n_valid` rows; events are split across the first `n_valid` rows (`R/create_date_var.R:295,417`) | Postprocess assigns missing codes at random rows | Missingness belongs to postprocess in v0.4. Proportions match in expectation | | Garbage is applied inside `create_date_var()`, before the rules, so before-entry garbage is erased | Rules run first; garbage afterwards (D5) | Stage order | -### D5: Contamination after the consistency rules (ratify) +### D5: Contamination after the consistency rules The consistency rules produce clean, analysis-ready dates. Missing codes and garbage are applied afterwards, by postprocess, as for every other variable. This matches #39, where formulas are computed on clean baseline values before contamination. -It changes one behaviour. In the legacy engine, garbage that places a date before entry is set to `NA` by the before-entry rule; `test-survival-garbage-deprecation.R:147` acknowledges this. The garbage tutorial states that survival garbage exists to create temporal violations such as death before entry, for testing cleaning pipelines. Under D5 those violations survive, so the pipeline does what the tutorial describes. This is a change of stage order, not of any rule, but it alters legacy output and so needs explicit ratification under D4. +It changes one behaviour. In the legacy engine, garbage that places a date before entry is set to `NA` by the before-entry rule; `test-survival-garbage-deprecation.R:147` acknowledges this. The garbage tutorial states that survival garbage exists to create temporal violations such as death before entry, for testing cleaning pipelines. Under D5 those violations survive, so the pipeline does what the tutorial describes. This is a change of stage order, not of any rule. Ratified 2026-09-24. + +One consequence: the rules use the true dates, and postprocess may then give a missing code to a date that censored another. The output can therefore show an event censored by a death that is itself recorded as missing. This mirrors real data, where a death is known to the registry but missing from the analysis file, and it is pinned by a test so it stays deliberate. ### D6: Stage and seed @@ -112,9 +114,11 @@ It changes one behaviour. In the legacy engine, garbage that places a date befor Survival variables set `depends_on = c(anchor, censored_by)`. `.order_formula_variables()` (`R/mock_spec_formula.R:88`) is generalized to order any derived type by `depends_on`, and the formula stage uses the generalized version. The purpose is forward compatibility: a later structural-equation stage, whether in MockData or a separate package, would read a single dependency field from the spec and add a type rather than a mechanism. -### D8: Add `is.na` to the formula allow-list (ratify) +### D8: Add `is.na` to the formula allow-list + +The #39 allow-list has no missingness test, so a formula cannot derive an event indicator from a survival date. With `is.na` added, users can write, for example, `as.integer(!is.na(death_date))` for status and `as.numeric(pmin(death_date, admin_censor_date, na.rm = TRUE) - interview_date)` for follow-up time in days. The #39 ADR permits widening the list on concrete need; this is one. `is.na` has no side effects and cannot reach outside the sandbox. Ratified 2026-09-24. -The #39 allow-list has no missingness test, so a formula cannot derive an event indicator from a survival date. With `is.na` added, users can write, for example, `as.integer(!is.na(death_date))` for status and `as.numeric(pmin(death_date, admin_censor_date, na.rm = TRUE) - interview_date)` for follow-up time in days. The #39 ADR permits widening the list on concrete need; this is one. `is.na` has no side effects and cannot reach outside the sandbox. +Derived columns describe the true generated values. The #39 stage order computes formulas before postprocess, and postprocess contaminates each column independently: garbage only replaces existing values (`R/mock_spec_postprocess.R:255`), but missing codes may land on any row (`:192`). So a derived `status` of 1 can sit beside a `death_date` shown as missing, and a derived follow-up time reflects the true date where the observed one is garbage. (Verified with #39 code: in 1,000 rows, all 300 rows where a source column shows its missing code keep the formula value computed from the true value.) For testing a cleaning pipeline this is useful, because the derived columns are the answer key. For an analysis-ready teaching dataset it is not, and v0.5 does not offer the alternative; see Future directions. ### D9: Legacy path @@ -157,6 +161,7 @@ Found while preparing this design. Each is reproduced exactly by the port. - New exports: `generate_survival_dates()`, `mock_survival()`, `mock_spec_survival()`. Additions to `_pkgdown.yml` in the same PR. - The formula allow-list grows by one function (D8). - Specs without survival variables: output unchanged. +- Derived columns (formulas, survival status or time) describe true values; contamination is applied to each column independently (D8). Documented in the formula and survival vignettes. ## Test strategy @@ -164,7 +169,8 @@ Found while preparing this design. Each is reproduced exactly by the port. 2. **Validator and adapter.** One failing case, with its message, for each rule in "Validation"; the missing-anchor error. 3. **Generation.** Event proportion within two-sided binomial bounds for each distribution (#23's direction); every non-`NA` date at or after its anchor; `censored_by` sets `NA` exactly where the censoring date is earlier and nowhere else; typed zero-row output for `n = 0`; skipping the stage makes postprocess fail on the missing column. 4. **Contract.** Frozen-stage test extended with `survival = 4L`; pinned reference values for one survival spec; all existing pins untouched. -5. **End to end.** `create_mock_data()` on the amended minimal example produces all five dates with no fallback; the simstudy backend produces the same survival columns; a `mockFormula` derives status and follow-up time from them (D8); D5 is pinned by a test showing before-entry garbage survives. +5. **End to end.** `create_mock_data()` on the amended minimal example produces all five dates with no fallback; the simstudy backend produces the same survival columns; a `mockFormula` derives status and follow-up time from them (D8); D5 is pinned by a test showing before-entry garbage survives, and by a test showing an event censored by a death that postprocess then records as missing. +6. **Draw order.** A test pins that survival draws follow dependency order with position tie-breaking, so a later change to either is visible. ## Migration @@ -176,4 +182,14 @@ Found while preparing this design. Each is reproduced exactly by the port. ## Future directions - **Sidecar overlay (v0.6, separate ADR).** A MockData-owned table keyed by variable, joined onto `variables` before spec construction, so all extension columns can live outside recodeflow files. The legacy `mock_data_config.csv` readers from v0.2 are the starting point to reconcile or retire. -- **Structural (causal) generation.** Out of scope. D1 and D7 keep one dependency field and one ordering mechanism for all derived variables, which is the base a DAG-driven stage would need. +- **Observed-value derivation.** An option to compute formulas from contaminated values, so derived status and time agree with what the output shows (D8). An ordering option on the formula stage; no architectural change. +- **Structural (causal) generation.** Out of scope. The maintainer's motivating case is exposure-dependent survival, such as smoking changing the hazard. Assessed against this design on 2026-09-24: + + | Extension | What it needs | Change to the base? | + |---|---|---| + | Hazard varies with a baseline exposure | Survival parameters (for example a hazard ratio) given as sandboxed expressions over generated columns; `depends_on` records the exposure. The ported event assignment is count-based (`floor(n * event_prop)`), so per-person risk needs a per-person draw as a second path | No; additive | + | Hazard varies with a derived exposure (a formula variable) | Formula, then survival, then formula. The fixed stage order in D6 (survival before formulas) cannot express this; survival and formula variables would share one dependency-ordered pass | Yes, contained. D7 prepares it, and because the v0.5 validator forbids survival dates that depend on formulas, merging the passes shifts no existing seeded output | + | Exposure changes during follow-up | The change date is itself an anchored date; the event hazard is piecewise around it. Needs chained survival dates | No; D4 draws in dependency order, so only the validator rule changes | + | Several rows per person (counting-process data) | A second output shape | Yes; MockData produces one row per person. Wide data with change dates covers most analyses | + + D1 and D7 keep one dependency field and one ordering mechanism for all derived variables, which is the base a DAG-driven stage would need. From 097841cac4eefc053b8df5cfd8b851a3bee4b45e Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Thu, 24 Sep 2026 06:27:34 -0400 Subject: [PATCH 03/16] Correct the survival-dates ADR from plan preparation (#40) Writing the implementation plan against the code exposed four points where the accepted ADR was wrong or silent: - Draw order: censored_by is part of depends_on, so dependency order draws a censoring date before the date it censors; the claim that this equals spec order was false. Each survival variable is now drawn and then has its own rules applied, in dependency order, which reproduces the legacy rule results and keeps value = f(parents) per variable. - Whole days: legacy dates are whole days because create_date_var() round- trips through character; the port floors follow-up days explicitly. - Direct API: mock_survival() is dropped, because a one-variable spec can never validate (its anchor must be in the same spec). - D10 (new): when the legacy generator runs on metadata carrying anchor, create_mock_data() stops and names the survival variables instead of dropping them. Survival dates also accept sourceFormat "analysis" only. Concerns now cite their issues (#54-#57, #15), plus one found while reading for the plan: exponential survival dates ignore `rate` (#56). --- development/adr/v05-survival-dates.md | 36 ++++++++++++++++----------- 1 file changed, 22 insertions(+), 14 deletions(-) diff --git a/development/adr/v05-survival-dates.md b/development/adr/v05-survival-dates.md index 4f3571a..9f7c0e7 100644 --- a/development/adr/v05-survival-dates.md +++ b/development/adr/v05-survival-dates.md @@ -19,6 +19,7 @@ | D7 | Dependencies | Survival dates record `depends_on = c(anchor, censored_by)`, the field formulas already use; one ordering utility serves every derived stage | | D8 | Formula allow-list | Add `is.na`, so survival status and time can be derived with `mockFormula` (amends the #39 ADR's D3) | | D9 | Legacy path | `create_wide_survival_data()` warns once as deprecated and is kept as the characterization reference | +| D10 | Legacy generator and `anchor` | If the legacy generator runs on metadata carrying `anchor`, `create_mock_data()` stops and names the survival variables instead of dropping them | ## Context @@ -86,10 +87,10 @@ The legacy statistics and rules are ported verbatim: 1. `n_events = floor(n * event_prop)`; the event indicator is shuffled with `sample()`. 2. Follow-up days for events, by `distribution`. **Uniform:** `runif(n_events, followup_min, followup_max)`. **Exponential:** `rexp(n_events, rate = 1 / ((followup_max - followup_min) / 3)) + followup_min`, capped at `followup_max`. **Gompertz:** `(1/shape) * log(1 - (shape/rate) * log(1 - u))` with `u ~ runif`, clamped to the window; `shape` defaults to 0.1 and `rate` to 0.0001. -3. Date = anchor + follow-up days for events; `NA` otherwise. -4. After all survival dates are drawn: any date earlier than its anchor becomes `NA`; then, for each variable with `censored_by`, rows where the censoring date is earlier become `NA`. +3. Date = anchor + `floor(follow-up days)` for events; `NA` otherwise. The legacy engine produces whole days because `create_date_var()` converts dates to character and back (`R/create_date_var.R:479`), which floors; the port floors explicitly. +4. Any date earlier than its anchor becomes `NA`; then, if the variable has `censored_by`, rows where the censoring date is earlier become `NA`. -Survival dates are drawn in dependency order (D7), ties broken by spec `position`. In v0.5 every anchor is a baseline date, so this equals position order and changes no output. It keeps chained dates (a survival date anchored on another, such as an event whose hazard changes at an exposure-change date) a validator-only change later. +Survival variables are processed one at a time in dependency order over `depends_on` (D7), using the #39 ordering: repeated passes in spec order, each taking the variables whose dependencies are already placed. Each variable is drawn and then has its own rules applied, so a date named in another's `censored_by` is final before it censors anything. For the legacy rule set this gives the legacy results. The draw order differs from spec order where `censored_by` points forward (in the minimal example, `death_date` is drawn before `primary_event_date`); that has no comparability cost, because the RNG streams differ from the legacy engine's anyway. Mutual `censored_by` references are reported as a dependency cycle. Processing in dependency order also keeps chained dates (a survival date anchored on another, such as an event whose hazard changes at an exposure-change date) a validator-only change later. Output cannot be bit-identical to `create_wide_survival_data()`, because the RNG streams differ (#38). Parity is therefore tested structurally and distributionally (Test strategy). Two further differences are inherent in the pipeline: @@ -124,6 +125,10 @@ Derived columns describe the true generated values. The #39 stage order computes `create_wide_survival_data()` is already tagged `@keywords deprecated`. v0.5 adds a one-time deprecation warning pointing to metadata-driven generation. The function is retained unchanged: it is the reference for the characterization tests that pin the rules before the port. +### D10: Legacy generator and `anchor` + +`create_mock_data()` uses the legacy generator when `validate = FALSE`, when `variable_details` is `NULL`, when only `variable_details` has a `databaseStart` column, or when another variable uses a feature the v0.4 pipeline does not support. The legacy dispatcher cannot generate survival dates from metadata; today it warns and drops them. With `anchor` in the metadata, that would silently produce partial survival data. Instead, the legacy path stops, names the survival variables, lists the four conditions that select it, and suggests `verbose = TRUE` to see which applied. This follows the no-backward-compatibility constraint: a loud error that names the fix, rather than plausible-looking partial output. + ## Validation, direct API and diagnostics **Validator.** A `survival` branch in `validate_mock_spec()` beside the `date` and `formula` branches (`R/mock_spec.R:968`): @@ -134,31 +139,34 @@ Derived columns describe the true generated values. The #39 stage order computes - `event_prop` in [0, 1] - `distribution` one of `uniform`, `exponential`, `gompertz`; `shape` and `rate` positive when supplied - `rtype` is `date` +- `sourceFormat` is `analysis` (other formats are not supported for survival dates in v0.5) `generate_survival_dates()` strict-validates at entry, like `evaluate_mock_formulas()`, and stops if an anchor column is absent from `data`. **Adapter.** `.recodeflow_variable_kind()` (`R/mock_spec_recodeflow.R:117`) returns `survival` for a date row with a non-blank `anchor`. Construction reads `anchor` and `censored_by` beside the existing `followup_*`, `event_prop`, `rate` and `shape` reads. The `recStart` range of a survival date is not used for generation, as in the legacy engine. -**Direct API.** `mock_survival()` returns a one-variable `mock_spec`; `mock_spec_survival()` returns a composable variable. Arguments: `name`, `anchor`, `followup_min`, `followup_max`, `event_prop`, `distribution = "uniform"`, `censored_by = NULL`, `shape = NULL`, `rate = NULL`, plus the usual missing-code, garbage, provenance and model-hint arguments. +**Direct API.** `mock_spec_survival()` returns a composable variable, used with `mock_spec()` beside its anchor. Arguments: `name`, `anchor`, `followup_min`, `followup_max`, `event_prop`, `distribution = "uniform"`, `censored_by = NULL`, `shape = NULL`, `rate = NULL`, `source_format = "analysis"`, plus the usual missing-code, garbage, provenance and model-hint arguments. There is no one-variable `mock_survival()` counterpart to `mock_formula()`: a spec holding only a survival variable can never validate, because its anchor must be in the same spec. **Diagnostics.** Following #39's D5 (`R/mock_spec_postprocess.R:35`): `derived = TRUE`, `anchor`, `censored_by`, `depends_on`, and `n_events`, the number of non-`NA` dates after the consistency rules and before postprocess. -## Concerns to file as issues (not fixed in #40) +## Concerns filed as issues (not fixed in #40) -Found while preparing this design. Each is reproduced exactly by the port. +Found while preparing this design and filed on 2026-09-24. Each is reproduced exactly by the port. -1. **Gompertz output is degenerate with the packaged parameters.** With `shape = 0.1` and `rate = 1e-04`, time in days, the inverse-CDF never exceeds about 94 days (median 65). In the minimal example every non-garbage `death_date` therefore clamps to `followup_min`, exactly 365 days after entry (388 of 400 deaths in a 2,000-row legacy run; the other 12 are the 3 per cent high garbage). Every non-garbage `primary_event_date` falls within about 94 days despite a 15-year window. No death precedes an event, so the competing-risk rule never fires. The survival tutorial is built on this fixture. Whether the fault lies in the parameterization, the time scale, or the fixture values needs a methodological decision before the tutorial is rewritten around `create_mock_data()`. -2. **Administrative censoring is drawn per person.** `admin_censor_date` has `event_prop = 1.0` and a follow-up window, so each person gets a random date between one and 20 years after entry (1,762 distinct values in 2,000 rows). Its `variable_details` row specifies a fixed date, 2024-12-31, which is ignored. Administrative censoring is usually a fixed study end date. -3. **Declared date missingness is ignored.** Every missing row for the minimal example's dates is `else` with `NA::b` and a proportion (0.05 for `death_date`). Neither path applies it: the v0.4 adapter skips `else` rows (`R/mock_spec_recodeflow.R:232`), and the legacy `death_date` is exactly 20.0 per cent non-missing, its `event_prop`. Belongs with #15. -4. **The documentation overstates censoring.** `create_wide_survival_data()` documents that "observation ends at min(event, death, ltfu, admin_censor)" (`:61`), but the code applies only death before event. Under D3, users encode additional rules with `censored_by`. -5. **Positional missing-code placement** in `create_date_var()`: the last rows are the missing ones, so row order carries meaning. -6. **Inert range rows.** The `recStart` range on a survival date's `variable_details` row (for example `[2002-01-01,2021-01-01]`) never bounds anything, which invites the reader to think it does. +1. **Gompertz output is degenerate with the packaged parameters** (#54). With `shape = 0.1` and `rate = 1e-04`, time in days, the inverse-CDF never exceeds about 94 days (median 65). In the minimal example every non-garbage `death_date` therefore clamps to `followup_min`, exactly 365 days after entry (388 of 400 deaths in a 2,000-row legacy run; the other 12 are the 3 per cent high garbage). Every non-garbage `primary_event_date` falls within about 94 days despite a 15-year window. No death precedes an event, so the competing-risk rule never fires. The survival tutorial is built on this fixture. Whether the fault lies in the parameterization, the time scale, or the fixture values needs a methodological decision before the tutorial is rewritten around `create_mock_data()`. +2. **Administrative censoring is drawn per person** (#55). `admin_censor_date` has `event_prop = 1.0` and a follow-up window, so each person gets a random date between one and 20 years after entry (1,762 distinct values in 2,000 rows). Its `variable_details` row specifies a fixed date, 2024-12-31, which is ignored. Administrative censoring is usually a fixed study end date. +3. **Declared date missingness is ignored** (evidence added to #15). Every missing row for the minimal example's dates is `else` with `NA::b` and a proportion (0.05 for `death_date`). Neither path applies it: the v0.4 adapter skips `else` rows (`R/mock_spec_recodeflow.R:232`), and the legacy `death_date` is exactly 20.0 per cent non-missing, its `event_prop`. Belongs with #15. +4. **The documentation overstates censoring** (#56). `create_wide_survival_data()` documents that "observation ends at min(event, death, ltfu, admin_censor)" (`:61`), but the code applies only death before event. Under D3, users encode additional rules with `censored_by`. +5. **Positional missing-code placement** (#56) in `create_date_var()`: the last rows are the missing ones, so row order carries meaning. +6. **Inert range rows** (#56). The `recStart` range on a survival date's `variable_details` row (for example `[2002-01-01,2021-01-01]`) never bounds anything, which invites the reader to think it does. +7. **Exponential survival dates ignore `rate`** (#56). The generator derives its rate from the window, `1 / ((followup_max - followup_min) / 3)` (`R/create_date_var.R:328`), while `reference-config.qmd` says the exponential distribution requires `rate`. ## Consequences - `create_mock_data()` generates complete survival data from metadata, closing the NEWS known issue from v0.2.0. - Survival metadata that previously generated silently wrong output now either generates correctly (with `anchor`) or fails validation with a message naming the fix. -- New exports: `generate_survival_dates()`, `mock_survival()`, `mock_spec_survival()`. Additions to `_pkgdown.yml` in the same PR. +- New exports: `generate_survival_dates()` and `mock_spec_survival()`. Additions to `_pkgdown.yml` in the same PR. +- The packaged minimal example now generates entirely through the v0.4 pipeline. Its two Gompertz dates were its only legacy-fallback triggers, so every vignette and test that runs `create_mock_data()` on it sees different values. - The formula allow-list grows by one function (D8). - Specs without survival variables: output unchanged. - Derived columns (formulas, survival status or time) describe true values; contamination is applied to each column independently (D8). Documented in the formula and survival vignettes. @@ -181,7 +189,7 @@ Found while preparing this design. Each is reproduced exactly by the port. ## Future directions -- **Sidecar overlay (v0.6, separate ADR).** A MockData-owned table keyed by variable, joined onto `variables` before spec construction, so all extension columns can live outside recodeflow files. The legacy `mock_data_config.csv` readers from v0.2 are the starting point to reconcile or retire. +- **Sidecar overlay (v0.6, separate ADR; #57).** A MockData-owned table keyed by variable, joined onto `variables` before spec construction, so all extension columns can live outside recodeflow files. The legacy `mock_data_config.csv` readers from v0.2 are the starting point to reconcile or retire. - **Observed-value derivation.** An option to compute formulas from contaminated values, so derived status and time agree with what the output shows (D8). An ordering option on the formula stage; no architectural change. - **Structural (causal) generation.** Out of scope. The maintainer's motivating case is exposure-dependent survival, such as smoking changing the hazard. Assessed against this design on 2026-09-24: From 0da65f6c92cadbc7e1418402b76fd55fe5766fb0 Mon Sep 17 00:00:00 2001 From: Doug Manuel Date: Thu, 24 Sep 2026 09:23:22 -0400 Subject: [PATCH 04/16] Add the #40 survival-dates implementation plan Seven tasks: characterize the legacy engine; the survival spec layer (constructor, validators, shared dependency ordering); the post-baseline survival stage with backend skips, diagnostics and an appended seed sub-stream; recodeflow and orchestrator wiring with the legacy guard; is.na in the formula allow-list; deprecation of create_wide_survival_data(); and documentation with full verification. The characterization tests and the stage's draw, rules and ordering were prototyped against the code before writing (5/5 and 11/11 passing), and the survival stage's pinned values were computed independently by emulating the seed primitive for sub-stream 4. Based on v05-survival-dates at 2548b51, which includes the #58 prerequisite (PR #59). --- .../plans/2026-09-24-survival-dates.md | 2593 +++++++++++++++++ 1 file changed, 2593 insertions(+) create mode 100644 development/plans/2026-09-24-survival-dates.md diff --git a/development/plans/2026-09-24-survival-dates.md b/development/plans/2026-09-24-survival-dates.md new file mode 100644 index 0000000..9146e7e --- /dev/null +++ b/development/plans/2026-09-24-survival-dates.md @@ -0,0 +1,2593 @@ +# Metadata-Driven Survival Dates Implementation Plan (#40) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** `create_mock_data()` generates entry, event, death, loss-to-follow-up and administrative-censoring dates from metadata alone, by computing `type = "survival"` variables from their anchor dates in a post-baseline stage. + +**Architecture:** A new spec type `survival` (rtype `date`). Both backends skip derived types (formula, survival) in baseline generation. A new exported `generate_survival_dates(data, spec, seed)` processes survival variables one at a time in dependency order (draw, then the before-anchor and `censored_by` rules) under `.with_mock_seed(stage = "survival")`. `create_mock_data()` runs baseline → survival → formulas → postprocess. The recodeflow adapter builds survival variables from `variables.csv` rows that set `anchor`. The legacy engine is unchanged apart from a deprecation warning, and the legacy dispatcher stops on anchored metadata. + +**Tech stack:** base R (no new dependency), testthat 3e (3.2.3), roxygen2. + +**Spec:** [development/adr/v05-survival-dates.md](../adr/v05-survival-dates.md) (ACCEPTED 2026-09-24, D1–D10). Issues: #40; concerns #54, #55, #56, #57 and #15; prerequisite #58 (PR #59). + +**Base:** branch `v05-survival-dates` at `2548b51`: `v05-formula-evaluator` (#52, which carries #51) + the ADR commits + PR #59 (#58 range missing codes and minimal-example fixture corrections) merged in. Do not start until `git log --oneline -1` shows `2548b51` or a descendant. + +**Prototype evidence (2026-09-24):** the Task 1 characterization tests were run against the legacy engine and pass; the Task 3 draw routine, rules and ordering were prototyped and pass 11 parity checks (competing risk both directions, exact event count, whole days inside the window, Gompertz #54 parity, exponential bounds, `n = 0`, flooring to zero events, `NA` anchors, draw order, cycle error). The Task 3 pinned values were computed independently by emulating `.with_mock_seed()` for stage index 4, not captured from the implementation. + +## Global Constraints + +- **Zero seeded-output change for specs without survival variables.** The #48 pins in `tests/testthat/test-seed-contract.R` (postprocess stage pin, pinned reference values) must pass untouched. If one fails, the code is wrong: fix the code, never the pin. The only permitted edit to that file's existing tests is extending the frozen-stage vector (Task 3) and dropping anchored rows in the two legacy-path tests (Task 4). +- **Survival draws use only `.MOCK_STAGES["survival"]`**, appended as `4L`. Never renumber indices 0–3. +- **Exact port (ADR D4).** The statistics and rules in Task 3 Step 3 come from `R/create_date_var.R:295-346` and `R/create_wide_survival_data.R:347,363`. Do not fix anything the ADR lists under Concerns (#54, #55, #56, #15), even where it looks wrong. Tests that pin those behaviours say so in a comment. +- **recodeflow semantics untouched.** `anchor` and `censored_by` are MockData extension columns on `variables.csv`. +- **CSV fixtures change only through R data-frame operations** (`read.csv`/`write.csv(row.names = FALSE, na = "")`), never sed, awk or hand edits (user CLAUDE.md). `write.csv` reproduces the minimal example byte for byte (verified). +- **No backward-compatibility obligation (0.x)**, but every new error names the fix, and every new error message is asserted by a test with a regex on a distinctive phrase. +- **Commits:** imperative subject in the repo's style with a `(#40)` suffix, wrapped body; no AI attribution (user CLAUDE.md overrides). Commit locally; do not push. +- **Test counts:** report all testthat columns, including `error`. Summing only `failed` hid #50. Use the snippet in Task 1 Step 1. +- **Environment:** the renv library lives under `~/Library/Caches` and macOS purges it. If `library(devtools)` fails, run `SDKROOT=$(xcrun --sdk macosx --show-sdk-path) Rscript -e 'renv::restore(prompt = FALSE)'` first. `devtools::check()` fails on this Mac's toolchain; use `R CMD build` then `R CMD check --as-cran --no-manual` on the tarball (Task 7). +- **Style for prose (NEWS, vignettes, roxygen):** Canadian Press spelling with -ize, sentence-case headings, "per cent" in running prose, no Unicode dashes in source (`~/github/ai-infrastructure/context/global_style.md`). + +## Review Focus + +1. **Legacy route with anchored metadata** (`validate = FALSE`, or another variable forcing the fallback): expect an error that names the survival variables. Test: Task 4, `legacy generator stops on anchored metadata instead of dropping survival dates`. +2. **Anchor not enabled** (role filtering drops the entry date but keeps its survival dates): expect a validation error naming the anchor, surfaced by `create_mock_data()`. Test: Task 4, `an anchor that is not enabled fails with a message naming it`. +3. **Character anchor column in standalone use** (a user reads dates from CSV and calls `generate_survival_dates()` directly): expect an error naming the column and its class. Test: Task 3, `generate_survival_dates rejects a non-Date anchor column`. +4. **Small n with a small event_prop** (n = 3, event_prop = 0.2, so `floor()` gives zero events): expect an all-`NA` `Date` column and `n_events = 0` in diagnostics, with no error or warning. Test: Task 3, `zero events from flooring is an all-NA column, not an error`. +5. **Mutual `censored_by`** (A censored by B, B censored by A): expect a dependency-cycle error at validation. Test: Task 2, `mutual censored_by is reported as a dependency cycle`. + +--- + +### Task 1: Characterize the legacy survival engine + +Pins what the port must reproduce, before any code changes. These tests describe current legacy behaviour and stay green throughout; the legacy engine is not changed (Task 6 adds a warning, which the Task 6 test helper pre-empts). + +**Files:** +- Create: `tests/testthat/test-survival-characterization.R` + +**Interfaces:** +- Consumes: `create_wide_survival_data()` (unchanged) +- Produces: nothing used by later tasks; the scenarios are mirrored against the new stage in Task 3. + +- [ ] **Step 1: Record the baseline** + +```bash +cd /Users/dmanuel/github/mock-data && Rscript -e ' +df <- as.data.frame(devtools::test(reporter = "silent", stop_on_failure = FALSE)) +cat(sprintf("FAIL %d | ERROR %d | WARN %d | SKIP %d | PASS %d\n", + sum(df$failed), sum(df$error), sum(df$warning), sum(df$skipped), sum(df$passed))) +bad <- df[df$failed > 0 | df$error, c("file", "test")]; if (nrow(bad)) print(bad)' +``` + +Expected at `2548b51` with simstudy installed: `FAIL 0 | ERROR 0 | WARN 35 | SKIP 3 | PASS 811`. Record the actual line; every later task compares against it. + +- [ ] **Step 2: Write the characterization tests** + +```r +# tests/testthat/test-survival-characterization.R +# Characterization of the legacy survival engine (create_wide_survival_data()), +# pinned before the #40 port. ADR: development/adr/v05-survival-dates.md (D4). +# These describe current behaviour, including behaviour filed as concerns +# (#54, #56); they are the reference the metadata-driven stage must match. + +legacy_fixture <- function(event_window, death_window, + event_prop = 1, death_prop = 1) { + list( + variables = data.frame( + variable = c("entry", "event", "death"), + variableType = c("Date", "Date", "Date"), + rType = c("date", "date", "date"), + role = c("enabled", "enabled", "enabled"), + distribution = c("uniform", "uniform", "uniform"), + followup_min = c(NA, event_window[1], death_window[1]), + followup_max = c(NA, event_window[2], death_window[2]), + event_prop = c(NA, event_prop, death_prop), + sourceFormat = c("analysis", "analysis", "analysis"), + stringsAsFactors = FALSE + ), + variable_details = data.frame( + variable = c("entry", "event", "death"), + recStart = c("[2001-01-01,2001-12-31]", + "[2001-01-01,2040-12-31]", + "[2001-01-01,2040-12-31]"), + recEnd = c("copy", "copy", "copy"), + proportion = c(1, 1, 1), + stringsAsFactors = FALSE + ) + ) +} + +run_legacy <- function(fx, n = 200, seed = 1) { + suppressWarnings(create_wide_survival_data( + var_entry_date = "entry", + var_event_date = "event", + var_death_date = "death", + databaseStart = "test", + variables = fx$variables, + variable_details = fx$variable_details, + n = n, + seed = seed + )) +} + +days_from_entry <- function(result, column) { + as.numeric(result[[column]] - result$entry) +} + +test_that("legacy: an event later than death is set to NA (competing risk)", { + result <- run_legacy(legacy_fixture(c(1000, 2000), c(0, 500))) + expect_true(all(is.na(result$event))) + expect_false(any(is.na(result$death))) +}) + +test_that("legacy: an event earlier than death is kept", { + result <- run_legacy(legacy_fixture(c(0, 500), c(1000, 2000))) + expect_false(any(is.na(result$event))) +}) + +test_that("legacy: exactly floor(n * event_prop) rows receive an event", { + result <- run_legacy(legacy_fixture(c(100, 200), c(100, 200), + event_prop = 0.3, death_prop = 0), + n = 200) + expect_identical(sum(!is.na(result$event)), 60L) + expect_true(all(is.na(result$death))) +}) + +test_that("legacy: follow-up days are whole days inside the window", { + result <- run_legacy(legacy_fixture(c(100, 200), c(5000, 6000))) + days <- days_from_entry(result, "event") + days <- days[!is.na(days)] + expect_true(all(days == floor(days))) + expect_true(all(days >= 100 & days <= 200)) +}) + +test_that("legacy: on the minimal example no date precedes the entry date", { + vars <- system.file("extdata", "minimal-example", "variables.csv", package = "MockData") + dets <- system.file("extdata", "minimal-example", "variable_details.csv", package = "MockData") + if (!nzchar(vars) || !nzchar(dets)) skip("minimal-example fixtures not installed") + variables <- read.csv(vars, stringsAsFactors = FALSE, check.names = FALSE) + variable_details <- read.csv(dets, stringsAsFactors = FALSE, check.names = FALSE) + result <- suppressWarnings(create_wide_survival_data( + var_entry_date = "interview_date", + var_event_date = "primary_event_date", + var_death_date = "death_date", + var_ltfu = "ltfu_date", + var_admin_censor = "admin_censor_date", + databaseStart = "minimal-example", + variables = variables, + variable_details = variable_details, + n = 2000, + seed = 1 + )) + for (column in c("primary_event_date", "death_date", "ltfu_date", "admin_censor_date")) { + ok <- !is.na(result[[column]]) + expect_true(all(result[[column]][ok] >= result$interview_date[ok]), info = column) + } + # #54: with the packaged Gompertz parameters every non-garbage death lands + # exactly at followup_min (365 days). Garbage deaths are 2025 or later. + # Update this expectation deliberately when #54 is resolved. + days <- as.numeric(result$death_date - result$interview_date) + clean <- !is.na(days) & result$death_date < as.Date("2025-01-01") + expect_true(any(clean)) + expect_true(all(days[clean] == 365)) +}) +``` + +- [ ] **Step 3: Run them — expected PASS (characterization)** + +Run: `Rscript -e 'devtools::load_all(quiet = TRUE); testthat::test_file("tests/testthat/test-survival-characterization.R")'` +Expected: 5 tests, 13 expectations, all PASS. If any fails, stop: the legacy engine differs from what the plan assumes; report rather than editing the test. + +- [ ] **Step 4: Commit** + +```bash +git add tests/testthat/test-survival-characterization.R +git commit -m "Characterize the legacy survival engine before the port (#40)" -m "Pins the create_wide_survival_data() behaviour the metadata-driven stage +must reproduce: the death-censors-event rule in both directions, the exact +floor(n * event_prop) event count, whole-day follow-up inside the window, +no date before entry on the minimal example, and the #54 Gompertz clamp." +``` + +--- + +### Task 2: Survival spec layer + +**Files:** +- Create: `R/mock_spec_survival.R` +- Modify: `R/mock_spec_formula.R:87-126` (generalize ordering) +- Modify: `R/mock_spec.R` (constructor after `mock_spec_formula()`, which ends near line 746; validator branch before the unsupported-type `else` near line 992; spec-level checks in `validate_mock_spec()` after the formula block near line 1046) +- Test: `tests/testthat/test-survival-vars.R` (new) + +**Interfaces:** +- Consumes: `.new_mock_spec_variable()`, `.is_formula_variable()`, `.formula_dependencies()`, `.is_blank()` (defined in `R/mock_spec_recodeflow.R:34`), `` `%||%` `` (`R/mock_spec.R:45`) +- Produces: + - `mock_spec_survival(name, anchor, followup_min, followup_max, event_prop, distribution = "uniform", censored_by = NULL, shape = NULL, rate = NULL, source_format = "analysis", missing_codes = character(0), missing_proportions = numeric(0), garbage_rules = list(), provenance = "direct", model_hint = "native-postprocess")` → `mock_spec_variable` with `type = "survival"`, `rtype = "date"`, fields `anchor`, `censored_by`, `followup_min`, `followup_max`, `event_prop`, `shape`, `rate`, `depends_on = c(anchor, censored_by)` + - `.is_survival_variable(variable)` → logical; `.is_derived_variable(variable)` → TRUE for `formula` or `survival` + - `.survival_dependencies(variable)` → character + - `.validate_survival_variable(variable)` → character vector of errors + - `.validate_survival_referents(spec)` → character vector of errors + - `.order_dependent_variables(names_in_scope, dependencies, label)` → character (stops with `"