Repository navigation
Metadata-driven survival dates in create_mock_data() (#40) - #61
Merged
Merged
Conversation
Design for generating survival dates inside create_mock_data() from metadata, agreed with the maintainer: full parity with create_wide_survival_data() (entry, event, death, LTFU, admin censoring and both consistency rules), exact port of the legacy statistics, and a post-baseline anchored stage modelled on the #39 formula stage rather than a multi-column generator. Metadata gains `anchor` and `censored_by` on variables.csv; the seed contract gains an appended `survival` sub-stream. Two decisions await ratification: applying garbage and missing codes after the consistency rules (D5), and adding is.na to the formula allow-list so status and follow-up time can be derived with mockFormula (D8). Records six concerns found while preparing the design, to be filed as issues. Most consequential: with the packaged Gompertz parameters, 388 of 400 legacy death dates land exactly at followup_min, so the minimal example's competing-risk rule never fires. Context claims were verified against the running code on 2026-09-22.
Maintainer ratified D5 (contamination after the consistency rules) and D8 (is.na in the formula allow-list) on 2026-09-24. Two adjustments keep future causal extensions open at no cost: survival dates are drawn in dependency order with position tie-breaking (identical to position order for every v0.5 spec), so chained dates later need only a validator change; and the ADR now assesses exposure-dependent hazards against the design, recording the one contained structural change they would need (merging the survival and formula stages into one dependency-ordered pass when a hazard depends on a derived exposure). Also records a verified semantic of the #39 stage order that D8 makes visible: derived columns describe true values, while postprocess contaminates each column independently, so a derived status can sit beside a date shown as missing. Documented as deliberate and pinned by tests.
Writing the implementation plan against the code exposed four points where the accepted ADR was wrong or silent: - Draw order: censored_by is part of depends_on, so dependency order draws a censoring date before the date it censors; the claim that this equals spec order was false. Each survival variable is now drawn and then has its own rules applied, in dependency order, which reproduces the legacy rule results and keeps value = f(parents) per variable. - Whole days: legacy dates are whole days because create_date_var() round- trips through character; the port floors follow-up days explicitly. - Direct API: mock_survival() is dropped, because a one-variable spec can never validate (its anchor must be in the same spec). - D10 (new): when the legacy generator runs on metadata carrying anchor, create_mock_data() stops and names the survival variables instead of dropping them. Survival dates also accept sourceFormat "analysis" only. Concerns now cite their issues (#54-#57, #15), plus one found while reading for the plan: exponential survival dates ignore `rate` (#56).
Seven tasks: characterize the legacy engine; the survival spec layer (constructor, validators, shared dependency ordering); the post-baseline survival stage with backend skips, diagnostics and an appended seed sub-stream; recodeflow and orchestrator wiring with the legacy guard; is.na in the formula allow-list; deprecation of create_wide_survival_data(); and documentation with full verification. The characterization tests and the stage's draw, rules and ordering were prototyped against the code before writing (5/5 and 11/11 passing), and the survival stage's pinned values were computed independently by emulating the seed primitive for sub-stream 4. Based on v05-survival-dates at 2548b51, which includes the #58 prerequisite (PR #59).
An external review (Astra) raised four points; two were reproduced against the code and change behaviour: - D8's example derived status and time from different windows. For a death on day 50 and administrative censoring on day 20 it gave status 1 with follow-up 20. Status and time now come from one observation window, ties count as observed events (matching the strict censoring rule), and the plan's tests pin both. In a 400-row check, 114 of 200 generated deaths fell after administrative censoring and are not observed. - Chained censoring gave a wrong observed outcome: with A (day 30) censored by B (day 20) censored by C (day 10), A was reported as observed although observation ended on day 10. v0.5 rejects a censored_by target that is itself censored. The other points are framing, now stated as limits: the three data layers (clean truth, observed data, analysis variables) and how to keep the clean layer; the fixed stage order and the fixed-count event allocation as v0.5 restrictions rather than the definition of survival generation; and a corrected assessment of exposure-dependent extensions, where time-varying exposure is a substantial generator extension and feedback between health states needs a longitudinal mechanism.
Pins the create_wide_survival_data() behaviour the metadata-driven stage must reproduce: the death-censors-event rule in both directions, the exact floor(n * event_prop) event count, whole-day follow-up inside the window, no date before entry on the minimal example, and the #54 Gompertz clamp.
mock_spec_survival() builds a type = "survival" date with anchor, censored_by, follow-up window, event_prop and distribution parameters, and records depends_on = c(anchor, censored_by). validate_mock_spec() checks each variable's parameters and the spec-level referents (anchor is a date in the spec; censored_by is an uncensored survival variable with the same anchor, since chained censoring gives wrong observed outcomes) and reports mutual censored_by as a dependency cycle. The #39 formula ordering becomes a shared .order_dependent_variables(), which formulas and survival dates both use (ADR D7); formula ordering and its messages are unchanged.
Survival variables are drawn from their anchor dates with the legacy statistics ported exactly (fixed floor(n * event_prop) event count, uniform/exponential/Gompertz follow-up in whole days) and processed in dependency order: each is drawn, then dates before the anchor and dates after an earlier censored_by date become NA. Draws use a new survival sub-stream appended to .MOCK_STAGES as index 4, so no existing index or pinned value moves. Both backends now skip derived types (formula and survival) in baseline generation. Postprocess diagnostics mark survival variables as derived with their anchor, censored_by, depends_on and n_events. Tests pin D5 (garbage survives the rules; the rules use true dates) and the survival stream, whose reference values were computed independently of this code.
#40) The adapter builds a survival variable from any date row that sets anchor (reading censored_by and the survival parameters), and rejects a date with survival parameters but no anchor with a message naming the fix. The orchestrator runs baseline, survival, formulas, then postprocess, and no longer treats survival dates as a legacy-fallback trigger. When the legacy generator is selected on anchored metadata, create_mock_data() stops and names the survival variables instead of dropping them (ADR D10). The minimal example gains anchor and censored_by, so it now generates entirely through the v0.4 pipeline. Two legacy-path seed tests drop its survival rows, and the validate = TRUE skipped-summary test reaches the legacy path through detail-level databaseStart instead of an anchor-less survival date, which the adapter now rejects.
Adds is.na to the formula allow-list (survival ADR D8, amending the #39 ADR's D3). Tests derive status and follow-up time from one observation window (a death after administrative censoring is not observed; a death on the censoring date is), and pin that formula columns describe clean truth: a missing code applied to the date afterwards leaves them unchanged.
Warns once per session, pointing to anchor/censored_by metadata and create_mock_data(). The function is otherwise unchanged: it is the reference the characterization tests pin. A test helper marks the warning as shown so the legacy survival tests gain no unrelated warnings.
Rewrites the survival tutorial around anchor metadata and create_mock_data(), including the competing-risk rule, status and follow-up time derived from one observation window, the three data layers (clean truth, observed data, analysis variables), and notes on #54 and #55. Updates the garbage tutorial's survival section, the configuration reference (anchor, censored_by, date distributions, the extension-column placement rule, and worked examples that lacked anchor), advanced topics, design philosophy, the recodeflow-users guide and the migration guide's fallback list. NEWS gains the #40 section and marks the v0.2.0 known issue resolved. The create_mock_data() fallback-mode example ran the minimal example with variable_details = NULL, which the D10 guard now stops because the fixture sets anchor; the example drops the anchored rows first. R CMD check --as-cran: 0 errors, 0 warnings, 2 notes.
The v0.4 wrapper appended "or call create_mock_data() with validate = FALSE to use the legacy generator" to every metadata error, including the new survival ones. Taking that advice either hits the D10 stop (anchored metadata) or silently drops the survival dates (metadata still lacking anchor), which is the behaviour #40 removes. When the metadata describes survival dates (anchor set, or survival parameters present), the error now says validate = FALSE is not a workaround and asks for the metadata fix instead. Found by the whole-branch review.
MockData's vignettes dogfood the package and R CMD check rebuilds them in CI, so they are integration tests, but only if a render fails when the code stops doing what the prose says. Every factual claim in the survival docs is now computed with inline R and guarded by a hidden chunk of named stopifnot() conditions, including the #54 and #55 caveats, which will fail the build until the docs are updated when those issues are fixed. The survival documentation is split across the site's sections: the tutorial keeps the teaching path; a new how-to (survival-dates-v05) covers adding survival dates to metadata, migrating from create_wide_survival_data(), deriving observed status and follow-up time from one observation window, and keeping the uncontaminated data; a new explanation (survival-design-v05) brings the ADR's reasoning onto the site: the stages, event allocation, rules on clean dates, the three data layers, why chained censoring is rejected, and the v0.5 restrictions. A mutation check showed the assertions catch regressions: flipping the censoring comparison failed the tutorial and the how-to ("no event is recorded after a death"), and making it non-strict failed the explanation's tie claim. CONTRIBUTING.md now states the rule and the pattern; #62 tracks retrofitting the other vignettes. R CMD check --as-cran: 0 errors, 0 warnings.
Three findings from a review of b0a4e97, each verified before changing: - The explanation claimed that adding survival dates does not change other variables' values. That holds for generated values, but post-processing draws missing codes and garbage from one stream shared by all columns, so a survival date with missing codes placed before another column changed 42 of 100 of its returned values. The claim is narrowed to generated values, and the page now demonstrates and asserts both halves. - The tutorial's 'every removed event has an earlier death' checked only that a death date existed. The demo event now falls on a fixed day, so the claim asserts the death came before it; with the censoring rule flipped, the render fails on exactly this claim. - The layers example used !is.na(death), which cannot tell clean from contaminated dates because the missing code is not NA. It now uses as.numeric(death - entry): 100 to 200 days on clean dates, about -37,095 if recomputed from the returned missing code, and both are asserted.
…40) The explanation said post-processing was the only exception to seed stability when a survival date is added. Survival dates also share one stage stream, drawn in dependency order, so a survival date drawn earlier changes later ones: adding one before death changed 70 of 100 clean death dates with the same seed (a reviewer saw 74 in another setup), while adding one after it changed none. The paragraph now states both exceptions and the page asserts all five facts: baseline values unchanged, earlier and later survival dates, and post-processing with and without missing codes.
DougManuel
added a commit
that referenced
this pull request
Sep 27, 2026
… 0.5.0 Brings #51, #52, #59 and #61 into release prep. NEWS: the dev entries for survival dates (#40), range missing codes (#58) and the minimal-example fixture corrections are filed under the 0.5.0 categories (breaking changes, new features, validation, bug fixes) plus new Deprecations and Known issues headings; every dev bullet is present, two in the forms release prep had already rewritten, which dev had not changed since the branches split. The heading carries the provisional release date. README mentions survival dates in the 0.5 summary and feature list. cran-comments corrects the local check record: the local check library lacks simstudy (3 notes, run with _R_CHECK_FORCE_SUGGESTS_=false), while CI installs all suggested packages and reports Status: OK.
This was referenced Sep 27, 2026
Merged
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #40.
create_mock_data()now generates survival dates (entry, event, death, loss to follow-up and administrative censoring) from metadata alone. Before this, survival data had to be generated separately withcreate_wide_survival_data(), a known issue listed since v0.2.0. Design:development/adr/v05-survival-dates.md(D1–D10, amended after an external design review). Plan:development/plans/2026-09-24-survival-dates.md.survival, computed after baseline generation from an anchor date by the new exportedgenerate_survival_dates().mock_spec_survival()is the direct API. The legacy statistics are ported exactly (D4): a fixedfloor(n * event_prop)event count, and uniform, exponential or Gompertz follow-up in whole days. Two rules apply: a date before its anchor becomesNA, and a date later than an earliercensored_bydate becomesNA.anchor(required) andcensored_by(optional) onvariables.csv, besidefollowup_*andevent_prop. A date with survival parameters but noanchorfails with a message naming the fix. Chained censoring is rejected, because it reports events as observed after observation ended (D7).survival = 4Lis appended; no existing stage index or pinned value moves.is.nais added to themockFormulaallow-list (D8). The docs show status and follow-up time derived from one observation window.anchor,create_mock_data()stops and names the survival variables instead of dropping them.create_wide_survival_data()warns once per session.anchor/censored_by, so it now generates entirely through the v0.4 pipeline. This depended on Expand range-notation missing codes on the v0.4 path (#58) #59.Documentation as integration tests
MockData's vignettes run the package's own code, and R CMD check rebuilds them in CI, so they are integration tests, but only if a render fails when the code stops doing what the prose says. The survival documentation now works that way (b0a4e97):
stopifnot()conditions (17 across the three pages). The Gompertz survival parameters produce degenerate follow-up times #54 and Administrative censoring date is drawn per person, not as a fixed study end date #55 caveats are assertions too, so fixing either issue fails the build until the docs are updated.tutorial-survival-data, slimmed to the teaching path.survival-dates-v05. Add survival dates to metadata; migrate fromcreate_wide_survival_data(), with both runnable; derive observed status and follow-up time from one observation window, inmockFormulaand in R, checked against each other; keep the uncontaminated data.survival-design-v05. Brings the ADR's reasoning onto the site: the stages, event allocation, rules on clean dates, the three data layers, why chained censoring is rejected, and the v0.5 restrictions.! no event is recorded after a death. Making it non-strict failed the explanation's tie claim.CONTRIBUTING.mdnow states the rule and the pattern. Retrofit claim assertions into the existing vignettes (vignettes are integration tests) #62 tracks retrofitting the other 14 vignettes, which carry 2 assertions between them.Verification (local, R 4.4.2 macOS arm64, simstudy installed)
errorcolumn. (The docs commit changes no R code.)R CMD check --as-cran --no-manualon the tarball, after the docs commit: 0 errors, 0 warnings; NOTEs are the new-submission notice, simstudy absent from the local check library (which runs outside renv), and the timestamp check. The vignette rebuild, including all 17 survival assertions, passes.pkgdown::build_site()succeeds; all ten changed or affected vignettes render.test-survival-characterization.R). The survival seed pin was computed independently by emulating the seed primitive, not captured from the code.Review
Built test-first, task by task. A fresh whole-branch reviewer (Opus) then ran its own checks:
Verdict: ready with one fix. That fix (b682a15): errors on survival metadata no longer recommend
validate = FALSE, which would have either hit the D10 stop or silently dropped survival dates.Deferred minors (not fixed here):
anchor/censored_byare not checked to be single stringstrimws()on CSV-read anchorsanchoron a non-date row is silently ignoredtutorial-dates.qmd, the minimal-example README, and the post-v0.4.0 plan's multi-column stepConcerns reproduced, not fixed (exact-port decision): #54 (Gompertz degeneracy), #55 (administrative censoring drawn per person), #56 (legacy discrepancies, now including floating-point flooring), the
else-row evidence on #15. New: #60 (minimal example'sheightgarbage range producesNaN; pre-existing, now on the default route).Review focus
R/mock_spec_survival.R: the port againstR/create_date_var.R:295-346, and per-variable rule application in dependency order.R/mock_spec_recodeflow.R: adapter kind detection and the no-anchor error.R/create_mock_data.R: the D10 guard and the survival-aware error advice.vignettes/tutorial-survival-data.qmd.After this merges: merge into
v05-release-prep(#53) and fold the #40 NEWS section into the 0.5.0 structure.🤖 Generated with Claude Code