diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 32805d0..bb5d863 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -131,6 +131,30 @@ covr::package_coverage() - If adding new test data, document its purpose - Keep test data small and focused +#### Vignettes are integration tests + +The vignettes run MockData's own code on realistic metadata, and `R CMD check` rebuilds every one of them in CI. That makes them integration tests, but only if a vignette fails when the code stops doing what its prose says. A vignette that prints a number without checking it renders happily after a regression, and a number typed into the prose goes stale without anyone noticing. + +When you write or change a vignette: + +- Compute every factual claim with inline R (`` `r n_deaths` ``) instead of typing the number into the prose. +- Guard the claims with a hidden chunk of named `stopifnot()` conditions, so the render, and therefore `R CMD check` and CI, fails with a message naming the broken claim: + + ````markdown + ```{r} + #| include: false + stopifnot( + "deaths are exactly floor(n * event_prop)" = n_deaths == floor(n * 0.3), + "no event is recorded after a death" = events_after_death == 0 + ) + ``` + ```` + +- State known issues as claims too, with the issue number in the message. When the issue is fixed, the render fails until the documentation is updated. +- For a new feature, check that the assertions can fail: break the behaviour in a scratch copy and check that the render stops. + +The survival vignettes (`tutorial-survival-data`, `survival-dates-v05`, `survival-design-v05`) follow this pattern. + ### Building Documentation #### Prerequisites for documentation builds diff --git a/NAMESPACE b/NAMESPACE index e6166e4..756719d 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -18,6 +18,7 @@ export(extract_proportions) export(generate_garbage_values) export(generate_mock_data_native) export(generate_mock_data_simstudy) +export(generate_survival_dates) export(get_cycle_variables) export(get_enabled_variables) export(get_raw_var_dependencies) @@ -39,6 +40,7 @@ export(mock_spec_continuous) export(mock_spec_date) export(mock_spec_formula) export(mock_spec_from_recodeflow) +export(mock_spec_survival) export(parse_range_notation) export(parse_variable_start) export(postprocess_mock_data) diff --git a/NEWS.md b/NEWS.md index f80ef11..52438b6 100644 --- a/NEWS.md +++ b/NEWS.md @@ -11,6 +11,48 @@ unchanged: derived variables without a `mockFormula` remain excluded from generation, and `Func::` dispatch is not yet supported. +## Survival dates in create_mock_data() (#40) + +- `create_mock_data()` now generates survival dates from metadata. A date + whose `variables.csv` row sets `anchor` (its entry-date variable) is a + survival date: `floor(n * event_prop)` rows receive a date between + `followup_min` and `followup_max` days after the anchor, drawn from a + uniform, exponential or Gompertz distribution, and the rest are `NA`. An + optional `censored_by` column names a competing survival date, such as + death, that sets this date to `NA` where it comes first. The statistics and + rules are those of `create_wide_survival_data()`, ported exactly. New + functions: `mock_spec_survival()` and `generate_survival_dates()`. This + resolves the known issue, listed since v0.2.0, that survival data had to be + generated separately. +- Migration: add `anchor` (and `censored_by` where a competing risk applies) + to each survival date's row. A date with `followup_min`, `followup_max` or + `event_prop` but no `anchor` now fails with a message naming the fix; + previously it was silently dropped or generated as a plain calendar date. +- Missing codes and garbage are applied to survival dates after the survival + rules. Garbage that places a date before entry is therefore kept, whereas + the legacy engine set such dates to `NA`. +- Derived columns, from `mockFormula` or survival dates, describe the clean + generated values; missing codes and garbage are then applied to each column + independently. `is.na` is added to the `mockFormula` allow-list so status + and follow-up time can be derived. Derive both from the same observation + window: `as.integer(!is.na(death_date))` only says a death date exists, + not that the death was observed before censoring. +- A `censored_by` target may not itself have `censored_by`. Chained censoring + would report some events as observed after observation ended, so it is + rejected in this version with a message naming the fix. +- The packaged minimal example now generates entirely through the v0.4 + pipeline, because its Gompertz survival dates no longer force the legacy + generator. Its seeded output differs from v0.4. +- If the legacy generator is selected (`validate = FALSE`, + `variable_details = NULL`, detail-level-only `databaseStart`, or another + unsupported variable) on metadata that sets `anchor`, `create_mock_data()` + stops and names the survival variables rather than dropping them. +- `create_wide_survival_data()` is deprecated and warns once per session. +- Known issues found while porting, reproduced unchanged: #54 (Gompertz + follow-up times are degenerate with the packaged parameters), #55 + (administrative censoring is drawn per person), #56 (smaller legacy + discrepancies). + ## Reproducibility (breaking change) - Seeded output changed once for the RNG-stream mechanism in this release. @@ -368,8 +410,8 @@ vars_with_garbage <- variables %>% ## Known issues -- Survival variable type must be generated manually with `create_wide_survival_data()` -- Cannot be used in `create_mock_data()` batch generation (requires paired variables) +- Survival variable type must be generated manually with `create_wide_survival_data()` (resolved in 0.5.0, #40) +- Cannot be used in `create_mock_data()` batch generation (requires paired variables) (resolved in 0.5.0, #40) --- diff --git a/R/create_mock_data.R b/R/create_mock_data.R index 0817207..6d6e0b5 100644 --- a/R/create_mock_data.R +++ b/R/create_mock_data.R @@ -18,6 +18,11 @@ if (variable$type == "formula") { return(FALSE) } + # type = "survival" is supported end-to-end (#40): both backends skip it, + # generate_survival_dates() computes it, postprocess treats it as a date. + if (variable$type == "survival") { + return(FALSE) + } formula <- variable$formula has_formula <- !is.null(formula) && @@ -43,6 +48,23 @@ names(spec$variables)[unsupported] } +#' @noRd +.has_survival_metadata <- function(variables) { + # A row sets anchor, or carries survival parameters (followup_min, + # followup_max, event_prop): the metadata describes survival dates. + if ("anchor" %in% names(variables)) { + anchors <- as.character(variables$anchor) + if (any(!is.na(anchors) & trimws(anchors) != "")) { + return(TRUE) + } + } + survival_fields <- intersect(c("followup_min", "followup_max", "event_prop"), names(variables)) + any(vapply(survival_fields, function(field) { + values <- as.character(variables[[field]]) + any(!is.na(values) & trimws(values) != "") + }, logical(1))) +} + #' @noRd .create_mock_data_v04_native_supported <- function(spec) { length(.create_mock_data_v04_unsupported_variables(spec)) == 0 @@ -76,13 +98,27 @@ role = "enabled" ), error = function(e) { + # Survival dates are generated only by the v0.4 pipeline (ADR + # v05-survival-dates D10): validate = FALSE would stop on anchored + # metadata or silently drop survival dates, so it is not offered then. + advice <- if (.has_survival_metadata(variables)) { + paste0( + "Fix the metadata as the message above describes. Survival dates ", + "are generated only by the v0.4 pipeline, so validate = FALSE is ", + "not a workaround for survival dates." + ) + } else { + paste0( + "Fix the metadata (for exponential variables, supply a positive ", + "'rate'), or call create_mock_data() with validate = FALSE to use ", + "the legacy generator, which warns and substitutes a uniform draw ", + "for invalid distribution parameters." + ) + } stop( conditionMessage(e), "\n", "Metadata validation failed while building the v0.4 specification. ", - "Fix the metadata (for exponential variables, supply a positive 'rate'), ", - "or call create_mock_data() with validate = FALSE to use the legacy ", - "generator, which warns and substitutes a uniform draw for invalid ", - "distribution parameters.", + advice, call. = FALSE ) } @@ -105,11 +141,14 @@ } baseline <- generate_mock_data_native(spec, n = n, seed = seed) - staged <- evaluate_mock_formulas(baseline, spec, seed = seed) - # Baseline generation, formula evaluation, and post-processing use distinct - # L'Ecuyer-CMRG sub-streams derived from the single public seed (see - # .with_mock_seed / ADR v05-seed-contract), so all three stages pass the - # same seed and select their own stage internally. + dated <- generate_survival_dates(baseline, spec, seed = seed) + staged <- evaluate_mock_formulas(dated, spec, seed = seed) + # Baseline generation, survival dates, formula evaluation, and + # post-processing use distinct L'Ecuyer-CMRG sub-streams derived from the + # single public seed (see .with_mock_seed / ADR v05-seed-contract), so all + # four stages pass the same seed and select their own stage internally. + # Survival precedes formulas so a mockFormula can use survival dates + # (ADR v05-survival-dates D6). postprocess_mock_data(staged, spec, seed = seed) } @@ -169,7 +208,10 @@ #' when `variable_details = NULL`, when detail-level `databaseStart` filtering is #' needed but the variables metadata has no `databaseStart` column, or when a #' variable uses a feature not yet supported by the v0.4 native backend. Set -#' `verbose = TRUE` to see which path was chosen. +#' `verbose = TRUE` to see which path was chosen. Survival dates (rows that +#' set `anchor`) are generated only by the v0.4 pipeline: if the legacy path +#' is selected for metadata that sets `anchor`, `create_mock_data()` stops and +#' names the survival variables. #' #' In the v0.4 path, baseline generation and post-processing draw from distinct, #' independent sub-streams derived from a single `seed`; output is reproducible @@ -206,9 +248,9 @@ #' see \code{vignette("reference-config", package = "MockData")}. #' #' @examples -#' # The packaged minimal example includes deliberately messy metadata -#' # (auto-normalized proportions, survival dates without an anchor): the -#' # warnings it generates are expected and demonstrate MockData's diagnostics. +#' # The packaged minimal example covers every variable type, including +#' # survival dates anchored on interview_date. It auto-normalizes some +#' # proportions, so warnings about that are expected. #' mock_data <- create_mock_data( #' databaseStart = "minimal-example", #' variables = system.file("extdata/minimal-example/variables.csv", @@ -225,12 +267,16 @@ #' # Columns with straightforward metadata generate cleanly: #' head(mock_data[, c("age", "smoking", "interview_date")]) #' -#' # Fallback mode: no variable_details, simple default generators +#' # Fallback mode: no variable_details, simple default generators. Survival +#' # dates need the v0.4 pipeline, so drop the rows that set an anchor first. +#' fallback_variables <- read.csv( +#' system.file("extdata/minimal-example/variables.csv", package = "MockData"), +#' stringsAsFactors = FALSE, +#' check.names = FALSE +#' ) #' mock_data <- create_mock_data( #' databaseStart = "minimal-example", -#' variables = system.file("extdata/minimal-example/variables.csv", -#' package = "MockData" -#' ), +#' variables = fallback_variables[fallback_variables$anchor == "", ], #' variable_details = NULL, #' n = 500 #' ) @@ -340,6 +386,27 @@ create_mock_data <- function(databaseStart, } } + # ADR v05-survival-dates D10: the legacy dispatcher cannot build survival + # dates from anchors (it would warn and drop them), so stop rather than + # return plausible-looking partial survival data. + if ("anchor" %in% names(enabled_vars)) { + anchor_values <- as.character(enabled_vars$anchor) + anchored <- enabled_vars$variable[ + !is.na(anchor_values) & trimws(anchor_values) != "" + ] + if (length(anchored) > 0) { + stop( + "Survival date variable(s) ", paste(anchored, collapse = ", "), + " (anchor set) are generated only by the v0.4 pipeline, but the ", + "legacy generator was selected. It is used when validate = FALSE, ", + "when variable_details is NULL, when only variable_details has a ", + "databaseStart column, or when another variable uses a feature the ", + "v0.4 pipeline does not support. Run with verbose = TRUE to see which.", + call. = FALSE + ) + } + } + if (nrow(enabled_vars) == 0) { stop("No enabled non-derived variables found in configuration. ", "Add 'enabled' to the role column for variables you want to generate, ", diff --git a/R/create_wide_survival_data.R b/R/create_wide_survival_data.R index 81a6e27..f2a0857 100644 --- a/R/create_wide_survival_data.R +++ b/R/create_wide_survival_data.R @@ -1,5 +1,13 @@ +# Session state for once-per-session notices (ADR v05-survival-dates D9). +.mockdata_state <- new.env(parent = emptyenv()) + #' Create wide survival data for cohort studies #' +#' **Deprecated as of MockData 0.5.0.** Add `anchor` (and `censored_by` for a +#' competing risk) to the survival dates' rows in `variables.csv` and generate +#' them with [create_mock_data()]; see `vignette("tutorial-survival-data")`. +#' The function still works and warns once per session. +#' #' Generates wide-format survival data (one row per individual) with up to 5 date #' variables (entry, event, death, loss-to-follow-up, administrative censoring). #' Applies temporal ordering constraints and supports garbage data generation for @@ -175,6 +183,18 @@ create_wide_survival_data <- function(var_entry_date, n, seed = NULL, prop_garbage = NULL) { + if (!isTRUE(.mockdata_state$wide_survival_warned)) { + .mockdata_state$wide_survival_warned <- TRUE + warning( + "create_wide_survival_data() is deprecated as of MockData 0.5.0. Add ", + "`anchor` (and `censored_by` for competing risks) to the survival ", + "dates' rows in variables.csv and generate them with create_mock_data(); ", + "see vignette(\"tutorial-survival-data\", package = \"MockData\"). ", + "This warning is shown once per session.", + call. = FALSE + ) + } + # ========== VALIDATION ========== diff --git a/R/mock_spec.R b/R/mock_spec.R index 04cb0fb..cba2c0a 100644 --- a/R/mock_spec.R +++ b/R/mock_spec.R @@ -743,6 +743,88 @@ mock_spec_formula <- function(name, variable } +#' Create a survival date variable specification +#' +#' `mock_spec_survival()` describes a date generated relative to an anchor +#' (entry) date: `floor(n * event_prop)` rows receive a date a follow-up time +#' after the anchor, drawn within `[followup_min, followup_max]` days, and the +#' rest are `NA` (censored). Survival variables are computed after baseline +#' generation by [generate_survival_dates()], so the anchor must be another +#' variable in the same [mock_spec()]. +#' +#' @param name Variable name. +#' @param anchor Name of the date variable this date is generated relative to. +#' @param followup_min,followup_max Follow-up window in days after the anchor. +#' @param event_prop Share of rows that receive an event, in `[0, 1]`. +#' @param distribution Follow-up time distribution: `"uniform"`, +#' `"exponential"`, or `"gompertz"`. +#' @param censored_by Optional name of a competing survival variable with the +#' same anchor. Where that date is earlier than this one, this one becomes +#' `NA`. +#' @param shape,rate Optional Gompertz parameters (defaults 0.1 and 0.0001). +#' The exponential distribution derives its rate from the follow-up window. +#' @param source_format Output format. Only `"analysis"` (R `Date`) is +#' supported for survival dates. +#' @param missing_codes,missing_proportions,garbage_rules,provenance,model_hint +#' As for other variable specifications; applied by post-processing after +#' the survival rules. +#' +#' @return A `mock_spec_variable` object of type `"survival"`. +#' @family mock specification APIs +#' @seealso [generate_survival_dates()], [mock_spec_date()] +#' +#' @examples +#' spec <- mock_spec( +#' mock_spec_date("entry", range = as.Date(c("2001-01-01", "2005-12-31"))), +#' mock_spec_survival("death", anchor = "entry", +#' followup_min = 365, followup_max = 7300, event_prop = 0.2), +#' mock_spec_survival("event", anchor = "entry", +#' followup_min = 0, followup_max = 5475, event_prop = 0.3, +#' censored_by = "death") +#' ) +#' validate_mock_spec(spec) +#' +#' @export +mock_spec_survival <- function(name, + anchor, + followup_min, + followup_max, + event_prop, + distribution = "uniform", + censored_by = NULL, + shape = NULL, + rate = NULL, + source_format = "analysis", + missing_codes = character(0), + missing_proportions = numeric(0), + garbage_rules = list(), + provenance = "direct", + model_hint = "native-postprocess") { + variable <- .new_mock_spec_variable( + name = name, + type = "survival", + rtype = "date", + distribution = distribution, + source_format = source_format, + missing_codes = missing_codes, + missing_proportions = missing_proportions, + garbage_rules = garbage_rules, + provenance = provenance, + model_hint = model_hint, + anchor = anchor, + censored_by = censored_by, + followup_min = followup_min, + followup_max = followup_max, + event_prop = event_prop, + shape = shape, + rate = rate + ) + # Computed once at construction, as for mock_spec_formula(); anchor and + # censored_by feed the shared dependency ordering (ADR D7). + variable$depends_on <- .survival_dependencies(variable) + variable +} + #' Check whether an object is a MockData specification #' #' @param x Object to check. @@ -988,6 +1070,8 @@ print.mock_spec_validation_result <- function(x, ...) { "Variable '", variable$name, "' has unsupported formula rType '", variable$rtype, "'." )) } + } else if (variable$type == "survival") { + errors <- c(errors, .validate_survival_variable(variable)) } else { errors <- c(errors, paste0("Variable '", variable$name, "' has unsupported type '", variable$type, "'.")) } @@ -1054,6 +1138,15 @@ validate_mock_spec <- function(spec, n = NULL, strict = TRUE) { if (!is.null(formula_error)) { errors <- c(errors, formula_error) } + + errors <- c(errors, .validate_survival_referents(spec)) + survival_error <- tryCatch({ + .order_survival_variables(spec) + NULL + }, error = function(e) conditionMessage(e)) + if (!is.null(survival_error)) { + errors <- c(errors, survival_error) + } } } diff --git a/R/mock_spec_formula.R b/R/mock_spec_formula.R index bd0d22e..c457e71 100644 --- a/R/mock_spec_formula.R +++ b/R/mock_spec_formula.R @@ -6,13 +6,14 @@ # Base functions/operators a mockFormula may call. Deliberately closed and # RNG-free: Phase A formulas are deterministic transformations of generated -# columns. Widen only on concrete need (ADR D3). +# columns. Widen only on concrete need (ADR D3). is.na was added for deriving +# survival status (ADR v05-survival-dates D8). .formula_allowlist <- c( "+", "-", "*", "/", "^", "%%", "%/%", "(", "<", "<=", ">", ">=", "==", "!=", "&", "|", "!", "ifelse", "pmin", "pmax", "min", "max", "abs", "round", "floor", "ceiling", "sqrt", "exp", "log", "log10", "sum", "mean", "cut", - "as.numeric", "as.integer", "as.factor", "factor", "c" + "as.numeric", "as.integer", "as.factor", "factor", "c", "is.na" ) #' @noRd @@ -85,29 +86,17 @@ } #' @noRd -.order_formula_variables <- function(spec) { - formula_names <- names(spec$variables)[ - vapply(spec$variables, .is_formula_variable, logical(1)) - ] - # Skip unparseable formulas here too (see .validate_formula_referents()): - # the per-variable validator already reports the parse error once, and a - # variable that can't be parsed has no computable dependency set — leaving - # it in `formula_names` would either duplicate the parse error (via - # .formula_dependencies()) or permanently block any variable that - # references it from ever being ordered. - parseable_names <- Filter(function(name) { - tryCatch({ - .parse_mock_formula(spec$variables[[name]]) - TRUE - }, error = function(e) FALSE) - }, formula_names) - - remaining <- parseable_names +.order_dependent_variables <- function(names_in_scope, dependencies, label) { + # Shared by every derived-variable stage (ADR v05-survival-dates D7): + # repeated passes in spec order, each placing the variables whose in-scope + # dependencies are already placed. Dependencies outside `names_in_scope` + # (baseline columns) are available from the start. + remaining <- names_in_scope ordered <- character(0) while (length(remaining) > 0) { progressed <- FALSE for (name in remaining) { - deps <- intersect(.formula_dependencies(spec$variables[[name]]), parseable_names) + deps <- intersect(dependencies[[name]], names_in_scope) if (all(deps %in% ordered)) { ordered <- c(ordered, name) remaining <- setdiff(remaining, name) @@ -116,7 +105,7 @@ } if (!progressed) { stop( - "Formula dependency cycle or unresolved ordering among: ", + label, " dependency cycle or unresolved ordering among: ", paste(remaining, collapse = ", "), call. = FALSE ) @@ -125,6 +114,28 @@ ordered } +#' @noRd +.order_formula_variables <- function(spec) { + formula_names <- names(spec$variables)[ + vapply(spec$variables, .is_formula_variable, logical(1)) + ] + # Skip unparseable formulas here too (see .validate_formula_referents()): + # the per-variable validator already reports the parse error once, and a + # variable that can't be parsed has no computable dependency set — leaving + # it in `formula_names` would either duplicate the parse error (via + # .formula_dependencies()) or permanently block any variable that + # references it from ever being ordered. + parseable_names <- Filter(function(name) { + tryCatch({ + .parse_mock_formula(spec$variables[[name]]) + TRUE + }, error = function(e) FALSE) + }, formula_names) + + dependencies <- lapply(spec$variables[parseable_names], .formula_dependencies) + .order_dependent_variables(parseable_names, dependencies, "Formula") +} + #' @noRd .formula_parent_env <- function() { # baseNamespace() (the brief's original spelling) is not itself an diff --git a/R/mock_spec_native.R b/R/mock_spec_native.R index 7a29680..d4ac437 100644 --- a/R/mock_spec_native.R +++ b/R/mock_spec_native.R @@ -7,13 +7,14 @@ # Named generation stages -> fixed L'Ecuyer-CMRG sub-stream indices. Indices # are FROZEN: adding a future stage must not renumber baseline/postprocess, or -# seeded output for existing features would shift. formula (#39) and correlate -# (#42) are reserved now though unused in this release. +# seeded output for existing features would shift. correlate (#42) is reserved +# though unused; survival (#40) is appended, never inserted. .MOCK_STAGES <- c( baseline = 0L, postprocess = 1L, formula = 2L, - correlate = 3L + correlate = 3L, + survival = 4L ) #' @noRd @@ -368,9 +369,10 @@ #' exit, so output is reproducible for a given seed and package version #' without perturbing the caller's RNG. #' -#' @return A data frame with `n` rows and one column per non-formula -#' `mock_spec` variable (`type = "formula"` variables are appended -#' afterwards by [evaluate_mock_formulas()]). +#' @return A data frame with `n` rows and one column per non-derived +#' `mock_spec` variable (`type = "survival"` and `type = "formula"` +#' variables are appended afterwards by [generate_survival_dates()] and +#' [evaluate_mock_formulas()]). #' @family mock generation APIs #' @seealso [mock_spec()], [mock_continuous()], [mock_spec_from_recodeflow()], #' [postprocess_mock_data()], [generate_mock_data_simstudy()], @@ -394,15 +396,15 @@ generate_mock_data_native <- function(spec, n, seed = NULL) { .check_native_backend_scope(spec) .with_mock_seed(seed, { - # type = "formula" variables have no distribution to sample from; they - # are computed post-baseline by evaluate_mock_formulas(). Filtering them + # Derived variables (type = "formula" or "survival") are computed + # post-baseline by evaluate_mock_formulas() and generate_survival_dates(). Filtering them # out here (rather than in .generate_native_variable()) leaves this # branch's assembly of `columns` byte-identical to before #39 for any # spec with no formula variables (the common, zero-seeded-output-change # case) - and a formula-only spec still falls through to the same # .empty_native_data(n) path a variable-less spec would use. generated_variables <- spec$variables[ - !vapply(spec$variables, .is_formula_variable, logical(1)) + !vapply(spec$variables, .is_derived_variable, logical(1)) ] if (length(generated_variables) == 0) { .empty_native_data(n) diff --git a/R/mock_spec_postprocess.R b/R/mock_spec_postprocess.R index cb19225..bad34a4 100644 --- a/R/mock_spec_postprocess.R +++ b/R/mock_spec_postprocess.R @@ -7,7 +7,7 @@ # ============================================================================== #' @noRd -.postprocess_empty_diagnostics <- function(spec, n) { +.postprocess_empty_diagnostics <- function(spec, n, data = NULL) { variables <- lapply(spec$variables, function(variable) { garbage_rule_names <- names(variable$garbage_rules) if (is.null(garbage_rule_names)) { @@ -41,6 +41,20 @@ entry$depends_on <- variable$depends_on } + # ADR v05-survival-dates D7: survival dates are derived from their anchor; + # n_events counts the dates present before post-processing. + if (.is_survival_variable(variable)) { + entry$derived <- TRUE + entry$anchor <- variable$anchor + entry$censored_by <- variable$censored_by + entry$depends_on <- variable$depends_on + entry$n_events <- if (is.null(data)) { + NA_integer_ + } else { + sum(!is.na(data[[variable$name]])) + } + } + entry }) @@ -385,7 +399,7 @@ postprocess_mock_data <- function(data, spec, seed = NULL, diagnostics = TRUE) { .with_mock_seed(seed, { output <- data - diag <- .postprocess_empty_diagnostics(spec, nrow(data)) + diag <- .postprocess_empty_diagnostics(spec, nrow(data), data) for (variable_name in names(spec$variables)) { variable <- spec$variables[[variable_name]] diff --git a/R/mock_spec_recodeflow.R b/R/mock_spec_recodeflow.R index 0ba13ae..a23edcc 100644 --- a/R/mock_spec_recodeflow.R +++ b/R/mock_spec_recodeflow.R @@ -119,6 +119,10 @@ variable_type <- tolower(.row_character(var_row, "variableType", "")) if (rtype == "date" || variable_type == "date") { + # A non-blank anchor makes a date a survival date (#40, ADR D3). + if (!.is_blank(.row_character(var_row, "anchor", ""))) { + return("survival") + } return("date") } if (variable_type == "categorical" || rtype %in% c("factor", "character", "logical")) { @@ -150,7 +154,8 @@ kind, continuous = "double", categorical = "factor", - date = "date" + date = "date", + survival = "date" ) } @@ -443,6 +448,41 @@ )) } + if (kind == "survival") { + return(mock_spec_survival( + name = variable, + anchor = .row_character(var_row, "anchor"), + censored_by = .row_character(var_row, "censored_by", NULL), + followup_min = .row_numeric(var_row, "followup_min"), + followup_max = .row_numeric(var_row, "followup_max"), + event_prop = .row_numeric(var_row, "event_prop"), + distribution = .recodeflow_distribution(var_row, details), + shape = .row_numeric(var_row, "shape"), + rate = .row_numeric(var_row, "rate"), + source_format = .row_character(var_row, "sourceFormat", "analysis"), + missing_codes = missing$codes, + missing_proportions = missing$proportions, + garbage_rules = garbage_rules, + provenance = provenance + )) + } + + # ADR v05-survival-dates D3: survival parameters without an anchor were + # silently dropped or generated as a plain calendar date before #40. + survival_fields <- c("followup_min", "followup_max", "event_prop") + present <- survival_fields[!vapply(survival_fields, function(field) { + is.na(.row_numeric(var_row, field)) + }, logical(1))] + if (length(present) > 0) { + stop( + "Date variable '", variable, "' has survival parameter(s) ", + paste(present, collapse = ", "), " but no anchor. Add anchor = ", + "\"\" to its variables row to generate it as a ", + "survival date, or remove the survival parameters.", + call. = FALSE + ) + } + source_format <- .row_character(var_row, "sourceFormat", "analysis") .new_mock_spec_variable( @@ -458,10 +498,7 @@ provenance = provenance, model_hint = "native-postprocess", rate = .row_numeric(var_row, "rate"), - shape = .row_numeric(var_row, "shape"), - followup_min = .row_numeric(var_row, "followup_min"), - followup_max = .row_numeric(var_row, "followup_max"), - event_prop = .row_numeric(var_row, "event_prop") + shape = .row_numeric(var_row, "shape") ) } @@ -478,9 +515,11 @@ #' that begin with `NA::` (expanding an integer range such as `[997,999]` into #' its individual codes, with the row's proportion split equally), preserves #' categorical levels and proportions, carries -#' `garbage_*` settings into `garbage_rules`, and stores survival/date fields -#' such as `rate`, `shape`, `followup_min`, `followup_max`, and `event_prop` on -#' date variables for later backend milestones. +#' `garbage_*` settings into `garbage_rules`, and builds a survival date +#' ([mock_spec_survival()]) from any date row that sets `anchor`, reading +#' `censored_by`, `followup_min`, `followup_max`, `event_prop`, `distribution`, +#' `shape` and `rate`. A date row with survival parameters but no `anchor` is +#' an error. #' #' By default, variables identified by `DerivedVar::` or `Func::` rows are #' excluded because they should be evaluated after raw mock variables are diff --git a/R/mock_spec_simstudy.R b/R/mock_spec_simstudy.R index 236a922..a2998b2 100644 --- a/R/mock_spec_simstudy.R +++ b/R/mock_spec_simstudy.R @@ -128,13 +128,14 @@ #' @noRd .native_only_variables <- function(spec) { - # Mirror generate_mock_data_native()'s skip: type = "formula" variables have - # no distribution to sample from in either backend. They are computed - # post-baseline by evaluate_mock_formulas(), not by .generate_native_variable() + # Mirror generate_mock_data_native()'s skip: derived variables (formula and + # survival) have no distribution to sample from in either backend. They are + # computed post-baseline by evaluate_mock_formulas() and + # generate_survival_dates(), not by .generate_native_variable() # (which has no "formula" branch and would stop with an unhelpful "Native # backend does not support variable type 'formula'"). Filter(function(variable) { - !.is_formula_variable(variable) && !.simstudy_can_generate(variable) + !.is_derived_variable(variable) && !.simstudy_can_generate(variable) }, spec$variables) } @@ -252,12 +253,12 @@ generate_mock_data_simstudy <- function(spec, n, seed = NULL) { simstudy_variables <- .simstudy_variables(spec) native_only_variables <- .native_only_variables(spec) - # type = "formula" variables are excluded from both `simstudy_variables` and + # Derived variables (formula and survival) are excluded from both `simstudy_variables` and # `native_only_variables` above; exclude them here too so the final # assembly doesn't index `columns` by names that were never generated (see # generate_mock_data_native()'s identical formula skip). generated_variable_names <- names(spec$variables)[ - !vapply(spec$variables, .is_formula_variable, logical(1)) + !vapply(spec$variables, .is_derived_variable, logical(1)) ] .with_mock_seed(seed, { diff --git a/R/mock_spec_survival.R b/R/mock_spec_survival.R new file mode 100644 index 0000000..1a9b566 --- /dev/null +++ b/R/mock_spec_survival.R @@ -0,0 +1,295 @@ +# ============================================================================== +# Metadata-driven survival dates (#40) — spec-side helpers. +# Generation lives in generate_survival_dates() (same file, added in Task 3). +# ADR: development/adr/v05-survival-dates.md +# ============================================================================== + +.survival_distributions <- c("uniform", "exponential", "gompertz") + +#' @noRd +.is_survival_variable <- function(variable) { + identical(variable$type, "survival") +} + +#' @noRd +.is_derived_variable <- function(variable) { + # Derived types are computed after baseline generation from already-generated + # columns, so both backends skip them (ADR v05-survival-dates D1, D7). + .is_formula_variable(variable) || .is_survival_variable(variable) +} + +#' @noRd +.survival_dependencies <- function(variable) { + if (!.is_survival_variable(variable)) { + return(character(0)) + } + deps <- as.character(c(variable$anchor, variable$censored_by)) + unique(deps[!is.na(deps) & nzchar(trimws(deps))]) +} + +#' @noRd +.validate_survival_variable <- function(variable) { + errors <- character(0) + label <- paste0("Survival variable '", variable$name, "'") + is_number <- function(x) { + is.numeric(x) && length(x) == 1 && !is.na(x) && is.finite(x) + } + + if (.is_blank(variable$anchor)) { + errors <- c(errors, paste0( + label, " requires an anchor (the name of its entry-date variable)." + )) + } + + followup_min <- variable$followup_min + followup_max <- variable$followup_max + if (!is_number(followup_min) || !is_number(followup_max)) { + errors <- c(errors, paste0( + label, " requires finite numeric followup_min and followup_max (days)." + )) + } else { + if (followup_min < 0) { + errors <- c(errors, paste0(label, " followup_min must be >= 0.")) + } + if (followup_min > followup_max) { + errors <- c(errors, paste0(label, " followup_min must be <= followup_max.")) + } + } + + event_prop <- variable$event_prop + if (!is_number(event_prop) || event_prop < 0 || event_prop > 1) { + errors <- c(errors, paste0(label, " event_prop must be a number in [0, 1].")) + } + + distribution <- variable$distribution %||% "uniform" + if (!distribution %in% .survival_distributions) { + errors <- c(errors, paste0( + label, " distribution must be one of: ", + paste(.survival_distributions, collapse = ", "), "." + )) + } + + for (param in c("shape", "rate")) { + value <- variable[[param]] + supplied <- !is.null(value) && !(length(value) == 1 && is.na(value)) + if (supplied && !(is_number(value) && value > 0)) { + errors <- c(errors, paste0( + label, " ", param, " must be a positive number when supplied." + )) + } + } + + if (!identical(variable$rtype, "date")) { + errors <- c(errors, paste0(label, " must have rType 'date'.")) + } + if (!identical(variable$source_format %||% "analysis", "analysis")) { + errors <- c(errors, paste0( + label, " supports sourceFormat 'analysis' only in this version." + )) + } + + errors +} + +#' @noRd +.validate_survival_referents <- function(spec) { + errors <- character(0) + for (variable in spec$variables) { + if (!.is_survival_variable(variable)) next + label <- paste0("Survival variable '", variable$name, "'") + + anchor <- variable$anchor + if (!.is_blank(anchor)) { + anchor_variable <- spec$variables[[anchor]] + if (is.null(anchor_variable)) { + errors <- c(errors, paste0( + label, " has anchor '", anchor, "', which is not a variable in the ", + "spec. Enable the entry-date variable or correct the anchor." + )) + } else if (!identical(anchor_variable$type, "date")) { + errors <- c(errors, paste0( + label, " has anchor '", anchor, "' of type '", anchor_variable$type, + "'; an anchor must be a date variable." + )) + } + } + + censor <- variable$censored_by + if (.is_blank(censor)) next + censor_variable <- spec$variables[[censor]] + if (identical(censor, variable$name)) { + errors <- c(errors, paste0(label, " cannot be censored by itself.")) + } else if (is.null(censor_variable)) { + errors <- c(errors, paste0( + label, " has censored_by '", censor, + "', which is not a variable in the spec." + )) + } else if (!.is_survival_variable(censor_variable)) { + errors <- c(errors, paste0( + label, " has censored_by '", censor, "'; censored_by '", censor, + "' must be a survival variable." + )) + } else if (!identical(censor_variable$anchor, anchor)) { + errors <- c(errors, paste0( + label, " has censored_by '", censor, + "', which must share its anchor ('", anchor, "')." + )) + } else if (!.is_blank(censor_variable$censored_by)) { + # ADR D7: with a chain (A by B, B by C) the sequential rules report A as + # observed after observation ended, so chains are rejected in v0.5. + errors <- c(errors, paste0( + label, " has censored_by '", censor, "', which is itself censored by '", + censor_variable$censored_by, "'. Chained censoring is not supported in ", + "this version; censor '", variable$name, "' by the earliest date ", + "directly, or derive observed outcomes after generation." + )) + } + } + errors +} + +#' @noRd +.order_survival_variables <- function(spec) { + survival_names <- names(spec$variables)[ + vapply(spec$variables, .is_survival_variable, logical(1)) + ] + dependencies <- lapply(spec$variables[survival_names], .survival_dependencies) + .order_dependent_variables(survival_names, dependencies, "Survival") +} + +#' @noRd +.survival_param <- function(value, default) { + if (is.null(value) || length(value) != 1 || is.na(value)) default else value +} + +#' @noRd +.survival_event_days <- function(variable, n_events) { + # Exact port of the legacy follow-up-time draws (R/create_date_var.R:305-340, + # ADR v05-survival-dates D4), including behaviour filed as #54 (Gompertz + # clamp) and #56 (exponential rate derived from the window, not `rate`). + followup_min <- variable$followup_min + followup_max <- variable$followup_max + distribution <- tolower(variable$distribution %||% "uniform") + + if (distribution == "gompertz") { + shape <- .survival_param(variable$shape, 0.1) + rate <- .survival_param(variable$rate, 0.0001) + u <- stats::runif(n_events) + days <- (1 / shape) * log(1 - (shape / rate) * log(1 - u)) + return(pmax(followup_min, pmin(followup_max, days))) + } + if (distribution == "exponential") { + rate_exp <- 1 / ((followup_max - followup_min) / 3) + days <- stats::rexp(n_events, rate = rate_exp) + followup_min + return(pmin(days, followup_max)) + } + stats::runif(n_events, min = followup_min, max = followup_max) +} + +#' @noRd +.draw_survival_dates <- function(variable, anchor_dates) { + n <- length(anchor_dates) + values <- rep(as.Date(NA), n) + if (n == 0) { + return(values) + } + # Legacy event assignment: a fixed count, shuffled (R/create_date_var.R:295). + n_events <- floor(n * variable$event_prop) + is_event <- c(rep(TRUE, n_events), rep(FALSE, n - n_events)) + is_event <- is_event[sample(n)] + if (n_events > 0) { + # floor(): legacy output is whole days because create_date_var() converts + # dates to character and back (R/create_date_var.R:479). + values[is_event] <- anchor_dates[is_event] + + floor(.survival_event_days(variable, n_events)) + } + values +} + +#' Generate survival dates from their anchor dates +#' +#' Computes each `type = "survival"` variable in `spec` from its anchor column +#' in `data`: `floor(n * event_prop)` rows receive a date a follow-up time +#' after the anchor (whole days, within `[followup_min, followup_max]`), and +#' the rest are `NA`. Variables are processed in dependency order; each is +#' drawn and then has its rules applied: a date earlier than its anchor +#' becomes `NA`, and, if `censored_by` is set, a date later than the censoring +#' date becomes `NA`. Draws use the isolated `survival` sub-stream of the seed +#' contract. +#' +#' Survival dates describe true values. [postprocess_mock_data()] applies +#' missing codes and garbage afterwards, independently for each column. +#' +#' @param data Data frame of generated baseline values (from +#' [generate_mock_data_native()] or [generate_mock_data_simstudy()]), +#' containing each anchor column as class `Date`. +#' @param spec A `mock_spec`. Strict-validated at entry. +#' @param seed Optional whole-number seed. The caller's RNG state and kind are +#' restored on exit. +#' +#' @return `data` with one appended `Date` column per survival variable. +#' Returns `data` unchanged if the spec has no survival variables. +#' @family mock generation APIs +#' @seealso [mock_spec_survival()], [evaluate_mock_formulas()], +#' [postprocess_mock_data()] +#' +#' @examples +#' spec <- mock_spec( +#' mock_spec_date("entry", range = as.Date(c("2001-01-01", "2005-12-31"))), +#' mock_spec_survival("death", anchor = "entry", +#' followup_min = 365, followup_max = 7300, event_prop = 0.2) +#' ) +#' baseline <- generate_mock_data_native(spec, n = 10, seed = 1) +#' generate_survival_dates(baseline, spec, seed = 1) +#' +#' @export +generate_survival_dates <- function(data, spec, seed = NULL) { + if (!is.data.frame(data)) { + stop("data must be a data frame.", call. = FALSE) + } + validate_mock_spec(spec, n = nrow(data), strict = TRUE) + + ordered <- .order_survival_variables(spec) + if (length(ordered) == 0) { + return(data) + } + + anchors <- unique(vapply(spec$variables[ordered], function(variable) { + variable$anchor + }, character(1))) + missing_anchors <- setdiff(anchors, names(data)) + if (length(missing_anchors) > 0) { + stop( + "data is missing anchor column(s) required by survival variables: ", + paste(missing_anchors, collapse = ", "), + call. = FALSE + ) + } + for (anchor in anchors) { + if (!inherits(data[[anchor]], "Date")) { + stop( + "Anchor column '", anchor, "' must be of class Date; got ", + class(data[[anchor]])[1], ". Convert it with as.Date() first.", + call. = FALSE + ) + } + } + + .with_mock_seed(seed, stage = "survival", { + for (name in ordered) { + variable <- spec$variables[[name]] + anchor_dates <- data[[variable$anchor]] + values <- .draw_survival_dates(variable, anchor_dates) + # Legacy rules (R/create_wide_survival_data.R:347,363), applied per + # variable in dependency order so a censoring date is already final. + values[!is.na(values) & !is.na(anchor_dates) & values < anchor_dates] <- + as.Date(NA) + if (!.is_blank(variable$censored_by)) { + censor <- data[[variable$censored_by]] + values[!is.na(values) & !is.na(censor) & censor < values] <- as.Date(NA) + } + data[[name]] <- values + } + data + }) +} diff --git a/_pkgdown.yml b/_pkgdown.yml index 64be322..faa3a30 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -27,6 +27,7 @@ reference: - mock_spec_categorical - mock_spec_date - mock_spec_formula + - mock_spec_survival - is_mock_spec - validate_mock_spec - mock_spec_model_hints @@ -39,6 +40,7 @@ reference: - mock_spec_from_recodeflow - generate_mock_data_native - generate_mock_data_simstudy + - generate_survival_dates - evaluate_mock_formulas - postprocess_mock_data @@ -116,6 +118,7 @@ articles: navbar: How-to guides contents: - recodeflow-metadata-v04 + - survival-dates-v05 - diagnostics-and-garbage-v04 - migrating-from-v03-v04 - choosing-a-backend-v04 @@ -126,6 +129,7 @@ articles: navbar: Explanation contents: - design-philosophy-v04 + - survival-design-v05 - advanced-topics - title: Reference diff --git a/development/adr/v05-formula-evaluator.md b/development/adr/v05-formula-evaluator.md index 21139b0..674861e 100644 --- a/development/adr/v05-formula-evaluator.md +++ b/development/adr/v05-formula-evaluator.md @@ -1,6 +1,6 @@ # ADR: v0.5 Formula-Derived Variables -**Status**: ACCEPTED 2026-07-07 — Phase A (algebraic formulas), via a `mockFormula` extension column on `variable_details`, evaluated as a post-baseline stage in a restricted sandbox. Any individual decision remains open to maintainer revision before implementation begins. Implemented on branch v05-formula-evaluator (Phase A). +**Status**: ACCEPTED 2026-07-07 — Phase A (algebraic formulas), via a `mockFormula` extension column on `variable_details`, evaluated as a post-baseline stage in a restricted sandbox. Any individual decision remains open to maintainer revision before implementation begins. Implemented on branch v05-formula-evaluator (Phase A). Allow-list amended 2026-09-24 by `v05-survival-dates.md` D8 (adds `is.na`). **Date**: 2026-07-07 **Decision owner**: MockData maintainers **Issue**: #39 diff --git a/development/adr/v05-survival-dates.md b/development/adr/v05-survival-dates.md new file mode 100644 index 0000000..c5a2c6e --- /dev/null +++ b/development/adr/v05-survival-dates.md @@ -0,0 +1,229 @@ +# ADR: v0.5 Metadata-Driven Survival Dates + +**Status**: ACCEPTED 2026-09-24. Design agreed with the maintainer in conversation on 2026-09-22 (scope, fidelity policy, architecture, metadata convention, semantics); D5 and D8 ratified on 2026-09-24. Amended the same day after an external design review (Astra): D8's example corrected to derive status and time from one observation window, chained censoring rejected (D7), the three data layers named (D5), and the fixed stage order and event allocation stated as v0.5 restrictions (D4, D6, Future directions). Implemented on branch v05-survival-dates per development/plans/2026-09-24-survival-dates.md. +**Date**: 2026-09-22 +**Decision owner**: MockData maintainers +**Issue**: #40 (also advances #23, #17) +**Depends on**: #38 (seed contract), #39 (formula stage; dependency-ordering machinery reused here) + +## Decision summary + +| # | Question | Decision | +|---|---|---| +| D1 | Architecture | Survival dates are a new variable type computed in a **post-baseline stage** from their already-generated anchor date. One column per spec variable; the assembly contract is unchanged | +| D2 | Scope | **Full parity** with `create_wide_survival_data()`: entry, event, death, loss to follow-up, administrative censoring, and both consistency rules | +| D3 | Metadata convention | Two columns on `variables.csv` beside `followup_min`/`followup_max`/`event_prop`: `anchor` (required) and `censored_by` (optional) | +| D4 | Fidelity | **Exact port** of the legacy statistics and rules; concerns are filed as issues, not fixed in #40 | +| D5 | Contamination order | Consistency rules run on clean dates; missing codes and garbage are applied afterwards by postprocess | +| D6 | Stage and seed | Pipeline becomes baseline, survival, formulas, postprocess. `.MOCK_STAGES` gains `survival = 4L` (appended; no existing index moves) | +| D7 | Dependencies | Survival dates record `depends_on = c(anchor, censored_by)`, the field formulas already use; one ordering utility serves every derived stage | +| D8 | Formula allow-list | Add `is.na`, so survival status and time can be derived with `mockFormula` (amends the #39 ADR's D3) | +| D9 | Legacy path | `create_wide_survival_data()` warns once as deprecated and is kept as the characterization reference | +| D10 | Legacy generator and `anchor` | If the legacy generator runs on metadata carrying `anchor`, `create_mock_data()` stops and names the survival variables instead of dropping them | + +## Context + +Survival data has been generated outside `create_mock_data()` since v0.2.0. The NEWS known-issue entry reads: "Cannot be used in `create_mock_data()` batch generation (requires paired variables)". Users call `create_wide_survival_data()` by hand. + +The legacy engine is a loop. It generates the entry date as a plain calendar date. It then generates each other date by calling `create_date_var()` with the entry date passed in a `df_mock` column named `anchor_date` (`R/create_wide_survival_data.R`, step 1). Finally it applies two rules: any date earlier than entry becomes `NA` (`:347`), and an event date later than the death date becomes `NA` (`:363`). The per-date statistics live in `create_date_var()` (`R/create_date_var.R:209-350`): a date is a survival date when `followup_min`, `followup_max` and `event_prop` are all present; `floor(n_valid * event_prop)` rows receive an event, shuffled; follow-up times are uniform, exponential or Gompertz within the window; non-events are `NA`. + +The v0.4 pipeline handles these dates incorrectly today, in both of its paths (verified 2026-09-22 on the minimal example): + +- A Gompertz or exponential survival date fails the native support check (`R/create_mock_data.R:36`), and the fallback is all-or-nothing for the spec. The legacy dispatcher then calls `create_date_var()` without an `anchor_date` column, so every survival date in the spec, uniform ones included, warns and is dropped. For the minimal example, `create_mock_data()` returns `interview_date` and none of the four survival dates. +- In a spec whose survival dates are all uniform, nothing forces the fallback, and the native backend generates them as plain calendar dates over their `recStart` range. The adapter reads `followup_min`, `followup_max` and `event_prop` onto the spec (`R/mock_spec_recodeflow.R:418`), but nothing downstream uses them. With `interview_date` and `ltfu_date` alone (n = 2,000), `ltfu_date` is 100 per cent non-missing although `event_prop` is 0.1, and 128 dates fall before entry. + +So #40 fixes a correctness defect as well as a usability gap. + +The design follows from #39. A formula variable is a column computed from other generated columns, evaluated in dependency order after baseline generation. A survival date has the same structure: a column computed from its anchor. The only difference is that it draws random numbers. Treating survival dates as a derived stage reuses #39's skip-in-baseline pattern, dependency ordering, strict validation at stage entry, and diagnostics marking. + +### Constraints + +- **One seeded-output break in v0.5**, already spent by #38. Specs without survival variables must produce byte-identical output; the #48 pinned-value tests enforce this. +- **No backward-compatibility obligation** for 0.x metadata (maintainer decision, 2026-09-22). Metadata that relied on the old silent behaviour may now fail validation, provided the error names the fix. +- **recodeflow semantics untouched.** The new columns are MockData extensions; cchsflow and chmsflow need no change. +- **Keep causal modelling open.** The maintainer expects mock data to be used to study causal structure (DAG-based simulation) in future. That is out of scope here, but no decision should make a later structural-equation stage harder. D1 and D7 are chosen with this in mind. + +## Decision points + +### D1: Architecture + +**(a) Post-baseline anchored stage (chosen).** A new type `survival` (rtype `date`). Both backends skip it in baseline generation, as they skip `formula` (`R/mock_spec_native.R:87,405`; `R/mock_spec_simstudy.R:137,260`). A new exported `generate_survival_dates(data, spec, seed = NULL)` computes the survival dates from the anchor columns in `data`, then applies the consistency rules. Postprocess then treats them like any other column. + +**(b) Multi-column generator (rejected).** One spec entry returning a data frame of two or more columns. This changes the one-column-per-variable contract in the native backend, the simstudy backend, postprocess and diagnostics. It also forces an awkward ownership question for a five-date family, since the entry date would be both a plain variable and a member of the group. + +**(c) Delegate to `create_wide_survival_data()` (rejected).** This bypasses spec validation, diagnostics, the seed contract and postprocess, and makes a deprecated function part of the v0.4 pipeline. + +### D2: Scope + +Full parity with the five legacy roles and both rules. No time-to-event or status columns are generated; the legacy engine produces none. D8 lets users derive them with `mockFormula`. + +### D3: Metadata convention + +| Column | On | Meaning | +|---|---|---| +| `anchor` | `variables.csv` | Name of the date variable this date is generated relative to. A non-blank `anchor` makes the variable type `survival` | +| `censored_by` | `variables.csv` | Optional. Name of a competing survival date. Where that date is earlier than this one, this one becomes `NA` | + +The minimal example becomes: + +``` +variable anchor censored_by followup_min followup_max event_prop distribution +interview_date uniform +primary_event_date interview_date death_date 0 5475 0.3 gompertz +death_date interview_date 365 7300 0.2 gompertz +ltfu_date interview_date 365 7300 0.1 uniform +admin_censor_date interview_date 365 7300 1.0 +``` + +A single `survival_role` column (`entry`, `event`, `death`, `ltfu`, `admin_censor`) was considered and rejected. It reproduces the legacy function's five argument slots, which fix one entry date and one competing-risk rule. `anchor` and `censored_by` express the same case and extend to a second cohort or a second competing risk without a new vocabulary. `censored_by` also makes the competing-risk rule visible in metadata. + +**Placement.** The columns are unprefixed and sit on `variables.csv`, beside the existing survival parameters. The rule, to be written into `reference-config.qmd`, is: MockData extension columns on `variables.csv` are unprefixed; MockData columns on `variable_details.csv`, where they sit among recodeflow's semantic columns, carry the `mock` prefix (as `mockFormula` does). A MockData-owned sidecar that keeps recodeflow files free of extension columns is a separate v0.6 decision (see Future directions); the spec layer does not depend on where columns are read from, so that change will not affect this design. + +**Validation of the convention.** A date variable with any of `followup_min`, `followup_max` or `event_prop` but no `anchor` is an error at spec construction, naming the fix (`anchor = ""`). A variable with `anchor` but without all three parameters is also an error. + +### D4: Fidelity + +The legacy statistics and rules are ported verbatim: + +1. `n_events = floor(n * event_prop)`; the event indicator is shuffled with `sample()`. +2. Follow-up days for events, by `distribution`. **Uniform:** `runif(n_events, followup_min, followup_max)`. **Exponential:** `rexp(n_events, rate = 1 / ((followup_max - followup_min) / 3)) + followup_min`, capped at `followup_max`. **Gompertz:** `(1/shape) * log(1 - (shape/rate) * log(1 - u))` with `u ~ runif`, clamped to the window; `shape` defaults to 0.1 and `rate` to 0.0001. +3. Date = anchor + `floor(follow-up days)` for events; `NA` otherwise. The legacy engine produces whole days because `create_date_var()` converts dates to character and back (`R/create_date_var.R:479`), which floors; the port floors explicitly. +4. Any date earlier than its anchor becomes `NA`; then, if the variable has `censored_by`, rows where the censoring date is earlier become `NA`. + +Survival variables are processed one at a time in dependency order over `depends_on` (D7), using the #39 ordering: repeated passes in spec order, each taking the variables whose dependencies are already placed. Each variable is drawn and then has its own rules applied. Because chained censoring is rejected (D7), a date named in another's `censored_by` is never itself censored, so every rule compares against an unchanged drawn date. For the legacy rule set this gives the legacy results. The draw order differs from spec order where `censored_by` points forward (in the minimal example, `death_date` is drawn before `primary_event_date`); that has no comparability cost, because the RNG streams differ from the legacy engine's anyway. + +The fixed-count allocation in step 1 is the legacy generation method, kept in v0.5 for parity. It does not define survival generation in general: a future hazard model, such as one where risk depends on exposure, must let event occurrence and timing follow that model, which a fixed count of `floor(n * event_prop)` events cannot do. Such a model would be a second generation method beside this one. + +Output cannot be bit-identical to `create_wide_survival_data()`, because the RNG streams differ (#38). Parity is therefore tested structurally and distributionally (Test strategy). Two further differences are inherent in the pipeline: + +| Legacy | New pipeline | Reason | +|---|---|---| +| Missing codes fill the last `n - n_valid` rows; events are split across the first `n_valid` rows (`R/create_date_var.R:295,417`) | Postprocess assigns missing codes at random rows | Missingness belongs to postprocess in v0.4. Proportions match in expectation | +| Garbage is applied inside `create_date_var()`, before the rules, so before-entry garbage is erased | Rules run first; garbage afterwards (D5) | Stage order | + +### D5: Contamination after the consistency rules + +The consistency rules produce clean, analysis-ready dates. Missing codes and garbage are applied afterwards, by postprocess, as for every other variable. This matches #39, where formulas are computed on clean baseline values before contamination. + +It changes one behaviour. In the legacy engine, garbage that places a date before entry is set to `NA` by the before-entry rule; `test-survival-garbage-deprecation.R:147` acknowledges this. The garbage tutorial states that survival garbage exists to create temporal violations such as death before entry, for testing cleaning pipelines. Under D5 those violations survive, so the pipeline does what the tutorial describes. This is a change of stage order, not of any rule. Ratified 2026-09-24. + +One consequence: the rules use the true dates, and postprocess may then give a missing code to a date that censored another. The output can therefore show an event censored by a death that is itself recorded as missing. This mirrors real data, where a death is known to the registry but missing from the analysis file, and it is pinned by a test so it stays deliberate. + +The pipeline therefore produces three layers, which the documentation names explicitly: + +- **Clean truth:** the generated dates, and any formula columns derived from them, before contamination. Available by running the stages directly: `generate_mock_data_native()`, then `generate_survival_dates()`, then `evaluate_mock_formulas()`. Keep that data frame before calling `postprocess_mock_data()`. +- **Observed data:** the output of `postprocess_mock_data()` (and of `create_mock_data()`), after missing codes and garbage. +- **Analysis variables:** status, follow-up time and similar quantities that the downstream analysis recalculates from the observed data. MockData does not produce this layer; a formula column computed from clean truth is not a substitute for it. + +D5 applies contamination between the first and second layers. It is the right default for testing cleaning pipelines and for simulating measurement error, and it is misleading only if someone assumes that formula columns were computed from the returned dates. + +### D6: Stage and seed + +`create_mock_data()` runs baseline, survival, formulas, then postprocess. Survival precedes formulas so that a `mockFormula` can use survival dates. `.MOCK_STAGES` (`R/mock_spec_native.R:12`) gains `survival = 4L`. Appending leaves indices 0 to 3 in place, so every existing pinned value holds; the frozen-stage test (`test-seed-contract.R:99`) is extended, not renumbered. Survival draws run inside `.with_mock_seed(seed, stage = "survival")`. + +The fixed order is a v0.5 restriction. It supports `death_date` then follow-up time, but not a hazard that depends on a derived exposure (for example, smoking intensity and duration, then pack-years, then `death_date`, then follow-up time), because pack-years would be computed too late. Lifting it means scheduling derived variables across types in one dependency-ordered pass; that is a scheduler refactor, which would keep the spec, the metadata adapters and the one-column-per-variable contract. + +### D7: Dependencies + +Survival variables set `depends_on = c(anchor, censored_by)`. `.order_formula_variables()` (`R/mock_spec_formula.R:88`) is generalized to order any derived type by `depends_on`, and the formula stage uses the generalized version. The purpose is forward compatibility: a later structural-equation stage, whether in MockData or a separate package, would read a single dependency field from the spec and add a type rather than a mechanism. + +`depends_on` is a scheduling field only. The two relationships it combines have different roles and stay as separate fields on the spec: `anchor` determines how a date is generated; `censored_by` determines what is observed. A future observation layer can therefore treat them differently without a spec change. + +**Chained censoring is rejected in v0.5.** A `censored_by` target may not itself have `censored_by`. With a chain (A censored by B, B censored by C), applying the rules in sequence gives a wrong result: with A on day 30, B on day 20 and C on day 10, B becomes `NA`, so A is no longer compared with an earlier date and is reported as observed on day 30, although observation ended on day 10 (reproduced 2026-09-24). Comparing against unchanged dates instead gives the right answer here only by coincidence. The legacy engine has a single rule and the minimal example no chain, so parity is unaffected. The longer-term design (Future directions) keeps generated event times and derives observed outcomes from all censoring sources at once. + +### D8: Add `is.na` to the formula allow-list + +The #39 allow-list has no missingness test, so a formula cannot derive an event indicator from a survival date. With `is.na` added, status and follow-up time can be derived, but they must come from the same observation window. `as.integer(!is.na(death_date))` only records that a death date exists: for a death on day 50 and administrative censoring on day 20 it gives status 1 beside a follow-up time of 20 days (reproduced 2026-09-24). The correct pair uses the end of observation for both: + +```r +end = pmin(death_date, ltfu_date, admin_censor_date, na.rm = TRUE) +status = as.integer(!is.na(death_date) & death_date == end) +followup_days = as.numeric(end - interview_date) +``` + +(A formula cannot return a `Date`, so `end` is written out inside both formulas.) Ties count as observed events: a death on the censoring date gives status 1. This matches the censoring rule, which removes an event only when the censoring date is strictly earlier. The external review points to simstudy's survival documentation () for the same separation of event times from observed outcomes. The #39 ADR permits widening the list on concrete need; this is one. `is.na` has no side effects and cannot reach outside the sandbox. Ratified 2026-09-24. + +Derived columns describe the true generated values. The #39 stage order computes formulas before postprocess, and postprocess contaminates each column independently: garbage only replaces existing values (`R/mock_spec_postprocess.R:255`), but missing codes may land on any row (`:192`). So a derived `status` of 1 can sit beside a `death_date` shown as missing, and a derived follow-up time reflects the true date where the observed one is garbage. (Verified with #39 code: in 1,000 rows, all 300 rows where a source column shows its missing code keep the formula value computed from the true value.) For testing a cleaning pipeline this is useful, because the derived columns are the answer key. For an analysis-ready teaching dataset it is not, and v0.5 does not offer the alternative; see Future directions. + +### D9: Legacy path + +`create_wide_survival_data()` is already tagged `@keywords deprecated`. v0.5 adds a one-time deprecation warning pointing to metadata-driven generation. The function is retained unchanged: it is the reference for the characterization tests that pin the rules before the port. + +### D10: Legacy generator and `anchor` + +`create_mock_data()` uses the legacy generator when `validate = FALSE`, when `variable_details` is `NULL`, when only `variable_details` has a `databaseStart` column, or when another variable uses a feature the v0.4 pipeline does not support. The legacy dispatcher cannot generate survival dates from metadata; today it warns and drops them. With `anchor` in the metadata, that would silently produce partial survival data. Instead, the legacy path stops, names the survival variables, lists the four conditions that select it, and suggests `verbose = TRUE` to see which applied. This follows the no-backward-compatibility constraint: a loud error that names the fix, rather than plausible-looking partial output. + +## Validation, direct API and diagnostics + +**Validator.** A `survival` branch in `validate_mock_spec()` beside the `date` and `formula` branches (`R/mock_spec.R:968`): + +- `anchor` names a spec variable of type `date` (not `survival` or `formula`; chained anchors are not supported) +- `censored_by`, if set, names a `survival` variable with the same `anchor`, and that variable has no `censored_by` of its own (no chained censoring, D7) +- `followup_min` and `followup_max` finite, non-negative, and `followup_min <= followup_max` +- `event_prop` in [0, 1] +- `distribution` one of `uniform`, `exponential`, `gompertz`; `shape` and `rate` positive when supplied +- `rtype` is `date` +- `sourceFormat` is `analysis` (other formats are not supported for survival dates in v0.5) + +`generate_survival_dates()` strict-validates at entry, like `evaluate_mock_formulas()`, and stops if an anchor column is absent from `data`. + +**Adapter.** `.recodeflow_variable_kind()` (`R/mock_spec_recodeflow.R:117`) returns `survival` for a date row with a non-blank `anchor`. Construction reads `anchor` and `censored_by` beside the existing `followup_*`, `event_prop`, `rate` and `shape` reads. The `recStart` range of a survival date is not used for generation, as in the legacy engine. + +**Direct API.** `mock_spec_survival()` returns a composable variable, used with `mock_spec()` beside its anchor. Arguments: `name`, `anchor`, `followup_min`, `followup_max`, `event_prop`, `distribution = "uniform"`, `censored_by = NULL`, `shape = NULL`, `rate = NULL`, `source_format = "analysis"`, plus the usual missing-code, garbage, provenance and model-hint arguments. There is no one-variable `mock_survival()` counterpart to `mock_formula()`: a spec holding only a survival variable can never validate, because its anchor must be in the same spec. + +**Diagnostics.** Following #39's D5 (`R/mock_spec_postprocess.R:35`): `derived = TRUE`, `anchor`, `censored_by`, `depends_on`, and `n_events`, the number of non-`NA` dates after the consistency rules and before postprocess. + +## Concerns filed as issues (not fixed in #40) + +Found while preparing this design and filed on 2026-09-24. Each is reproduced exactly by the port. + +1. **Gompertz output is degenerate with the packaged parameters** (#54). With `shape = 0.1` and `rate = 1e-04`, time in days, the inverse-CDF never exceeds about 94 days (median 65). In the minimal example every non-garbage `death_date` therefore clamps to `followup_min`, exactly 365 days after entry (388 of 400 deaths in a 2,000-row legacy run; the other 12 are the 3 per cent high garbage). Every non-garbage `primary_event_date` falls within about 94 days despite a 15-year window. No death precedes an event, so the competing-risk rule never fires. The survival tutorial is built on this fixture. Whether the fault lies in the parameterization, the time scale, or the fixture values needs a methodological decision before the tutorial is rewritten around `create_mock_data()`. +2. **Administrative censoring is drawn per person** (#55). `admin_censor_date` has `event_prop = 1.0` and a follow-up window, so each person gets a random date between one and 20 years after entry (1,762 distinct values in 2,000 rows). Its `variable_details` row specifies a fixed date, 2024-12-31, which is ignored. Administrative censoring is usually a fixed study end date. +3. **Declared date missingness is ignored** (evidence added to #15). Every missing row for the minimal example's dates is `else` with `NA::b` and a proportion (0.05 for `death_date`). Neither path applies it: the v0.4 adapter skips `else` rows (`R/mock_spec_recodeflow.R:232`), and the legacy `death_date` is exactly 20.0 per cent non-missing, its `event_prop`. Belongs with #15. +4. **The documentation overstates censoring** (#56). `create_wide_survival_data()` documents that "observation ends at min(event, death, ltfu, admin_censor)" (`:61`), but the code applies only death before event. Under D3, users encode additional rules with `censored_by`. +5. **Positional missing-code placement** (#56) in `create_date_var()`: the last rows are the missing ones, so row order carries meaning. +6. **Inert range rows** (#56). The `recStart` range on a survival date's `variable_details` row (for example `[2002-01-01,2021-01-01]`) never bounds anything, which invites the reader to think it does. +7. **Exponential survival dates ignore `rate`** (#56). The generator derives its rate from the window, `1 / ((followup_max - followup_min) / 3)` (`R/create_date_var.R:328`), while `reference-config.qmd` says the exponential distribution requires `rate`. + +## Consequences + +- `create_mock_data()` generates complete survival data from metadata, closing the NEWS known issue from v0.2.0. +- Survival metadata that previously generated silently wrong output now either generates correctly (with `anchor`) or fails validation with a message naming the fix. +- New exports: `generate_survival_dates()` and `mock_spec_survival()`. Additions to `_pkgdown.yml` in the same PR. +- The packaged minimal example now generates entirely through the v0.4 pipeline. Its two Gompertz dates were its only legacy-fallback triggers, so every vignette and test that runs `create_mock_data()` on it sees different values. +- The formula allow-list grows by one function (D8). +- Specs without survival variables: output unchanged. +- Derived columns (formulas, survival status or time) describe true values; contamination is applied to each column independently (D8). Documented in the formula and survival vignettes. + +## Test strategy + +1. **Characterization first.** Pin the legacy engine's two rules and its structural invariants on `create_wide_survival_data()` before any port code, as the reference. +2. **Validator and adapter.** One failing case, with its message, for each rule in "Validation"; the missing-anchor error. +3. **Generation.** Event proportion within two-sided binomial bounds for each distribution (#23's direction); every non-`NA` date at or after its anchor; `censored_by` sets `NA` exactly where the censoring date is earlier and nowhere else; typed zero-row output for `n = 0`; skipping the stage makes postprocess fail on the missing column. +4. **Contract.** Frozen-stage test extended with `survival = 4L`; pinned reference values for one survival spec; all existing pins untouched. +5. **End to end.** `create_mock_data()` on the amended minimal example produces all five dates with no fallback; the simstudy backend produces the same survival columns; a `mockFormula` derives status and follow-up time from them (D8); D5 is pinned by a test showing before-entry garbage survives, and by a test showing an event censored by a death that postprocess then records as missing. +6. **Draw order.** A test pins that survival draws follow dependency order with position tie-breaking, so a later change to either is visible. + +## Migration + +- Metadata: add `anchor` (and `censored_by` where a competing risk applies) to each survival date's row. Validation names the missing column. +- Code calling `create_wide_survival_data()` keeps working, with a deprecation warning. Replace it with `create_mock_data()` on metadata carrying `anchor`. +- Seeded survival output differs from the legacy function's; regenerate pinned expected values once. +- NEWS: feature entry; removal of the v0.2.0 known-issue entry with a migration note; the D5 behaviour change stated explicitly. + +## Future directions + +- **Sidecar overlay (v0.6, separate ADR; #57).** A MockData-owned table keyed by variable, joined onto `variables` before spec construction, so all extension columns can live outside recodeflow files. The legacy `mock_data_config.csv` readers from v0.2 are the starting point to reconcile or retire. +- **Observation layer.** Keep the generated event times and derive observed outcomes (status, time, cause) from all censoring sources at once, instead of overwriting censored dates with `NA`. This handles several censoring sources per event cleanly and removes the chaining problem in D7. An option to compute formulas from observed rather than clean values (D5) belongs with it. +- **Structural (causal) generation.** Out of scope. The maintainer's motivating case is exposure-dependent survival, such as smoking changing the hazard. Assessed on 2026-09-24, with corrections from an external review: + + | Use case | What would need adding | Architectural implication | + |---|---|---| + | Baseline smokers have a different hazard | A survival model using each person's exposure and covariates, with event occurrence following the model rather than the fixed allocation (D4) | Fits the existing spec and generator pattern | + | Hazard depends on derived pack-years | Formulas evaluated before the survival variables that need them | Scheduling refactor (D6) | + | Someone quits smoking during follow-up | Exposure history, and a hazard that changes over time using only the exposure known at each time; applying final smoking status retrospectively would use information from the future | Substantial generator extension; the core spec may remain | + | Illness changes smoking, which changes later outcomes | Sequential updates to exposures and health states | A longitudinal simulation mechanism; the current execution architecture cannot be assumed to hold | + + A causal interpretation of any of these would also need explicit assumptions about confounding and interventions, which the metadata does not yet express. + + Future exposure-dependent survival models should reuse the spec, the metadata adapters and the column assembly contract. They may need additional model parameters, dependency scheduling across variable types, and separate event-generation and observation rules. The fixed stage order and the legacy event-allocation method are v0.5 restrictions. diff --git a/development/plans/2026-09-24-survival-dates.md b/development/plans/2026-09-24-survival-dates.md new file mode 100644 index 0000000..e26fa24 --- /dev/null +++ b/development/plans/2026-09-24-survival-dates.md @@ -0,0 +1,2660 @@ +# Metadata-Driven Survival Dates Implementation Plan (#40) + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** `create_mock_data()` generates entry, event, death, loss-to-follow-up and administrative-censoring dates from metadata alone, by computing `type = "survival"` variables from their anchor dates in a post-baseline stage. + +**Architecture:** A new spec type `survival` (rtype `date`). Both backends skip derived types (formula, survival) in baseline generation. A new exported `generate_survival_dates(data, spec, seed)` processes survival variables one at a time in dependency order (draw, then the before-anchor and `censored_by` rules) under `.with_mock_seed(stage = "survival")`. `create_mock_data()` runs baseline → survival → formulas → postprocess. The recodeflow adapter builds survival variables from `variables.csv` rows that set `anchor`. The legacy engine is unchanged apart from a deprecation warning, and the legacy dispatcher stops on anchored metadata. + +**Tech stack:** base R (no new dependency), testthat 3e (3.2.3), roxygen2. + +**Spec:** [development/adr/v05-survival-dates.md](../adr/v05-survival-dates.md) (ACCEPTED 2026-09-24, D1–D10; amended the same day after an external design review: chained censoring rejected, D8's status/time example corrected, data layers named). Issues: #40; concerns #54, #55, #56, #57 and #15; prerequisite #58 (PR #59). + +**Base:** branch `v05-survival-dates` at `2548b51`: `v05-formula-evaluator` (#52, which carries #51) + the ADR commits + PR #59 (#58 range missing codes and minimal-example fixture corrections) merged in. Do not start until `git log --oneline -1` shows `2548b51` or a descendant. + +**Prototype evidence (2026-09-24):** the Task 1 characterization tests were run against the legacy engine and pass; the Task 3 draw routine, rules and ordering were prototyped and pass 11 parity checks (competing risk both directions, exact event count, whole days inside the window, Gompertz #54 parity, exponential bounds, `n = 0`, flooring to zero events, `NA` anchors, draw order, cycle error). The Task 3 pinned values were computed independently by emulating `.with_mock_seed()` for stage index 4, not captured from the implementation. + +## Global Constraints + +- **Zero seeded-output change for specs without survival variables.** The #48 pins in `tests/testthat/test-seed-contract.R` (postprocess stage pin, pinned reference values) must pass untouched. If one fails, the code is wrong: fix the code, never the pin. The only permitted edit to that file's existing tests is extending the frozen-stage vector (Task 3) and dropping anchored rows in the two legacy-path tests (Task 4). +- **Survival draws use only `.MOCK_STAGES["survival"]`**, appended as `4L`. Never renumber indices 0–3. +- **Exact port (ADR D4).** The statistics and rules in Task 3 Step 3 come from `R/create_date_var.R:295-346` and `R/create_wide_survival_data.R:347,363`. Do not fix anything the ADR lists under Concerns (#54, #55, #56, #15), even where it looks wrong. Tests that pin those behaviours say so in a comment. +- **recodeflow semantics untouched.** `anchor` and `censored_by` are MockData extension columns on `variables.csv`. +- **CSV fixtures change only through R data-frame operations** (`read.csv`/`write.csv(row.names = FALSE, na = "")`), never sed, awk or hand edits (user CLAUDE.md). `write.csv` reproduces the minimal example byte for byte (verified). +- **No backward-compatibility obligation (0.x)**, but every new error names the fix, and every new error message is asserted by a test with a regex on a distinctive phrase. +- **Commits:** imperative subject in the repo's style with a `(#40)` suffix, wrapped body; no AI attribution (user CLAUDE.md overrides). Commit locally; do not push. +- **Test counts:** report all testthat columns, including `error`. Summing only `failed` hid #50. Use the snippet in Task 1 Step 1. +- **Environment:** the renv library lives under `~/Library/Caches` and macOS purges it. If `library(devtools)` fails, run `SDKROOT=$(xcrun --sdk macosx --show-sdk-path) Rscript -e 'renv::restore(prompt = FALSE)'` first. `devtools::check()` fails on this Mac's toolchain; use `R CMD build` then `R CMD check --as-cran --no-manual` on the tarball (Task 7). +- **Style for prose (NEWS, vignettes, roxygen):** Canadian Press spelling with -ize, sentence-case headings, "per cent" in running prose, no Unicode dashes in source (`~/github/ai-infrastructure/context/global_style.md`). + +## Review Focus + +1. **Legacy route with anchored metadata** (`validate = FALSE`, or another variable forcing the fallback): expect an error that names the survival variables. Test: Task 4, `legacy generator stops on anchored metadata instead of dropping survival dates`. +2. **Anchor not enabled** (role filtering drops the entry date but keeps its survival dates): expect a validation error naming the anchor, surfaced by `create_mock_data()`. Test: Task 4, `an anchor that is not enabled fails with a message naming it`. +3. **Character anchor column in standalone use** (a user reads dates from CSV and calls `generate_survival_dates()` directly): expect an error naming the column and its class. Test: Task 3, `generate_survival_dates rejects a non-Date anchor column`. +4. **Small n with a small event_prop** (n = 3, event_prop = 0.2, so `floor()` gives zero events): expect an all-`NA` `Date` column and `n_events = 0` in diagnostics, with no error or warning. Test: Task 3, `zero events from flooring is an all-NA column, not an error`. +5. **Mutual `censored_by`** (A censored by B, B censored by A): expect a dependency-cycle error at validation. Test: Task 2, `mutual censored_by is reported as a dependency cycle`. + +--- + +### Task 1: Characterize the legacy survival engine + +Pins what the port must reproduce, before any code changes. These tests describe current legacy behaviour and stay green throughout; the legacy engine is not changed (Task 6 adds a warning, which the Task 6 test helper pre-empts). + +**Files:** +- Create: `tests/testthat/test-survival-characterization.R` + +**Interfaces:** +- Consumes: `create_wide_survival_data()` (unchanged) +- Produces: nothing used by later tasks; the scenarios are mirrored against the new stage in Task 3. + +- [x] **Step 1: Record the baseline** + +```bash +cd /Users/dmanuel/github/mock-data && Rscript -e ' +df <- as.data.frame(devtools::test(reporter = "silent", stop_on_failure = FALSE)) +cat(sprintf("FAIL %d | ERROR %d | WARN %d | SKIP %d | PASS %d\n", + sum(df$failed), sum(df$error), sum(df$warning), sum(df$skipped), sum(df$passed))) +bad <- df[df$failed > 0 | df$error, c("file", "test")]; if (nrow(bad)) print(bad)' +``` + +Expected at `2548b51` with simstudy installed: `FAIL 0 | ERROR 0 | WARN 35 | SKIP 3 | PASS 811`. Record the actual line; every later task compares against it. + +- [x] **Step 2: Write the characterization tests** + +```r +# tests/testthat/test-survival-characterization.R +# Characterization of the legacy survival engine (create_wide_survival_data()), +# pinned before the #40 port. ADR: development/adr/v05-survival-dates.md (D4). +# These describe current behaviour, including behaviour filed as concerns +# (#54, #56); they are the reference the metadata-driven stage must match. + +legacy_fixture <- function(event_window, death_window, + event_prop = 1, death_prop = 1) { + list( + variables = data.frame( + variable = c("entry", "event", "death"), + variableType = c("Date", "Date", "Date"), + rType = c("date", "date", "date"), + role = c("enabled", "enabled", "enabled"), + distribution = c("uniform", "uniform", "uniform"), + followup_min = c(NA, event_window[1], death_window[1]), + followup_max = c(NA, event_window[2], death_window[2]), + event_prop = c(NA, event_prop, death_prop), + sourceFormat = c("analysis", "analysis", "analysis"), + stringsAsFactors = FALSE + ), + variable_details = data.frame( + variable = c("entry", "event", "death"), + recStart = c("[2001-01-01,2001-12-31]", + "[2001-01-01,2040-12-31]", + "[2001-01-01,2040-12-31]"), + recEnd = c("copy", "copy", "copy"), + proportion = c(1, 1, 1), + stringsAsFactors = FALSE + ) + ) +} + +run_legacy <- function(fx, n = 200, seed = 1) { + suppressWarnings(create_wide_survival_data( + var_entry_date = "entry", + var_event_date = "event", + var_death_date = "death", + databaseStart = "test", + variables = fx$variables, + variable_details = fx$variable_details, + n = n, + seed = seed + )) +} + +days_from_entry <- function(result, column) { + as.numeric(result[[column]] - result$entry) +} + +test_that("legacy: an event later than death is set to NA (competing risk)", { + result <- run_legacy(legacy_fixture(c(1000, 2000), c(0, 500))) + expect_true(all(is.na(result$event))) + expect_false(any(is.na(result$death))) +}) + +test_that("legacy: an event earlier than death is kept", { + result <- run_legacy(legacy_fixture(c(0, 500), c(1000, 2000))) + expect_false(any(is.na(result$event))) +}) + +test_that("legacy: exactly floor(n * event_prop) rows receive an event", { + result <- run_legacy(legacy_fixture(c(100, 200), c(100, 200), + event_prop = 0.3, death_prop = 0), + n = 200) + expect_identical(sum(!is.na(result$event)), 60L) + expect_true(all(is.na(result$death))) +}) + +test_that("legacy: follow-up days are whole days inside the window", { + result <- run_legacy(legacy_fixture(c(100, 200), c(5000, 6000))) + days <- days_from_entry(result, "event") + days <- days[!is.na(days)] + expect_true(all(days == floor(days))) + expect_true(all(days >= 100 & days <= 200)) +}) + +test_that("legacy: on the minimal example no date precedes the entry date", { + vars <- system.file("extdata", "minimal-example", "variables.csv", package = "MockData") + dets <- system.file("extdata", "minimal-example", "variable_details.csv", package = "MockData") + if (!nzchar(vars) || !nzchar(dets)) skip("minimal-example fixtures not installed") + variables <- read.csv(vars, stringsAsFactors = FALSE, check.names = FALSE) + variable_details <- read.csv(dets, stringsAsFactors = FALSE, check.names = FALSE) + result <- suppressWarnings(create_wide_survival_data( + var_entry_date = "interview_date", + var_event_date = "primary_event_date", + var_death_date = "death_date", + var_ltfu = "ltfu_date", + var_admin_censor = "admin_censor_date", + databaseStart = "minimal-example", + variables = variables, + variable_details = variable_details, + n = 2000, + seed = 1 + )) + for (column in c("primary_event_date", "death_date", "ltfu_date", "admin_censor_date")) { + ok <- !is.na(result[[column]]) + expect_true(all(result[[column]][ok] >= result$interview_date[ok]), info = column) + } + # #54: with the packaged Gompertz parameters every non-garbage death lands + # exactly at followup_min (365 days). Garbage deaths are 2025 or later. + # Update this expectation deliberately when #54 is resolved. + days <- as.numeric(result$death_date - result$interview_date) + clean <- !is.na(days) & result$death_date < as.Date("2025-01-01") + expect_true(any(clean)) + expect_true(all(days[clean] == 365)) +}) +``` + +- [x] **Step 3: Run them — expected PASS (characterization)** + +Run: `Rscript -e 'devtools::load_all(quiet = TRUE); testthat::test_file("tests/testthat/test-survival-characterization.R")'` +Expected: 5 tests, 13 expectations, all PASS. If any fails, stop: the legacy engine differs from what the plan assumes; report rather than editing the test. + +- [x] **Step 4: Commit** + +```bash +git add tests/testthat/test-survival-characterization.R +git commit -m "Characterize the legacy survival engine before the port (#40)" -m "Pins the create_wide_survival_data() behaviour the metadata-driven stage +must reproduce: the death-censors-event rule in both directions, the exact +floor(n * event_prop) event count, whole-day follow-up inside the window, +no date before entry on the minimal example, and the #54 Gompertz clamp." +``` + +--- + +### Task 2: Survival spec layer + +**Files:** +- Create: `R/mock_spec_survival.R` +- Modify: `R/mock_spec_formula.R:87-126` (generalize ordering) +- Modify: `R/mock_spec.R` (constructor after `mock_spec_formula()`, which ends near line 746; validator branch before the unsupported-type `else` near line 992; spec-level checks in `validate_mock_spec()` after the formula block near line 1046) +- Test: `tests/testthat/test-survival-vars.R` (new) + +**Interfaces:** +- Consumes: `.new_mock_spec_variable()`, `.is_formula_variable()`, `.formula_dependencies()`, `.is_blank()` (defined in `R/mock_spec_recodeflow.R:34`), `` `%||%` `` (`R/mock_spec.R:45`) +- Produces: + - `mock_spec_survival(name, anchor, followup_min, followup_max, event_prop, distribution = "uniform", censored_by = NULL, shape = NULL, rate = NULL, source_format = "analysis", missing_codes = character(0), missing_proportions = numeric(0), garbage_rules = list(), provenance = "direct", model_hint = "native-postprocess")` → `mock_spec_variable` with `type = "survival"`, `rtype = "date"`, fields `anchor`, `censored_by`, `followup_min`, `followup_max`, `event_prop`, `shape`, `rate`, `depends_on = c(anchor, censored_by)` + - `.is_survival_variable(variable)` → logical; `.is_derived_variable(variable)` → TRUE for `formula` or `survival` + - `.survival_dependencies(variable)` → character + - `.validate_survival_variable(variable)` → character vector of errors + - `.validate_survival_referents(spec)` → character vector of errors (includes the chained-censoring rejection) + - `.order_dependent_variables(names_in_scope, dependencies, label)` → character (stops with `"