Metadata-driven survival dates in create_mock_data() (#40)

CONTRIBUTING.md

@@ -131,6 +131,30 @@ covr::package_coverage()
 - If adding new test data, document its purpose
 - Keep test data small and focused
 
+#### Vignettes are integration tests
+
+The vignettes run MockData's own code on realistic metadata, and `R CMD check` rebuilds every one of them in CI. That makes them integration tests, but only if a vignette fails when the code stops doing what its prose says. A vignette that prints a number without checking it renders happily after a regression, and a number typed into the prose goes stale without anyone noticing.
+
+When you write or change a vignette:
+
+- Compute every factual claim with inline R (`` `r n_deaths` ``) instead of typing the number into the prose.
+- Guard the claims with a hidden chunk of named `stopifnot()` conditions, so the render, and therefore `R CMD check` and CI, fails with a message naming the broken claim:
+
+  ````markdown
+  ```{r}
+  #| include: false
+  stopifnot(
+    "deaths are exactly floor(n * event_prop)" = n_deaths == floor(n * 0.3),
+    "no event is recorded after a death" = events_after_death == 0
+  )
+  ```
+  ````
+
+- State known issues as claims too, with the issue number in the message. When the issue is fixed, the render fails until the documentation is updated.
+- For a new feature, check that the assertions can fail: break the behaviour in a scratch copy and check that the render stops.
+
+The survival vignettes (`tutorial-survival-data`, `survival-dates-v05`, `survival-design-v05`) follow this pattern.
+
 ### Building Documentation
 
 #### Prerequisites for documentation builds

NAMESPACE

@@ -18,6 +18,7 @@ export(extract_proportions)
 export(generate_garbage_values)
 export(generate_mock_data_native)
 export(generate_mock_data_simstudy)
+export(generate_survival_dates)
 export(get_cycle_variables)
 export(get_enabled_variables)
 export(get_raw_var_dependencies)
@@ -39,6 +40,7 @@ export(mock_spec_continuous)
 export(mock_spec_date)
 export(mock_spec_formula)
 export(mock_spec_from_recodeflow)
+export(mock_spec_survival)
 export(parse_range_notation)
 export(parse_variable_start)
 export(postprocess_mock_data)

NEWS.md

@@ -11,6 +11,48 @@
   unchanged: derived variables without a `mockFormula` remain excluded from
   generation, and `Func::` dispatch is not yet supported.
 
+## Survival dates in create_mock_data() (#40)
+
+- `create_mock_data()` now generates survival dates from metadata. A date
+  whose `variables.csv` row sets `anchor` (its entry-date variable) is a
+  survival date: `floor(n * event_prop)` rows receive a date between
+  `followup_min` and `followup_max` days after the anchor, drawn from a
+  uniform, exponential or Gompertz distribution, and the rest are `NA`. An
+  optional `censored_by` column names a competing survival date, such as
+  death, that sets this date to `NA` where it comes first. The statistics and
+  rules are those of `create_wide_survival_data()`, ported exactly. New
+  functions: `mock_spec_survival()` and `generate_survival_dates()`. This
+  resolves the known issue, listed since v0.2.0, that survival data had to be
+  generated separately.
+- Migration: add `anchor` (and `censored_by` where a competing risk applies)
+  to each survival date's row. A date with `followup_min`, `followup_max` or
+  `event_prop` but no `anchor` now fails with a message naming the fix;
+  previously it was silently dropped or generated as a plain calendar date.
+- Missing codes and garbage are applied to survival dates after the survival
+  rules. Garbage that places a date before entry is therefore kept, whereas
+  the legacy engine set such dates to `NA`.
+- Derived columns, from `mockFormula` or survival dates, describe the clean
+  generated values; missing codes and garbage are then applied to each column
+  independently. `is.na` is added to the `mockFormula` allow-list so status
+  and follow-up time can be derived. Derive both from the same observation
+  window: `as.integer(!is.na(death_date))` only says a death date exists,
+  not that the death was observed before censoring.
+- A `censored_by` target may not itself have `censored_by`. Chained censoring
+  would report some events as observed after observation ended, so it is
+  rejected in this version with a message naming the fix.
+- The packaged minimal example now generates entirely through the v0.4
+  pipeline, because its Gompertz survival dates no longer force the legacy
+  generator. Its seeded output differs from v0.4.
+- If the legacy generator is selected (`validate = FALSE`,
+  `variable_details = NULL`, detail-level-only `databaseStart`, or another
+  unsupported variable) on metadata that sets `anchor`, `create_mock_data()`
+  stops and names the survival variables rather than dropping them.
+- `create_wide_survival_data()` is deprecated and warns once per session.
+- Known issues found while porting, reproduced unchanged: #54 (Gompertz
+  follow-up times are degenerate with the packaged parameters), #55
+  (administrative censoring is drawn per person), #56 (smaller legacy
+  discrepancies).
+
 ## Reproducibility (breaking change)
 
 - Seeded output changed once for the RNG-stream mechanism in this release.
@@ -368,8 +410,8 @@ vars_with_garbage <- variables %>%
 
 ## Known issues
 
-- Survival variable type must be generated manually with `create_wide_survival_data()`
-- Cannot be used in `create_mock_data()` batch generation (requires paired variables)
+- Survival variable type must be generated manually with `create_wide_survival_data()` (resolved in 0.5.0, #40)
+- Cannot be used in `create_mock_data()` batch generation (requires paired variables) (resolved in 0.5.0, #40)
 
 ---
 

R/create_mock_data.R

@@ -18,6 +18,11 @@
     if (variable$type == "formula") {
       return(FALSE)
     }
+    # type = "survival" is supported end-to-end (#40): both backends skip it,
+    # generate_survival_dates() computes it, postprocess treats it as a date.
+    if (variable$type == "survival") {
+      return(FALSE)
+    }
 
     formula <- variable$formula
     has_formula <- !is.null(formula) &&
@@ -43,6 +48,23 @@
   names(spec$variables)[unsupported]
 }
 
+#' @noRd
+.has_survival_metadata <- function(variables) {
+  # A row sets anchor, or carries survival parameters (followup_min,
+  # followup_max, event_prop): the metadata describes survival dates.
+  if ("anchor" %in% names(variables)) {
+    anchors <- as.character(variables$anchor)
+    if (any(!is.na(anchors) & trimws(anchors) != "")) {
+      return(TRUE)
+    }
+  }
+  survival_fields <- intersect(c("followup_min", "followup_max", "event_prop"), names(variables))
+  any(vapply(survival_fields, function(field) {
+    values <- as.character(variables[[field]])
+    any(!is.na(values) & trimws(values) != "")
+  }, logical(1)))
+}
+
 #' @noRd
 .create_mock_data_v04_native_supported <- function(spec) {
   length(.create_mock_data_v04_unsupported_variables(spec)) == 0
@@ -76,13 +98,27 @@
       role = "enabled"
     ),
     error = function(e) {
+      # Survival dates are generated only by the v0.4 pipeline (ADR
+      # v05-survival-dates D10): validate = FALSE would stop on anchored
+      # metadata or silently drop survival dates, so it is not offered then.
+      advice <- if (.has_survival_metadata(variables)) {
+        paste0(
+          "Fix the metadata as the message above describes. Survival dates ",
+          "are generated only by the v0.4 pipeline, so validate = FALSE is ",
+          "not a workaround for survival dates."
+        )
+      } else {
+        paste0(
+          "Fix the metadata (for exponential variables, supply a positive ",
+          "'rate'), or call create_mock_data() with validate = FALSE to use ",
+          "the legacy generator, which warns and substitutes a uniform draw ",
+          "for invalid distribution parameters."
+        )
+      }
       stop(
         conditionMessage(e), "\n",
         "Metadata validation failed while building the v0.4 specification. ",
-        "Fix the metadata (for exponential variables, supply a positive 'rate'), ",
-        "or call create_mock_data() with validate = FALSE to use the legacy ",
-        "generator, which warns and substitutes a uniform draw for invalid ",
-        "distribution parameters.",
+        advice,
         call. = FALSE
       )
     }
@@ -105,11 +141,14 @@
   }
 
   baseline <- generate_mock_data_native(spec, n = n, seed = seed)
-  staged <- evaluate_mock_formulas(baseline, spec, seed = seed)
-  # Baseline generation, formula evaluation, and post-processing use distinct
-  # L'Ecuyer-CMRG sub-streams derived from the single public seed (see
-  # .with_mock_seed / ADR v05-seed-contract), so all three stages pass the
-  # same seed and select their own stage internally.
+  dated <- generate_survival_dates(baseline, spec, seed = seed)
+  staged <- evaluate_mock_formulas(dated, spec, seed = seed)
+  # Baseline generation, survival dates, formula evaluation, and
+  # post-processing use distinct L'Ecuyer-CMRG sub-streams derived from the
+  # single public seed (see .with_mock_seed / ADR v05-seed-contract), so all
+  # four stages pass the same seed and select their own stage internally.
+  # Survival precedes formulas so a mockFormula can use survival dates
+  # (ADR v05-survival-dates D6).
   postprocess_mock_data(staged, spec, seed = seed)
 }
 
@@ -169,7 +208,10 @@
 #' when `variable_details = NULL`, when detail-level `databaseStart` filtering is
 #' needed but the variables metadata has no `databaseStart` column, or when a
 #' variable uses a feature not yet supported by the v0.4 native backend. Set
-#' `verbose = TRUE` to see which path was chosen.
+#' `verbose = TRUE` to see which path was chosen. Survival dates (rows that
+#' set `anchor`) are generated only by the v0.4 pipeline: if the legacy path
+#' is selected for metadata that sets `anchor`, `create_mock_data()` stops and
+#' names the survival variables.
 #'
 #' In the v0.4 path, baseline generation and post-processing draw from distinct,
 #' independent sub-streams derived from a single `seed`; output is reproducible
@@ -206,9 +248,9 @@
 #' see \code{vignette("reference-config", package = "MockData")}.
 #'
 #' @examples
-#' # The packaged minimal example includes deliberately messy metadata
-#' # (auto-normalized proportions, survival dates without an anchor): the
-#' # warnings it generates are expected and demonstrate MockData's diagnostics.
+#' # The packaged minimal example covers every variable type, including
+#' # survival dates anchored on interview_date. It auto-normalizes some
+#' # proportions, so warnings about that are expected.
 #' mock_data <- create_mock_data(
 #'   databaseStart = "minimal-example",
 #'   variables = system.file("extdata/minimal-example/variables.csv",
@@ -225,12 +267,16 @@
 #' # Columns with straightforward metadata generate cleanly:
 #' head(mock_data[, c("age", "smoking", "interview_date")])
 #'
-#' # Fallback mode: no variable_details, simple default generators
+#' # Fallback mode: no variable_details, simple default generators. Survival
+#' # dates need the v0.4 pipeline, so drop the rows that set an anchor first.
+#' fallback_variables <- read.csv(
+#'   system.file("extdata/minimal-example/variables.csv", package = "MockData"),
+#'   stringsAsFactors = FALSE,
+#'   check.names = FALSE
+#' )
 #' mock_data <- create_mock_data(
 #'   databaseStart = "minimal-example",
-#'   variables = system.file("extdata/minimal-example/variables.csv",
-#'     package = "MockData"
-#'   ),
+#'   variables = fallback_variables[fallback_variables$anchor == "", ],
 #'   variable_details = NULL,
 #'   n = 500
 #' )
@@ -340,6 +386,27 @@ create_mock_data <- function(databaseStart,
     }
   }
 
+  # ADR v05-survival-dates D10: the legacy dispatcher cannot build survival
+  # dates from anchors (it would warn and drop them), so stop rather than
+  # return plausible-looking partial survival data.
+  if ("anchor" %in% names(enabled_vars)) {
+    anchor_values <- as.character(enabled_vars$anchor)
+    anchored <- enabled_vars$variable[
+      !is.na(anchor_values) & trimws(anchor_values) != ""
+    ]
+    if (length(anchored) > 0) {
+      stop(
+        "Survival date variable(s) ", paste(anchored, collapse = ", "),
+        " (anchor set) are generated only by the v0.4 pipeline, but the ",
+        "legacy generator was selected. It is used when validate = FALSE, ",
+        "when variable_details is NULL, when only variable_details has a ",
+        "databaseStart column, or when another variable uses a feature the ",
+        "v0.4 pipeline does not support. Run with verbose = TRUE to see which.",
+        call. = FALSE
+      )
+    }
+  }
+
   if (nrow(enabled_vars) == 0) {
     stop("No enabled non-derived variables found in configuration. ",
          "Add 'enabled' to the role column for variables you want to generate, ",

R/create_wide_survival_data.R

@@ -1,5 +1,13 @@
+# Session state for once-per-session notices (ADR v05-survival-dates D9).
+.mockdata_state <- new.env(parent = emptyenv())
+
 #' Create wide survival data for cohort studies
 #'
+#' **Deprecated as of MockData 0.5.0.** Add `anchor` (and `censored_by` for a
+#' competing risk) to the survival dates' rows in `variables.csv` and generate
+#' them with [create_mock_data()]; see `vignette("tutorial-survival-data")`.
+#' The function still works and warns once per session.
+#'
 #' Generates wide-format survival data (one row per individual) with up to 5 date
 #' variables (entry, event, death, loss-to-follow-up, administrative censoring).
 #' Applies temporal ordering constraints and supports garbage data generation for
@@ -175,6 +183,18 @@ create_wide_survival_data <- function(var_entry_date,
                                        n,
                                        seed = NULL,
                                        prop_garbage = NULL) {
+  if (!isTRUE(.mockdata_state$wide_survival_warned)) {
+    .mockdata_state$wide_survival_warned <- TRUE
+    warning(
+      "create_wide_survival_data() is deprecated as of MockData 0.5.0. Add ",
+      "`anchor` (and `censored_by` for competing risks) to the survival ",
+      "dates' rows in variables.csv and generate them with create_mock_data(); ",
+      "see vignette(\"tutorial-survival-data\", package = \"MockData\"). ",
+      "This warning is shown once per session.",
+      call. = FALSE
+    )
+  }
+
 
   # ========== VALIDATION ==========