diff --git a/DESCRIPTION b/DESCRIPTION index 596ab22..4177b06 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,6 +1,6 @@ Package: MockData Title: Generate Mock Data from Metadata Specifications -Version: 0.4.0 +Version: 0.5.0 Authors@R: c( person("Juan", "Li", role = "aut", email = "juli@ohri.ca"), person("Douglas", "Manuel", role = c("aut", "cre"), email = "dmanuel@ohri.ca"), diff --git a/NEWS.md b/NEWS.md index 52438b6..ac55c6c 100644 --- a/NEWS.md +++ b/NEWS.md @@ -1,18 +1,55 @@ -# MockData (development version) +# MockData 0.5.0 (2026-09-27) -## Formula-derived variables (#39, Phase A) +## Breaking changes -- Variables can now be generated from algebraic expressions over other - generated variables via a new `mockFormula` column in `variable_details` - (e.g. `weight / (height^2)`), or the new `mock_formula()` direct API. - Formulas are evaluated in dependency order by `evaluate_mock_formulas()`, - in a restricted environment exposing only the generated columns and a - fixed allow-list of base functions. `DerivedVar::`/`Func::` semantics are - unchanged: derived variables without a `mockFormula` remain excluded from - generation, and `Func::` dispatch is not yet supported. +- Seeded output changed once for the RNG-stream mechanism in this release. + Within the `create_mock_data()` pipeline (`generate_mock_data_native()`, + `generate_mock_data_simstudy()`, `postprocess_mock_data()`), MockData now + derives all randomness from independent L'Ecuyer-CMRG sub-streams (one per + generation stage) seeded from the single public `seed`, replacing the + previous Mersenne-Twister `seed` / `seed + 1` scheme. The standalone + `create_*` helpers (`create_cat_var()`, `create_con_var()`, + `create_date_var()`, `create_survival_dates()`, + `create_wide_survival_data()`, `sample_with_proportions()`, + `make_garbage()`, `apply_garbage()`) called directly are unaffected and + still seed a single Mersenne-Twister stream per call. For a given seed + **and package version** pipeline output is reproducible and independent of + the session's ambient `RNGkind()`; it is not comparable across the v0.4 → + v0.5 boundary. For seeded calls, generation no longer alters the caller's + RNG state (previously the legacy `validate = FALSE` path reset it). (#38) -## Survival dates in create_mock_data() (#40) + **Migrating:** if your tests pin values from seeded v0.4 runs of + `create_mock_data()` or the `mock_spec` pipeline, regenerate those expected + values once under v0.5 and pin them again. The same seed keeps giving + identical output within this version. Standalone `create_*` calls need no + change. +- Migration: add `anchor` (and `censored_by` where a competing risk applies) + to each survival date's row. A date with `followup_min`, `followup_max` or + `event_prop` but no `anchor` now fails with a message naming the fix; + previously it was silently dropped or generated as a plain calendar date. +- Missing codes and garbage are applied to survival dates after the survival + rules. Garbage that places a date before entry is therefore kept, whereas + the legacy engine set such dates to `NA`. +- The packaged minimal example now generates entirely through the v0.4 + pipeline, because its Gompertz survival dates no longer force the legacy + generator. Its seeded output differs from v0.4. +- If the legacy generator is selected (`validate = FALSE`, + `variable_details = NULL`, detail-level-only `databaseStart`, or another + unsupported variable) on metadata that sets `anchor`, `create_mock_data()` + stops and names the survival variables rather than dropping them. + +## New features + +- Formula-derived variables (#39, Phase A). Variables can now be generated + from algebraic expressions over other generated variables via a new + `mockFormula` column in `variable_details` (e.g. `weight / (height^2)`), + or the new `mock_formula()` direct API. Formulas are evaluated in + dependency order by `evaluate_mock_formulas()`, in a restricted environment + exposing only the generated columns and a fixed allow-list of base + functions. `DerivedVar::`/`Func::` semantics are unchanged: derived + variables without a `mockFormula` remain excluded from generation, and + `Func::` dispatch is not yet supported. - `create_mock_data()` now generates survival dates from metadata. A date whose `variables.csv` row sets `anchor` (its entry-date variable) is a survival date: `floor(n * event_prop)` rows receive a date between @@ -24,60 +61,16 @@ functions: `mock_spec_survival()` and `generate_survival_dates()`. This resolves the known issue, listed since v0.2.0, that survival data had to be generated separately. -- Migration: add `anchor` (and `censored_by` where a competing risk applies) - to each survival date's row. A date with `followup_min`, `followup_max` or - `event_prop` but no `anchor` now fails with a message naming the fix; - previously it was silently dropped or generated as a plain calendar date. -- Missing codes and garbage are applied to survival dates after the survival - rules. Garbage that places a date before entry is therefore kept, whereas - the legacy engine set such dates to `NA`. - Derived columns, from `mockFormula` or survival dates, describe the clean generated values; missing codes and garbage are then applied to each column independently. `is.na` is added to the `mockFormula` allow-list so status and follow-up time can be derived. Derive both from the same observation window: `as.integer(!is.na(death_date))` only says a death date exists, not that the death was observed before censoring. -- A `censored_by` target may not itself have `censored_by`. Chained censoring - would report some events as observed after observation ended, so it is - rejected in this version with a message naming the fix. -- The packaged minimal example now generates entirely through the v0.4 - pipeline, because its Gompertz survival dates no longer force the legacy - generator. Its seeded output differs from v0.4. -- If the legacy generator is selected (`validate = FALSE`, - `variable_details = NULL`, detail-level-only `databaseStart`, or another - unsupported variable) on metadata that sets `anchor`, `create_mock_data()` - stops and names the survival variables rather than dropping them. -- `create_wide_survival_data()` is deprecated and warns once per session. -- Known issues found while porting, reproduced unchanged: #54 (Gompertz - follow-up times are degenerate with the packaged parameters), #55 - (administrative censoring is drawn per person), #56 (smaller legacy - discrepancies). - -## Reproducibility (breaking change) - -- Seeded output changed once for the RNG-stream mechanism in this release. - Within the `create_mock_data()` pipeline (`generate_mock_data_native()`, - `generate_mock_data_simstudy()`, `postprocess_mock_data()`), MockData now - derives all randomness from independent L'Ecuyer-CMRG sub-streams (one per - generation stage) seeded from the single public `seed`, replacing the - previous Mersenne-Twister `seed` / `seed + 1` scheme. The standalone - `create_*` helpers (`create_cat_var()`, `create_con_var()`, - `create_date_var()`, `create_survival_dates()`, - `create_wide_survival_data()`, `sample_with_proportions()`, - `make_garbage()`, `apply_garbage()`) called directly are unaffected and - still seed a single Mersenne-Twister stream per call. For a given seed - **and package version** pipeline output is reproducible and independent of - the session's ambient `RNGkind()`; it is not comparable across the v0.4 → - v0.5 boundary. Pin your own expected values against the version you use. - For seeded calls, generation no longer alters the caller's RNG state - (previously the legacy `validate = FALSE` path reset it). - - `create_mock_data()` now accepts `n = 0`, returning a zero-row data frame with the full generated schema (useful for schema tests), and rejects fractional, negative, `NA`, and non-finite values of `n` with a clear message. The validation now matches `generate_mock_data_native()`. -- Calling `create_mock_data()` without `databaseStart` now fails upfront with - a message naming the argument, instead of a raw missing-argument error. - The native backend now supports `distribution = "exponential"` (parity with the legacy generator), removing a forced legacy-fallback for exponential metadata. When the optional simstudy backend is selected, exponential @@ -86,6 +79,23 @@ their generation. Native exponential values are truncated to the declared `range` by inverse-CDF sampling, rather than clipped at the range maximum (with a point mass at the boundary) as the legacy `rexp()` path does. + +## Validation and error messages + +- Calling `create_mock_data()` without `databaseStart` now fails upfront with + a message naming the argument, instead of a raw missing-argument error. +- With `validate = TRUE` (the default), invalid distribution parameters in + metadata — e.g. `distribution = "exponential"` without a positive `rate` — + now stop generation with a message naming the variable and how to fix it, + instead of warning and substituting a uniform draw. The legacy + warn-and-substitute behaviour remains available via `validate = FALSE`. + +- A `censored_by` target may not itself have `censored_by`. Chained censoring + would report some events as observed after observation ended, so it is + rejected in this version with a message naming the fix. + +## Bug fixes + - The simstudy backend now returns a typed zero-row data frame for `n = 0`, identical to the native backend's output, instead of failing inside `simstudy::genData()` (whose `1:n` id table has two rows when `n = 0`). @@ -103,11 +113,17 @@ and #58, every variable in the example except its survival dates now generates through the v0.4 pipeline. Seeded output for the example changes. -- With `validate = TRUE` (the default), invalid distribution parameters in - metadata — e.g. `distribution = "exponential"` without a positive `rate` — - now stop generation with a message naming the variable and how to fix it, - instead of warning and substituting a uniform draw. The legacy - warn-and-substitute behaviour remains available via `validate = FALSE`. + +## Deprecations + +- `create_wide_survival_data()` is deprecated and warns once per session. + +## Known issues + +- Known issues found while porting, reproduced unchanged: #54 (Gompertz + follow-up times are degenerate with the packaged parameters), #55 + (administrative censoring is drawn per person), #56 (smaller legacy + discrepancies). # MockData 0.4.0 (2026-06-10) diff --git a/README.md b/README.md index f71d571..ba0ea42 100644 --- a/README.md +++ b/README.md @@ -3,19 +3,21 @@ [![Lifecycle: experimental](https://img.shields.io/badge/lifecycle-experimental-orange.svg)](https://lifecycle.r-lib.org/articles/stages.html#experimental) -[![Version: 0.4.0](https://img.shields.io/badge/version-0.4.0-blue.svg)](https://github.com/Big-Life-Lab/MockData) +[![Version: 0.5.0](https://img.shields.io/badge/version-0.5.0-blue.svg)](https://github.com/Big-Life-Lab/MockData) [![pkgdown](https://github.com/Big-Life-Lab/MockData/actions/workflows/pkgdown.yaml/badge.svg)](https://github.com/Big-Life-Lab/MockData/actions/workflows/pkgdown.yaml) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT) -**Status: Experimental v0.4.0 release** +**Status: Experimental v0.5.0 release** MockData is a work-in-progress R package for generating mock testing data from -small metadata specifications. Version 0.4 introduces the +small metadata specifications. Version 0.4 introduced the `mock_spec` architecture: direct specification helpers, a recodeflow metadata adapter, native generation, optional `simstudy` generation, and post-processing -diagnostics. It is useful today for development and documentation workflows, +diagnostics. Version 0.5 adds formula-derived variables (`mockFormula`), +survival dates generated from metadata (`anchor`, `censored_by`), and a +per-stage seed contract. It is useful today for development and documentation workflows, especially when paired with recodeflow-style metadata (see below), but it should be treated as experimental infrastructure rather than a stable released package. @@ -23,14 +25,14 @@ People are using MockData and reporting that it is helpful. We take that as an encouraging signal, not as evidence that the package is mature. Please review the generated data before using it in any workflow that matters. -v0.4 keeps the existing v0.3 public functions available. Several -larger ideas were deliberately deferred rather than rushed into this release: -formula-derived variables, multi-variable correlation, Table 1 bootstrap, -LinkML/schema-first integration, and advanced survival simulation. MockData +v0.5 keeps the existing v0.3 public functions available. Several +larger ideas remain deliberately deferred rather than rushed into a release: +`Func::` dispatch for derived variables, multi-variable correlation, Table 1 +bootstrap, LinkML/schema-first integration, and advanced survival simulation. MockData also remains framed as mock data for development and QA, not synthetic data for privacy release or inference. -Please try v0.4 with representative `variables.csv` and +Please try v0.5 with representative `variables.csv` and `variable_details.csv` files and report bugs, surprising output, confusing diagnostics, or migration issues in [GitHub Issues](https://github.com/Big-Life-Lab/MockData/issues). Broader design @@ -43,7 +45,7 @@ feedback is welcome in - Use inline metadata for small examples, without creating separate CSV files - Create categorical, continuous, and date-like fields from simple specifications - Add intentional invalid or out-of-range "garbage" values for QA testing -- Generate basic time-to-event style mock dates for survival-analysis workflows +- Generate survival dates (entry, events, competing risks, censoring) from metadata - Support examples and tutorials without exposing real person-level data **Current development limitations:** diff --git a/cran-comments.md b/cran-comments.md index 6e74b6a..778e2d1 100644 --- a/cran-comments.md +++ b/cran-comments.md @@ -5,8 +5,10 @@ Checked with: - R 4.4.2 -- macOS 26.5, aarch64-apple-darwin20 -- `R CMD check --as-cran --no-manual` +- macOS 27.0, aarch64-apple-darwin20 +- `R CMD check --as-cran --no-manual` on the built tarball +- `_R_CHECK_FORCE_SUGGESTS_=false`, because the local check library does not + include the suggested package `simstudy` Result: @@ -17,17 +19,23 @@ Result: Notes: - New submission. -- `simstudy` was not installed in the local check environment, so the local - check used `_R_CHECK_FORCE_SUGGESTS_=false`. `simstudy` is listed in - `Suggests` and is used only for the optional backend. +- `simstudy` (suggested) not available for checking locally. It is installed + in continuous integration, where its tests run. - The local check reported `unable to verify current time`. +## Continuous integration + +The `R-CMD-check` workflow runs the same check on Ubuntu 24.04 with +R 4.6.1, with all suggested packages installed, for every push to +`main`/`dev` and every pull request into them. Latest result: `Status: OK` +(0 errors, 0 warnings, 0 notes). + ## Release notes This is the first CRAN submission of MockData. The package generates mock testing data from metadata specifications supplied as data frames or CSV files. -It supports categorical, continuous, date, garbage-data, and survival-style -variables for testing data pipelines and documentation examples. +It supports categorical, continuous, date, garbage-data, formula-derived, and +survival-style variables for testing data pipelines and documentation examples. MockData generates mock data for software testing and workflow validation. It does not generate privacy-preserving synthetic data for inference or data diff --git a/development/post-v040-development-plan.md b/development/post-v040-development-plan.md index d8ade5f..c60d902 100644 --- a/development/post-v040-development-plan.md +++ b/development/post-v040-development-plan.md @@ -573,10 +573,10 @@ Tasks 5 and 6 **must land in the same minor release**: both change seeded output ### Task 8: v0.5 release assembly -- [ ] Tasks 5 + 6 merged together; one NEWS migration section covering the seeded-output break with before/after guidance. -- [ ] Pinned-value seed regression tests in place (#22 closed). -- [ ] `devtools::check()` clean; `roxygen2::roxygenize()` then `pkgdown::build_site()` clean; `renv::snapshot()` if dependencies moved. -- [ ] Version bump to 0.5.0; downstream heads-up to cchsflow/chmsflow maintainers if any metadata-extension columns were added. +- [x] Tasks 5 + 6 merged together; one NEWS migration section covering the seeded-output break with before/after guidance. (#38, #39 and #40 all ship in 0.5.0; NEWS "Breaking changes" carries the migrating paragraph.) +- [x] Pinned-value seed regression tests in place (#22 closed). (`tests/testthat/test-seed-contract.R`; #22 closed by the release PR.) +- [x] `devtools::check()` clean; `roxygen2::roxygenize()` then `pkgdown::build_site()` clean; `renv::snapshot()` if dependencies moved. (2026-09-27: `R CMD check --as-cran` on the 0.5.0 tarball, 0 errors, 0 warnings, 3 documented notes; `devtools::check()` is blocked by the local toolchain. pkgdown builds; the lockfile was updated in #51.) +- [x] Version bump to 0.5.0; downstream heads-up to cchsflow/chmsflow maintainers if any metadata-extension columns were added. (Maintainer decision 2026-09-22: the extension columns `mockFormula`, `anchor` and `censored_by` are documented in MockData, in `reference-config.qmd`'s placement rule, rather than in a separate notice. They are optional and change nothing for those packages.) ---