Skip to content

scientific(selection): select candidate K from admitted rolling-origin predictive scores #687

Description

@seonghobae

Finding

#684 provides an owner-issued fixed-training prevalence-mean predictive log likelihood, #685 admits cutoff/leakage-safe rolling-origin partitions, and #686 binds one fitted training state plus one evaluation payload to that admitted partition. The remaining model-selection gate still chooses candidate K from the in-sample Schwarz score. There is no owner path that compares multiple candidate training fits on the same admitted rolling-origin evaluation evidence.

That leaves #680's core selection claim incomplete: TEPP can now measure one admissible predictive diagnostic, but it cannot yet make the candidate-K decision from that diagnostic without caller-side recomposition.

Required RED / repair

Add a narrow model_selection gate that:

  • accepts one admitted RollingOriginPartition, a nonempty set of owner-issued ReferenceTopicTrainingFit candidates, and one exact evaluation payload;
  • derives each candidate K from the fitted model's topic dimension; no detached caller-supplied K label;
  • rejects duplicate fitted topic dimensions rather than counting two fits of the same K as distinct candidates;
  • scores every candidate only through scientific(selection): bind predictive rows to admitted rolling-origin partition #686 rolling_origin_prevalence_mean_predictive_log_likelihood(...), preserving partition/training/evaluation identity checks and the frozen training prevalence basis;
  • selects the highest finite rolling-origin prevalence-mean predictive log likelihood, with deterministic smaller-K tie breaking;
  • remains invariant to candidate input order;
  • keeps the existing in-sample Schwarz selector unchanged as a separately named diagnostic path;
  • does not call this STM document-completion likelihood and does not let an LLM vote choose the numerical optimum.

Use actual converged CPU f64 fits for at least two candidate topic dimensions in the positive contract. Empty candidates, duplicate K, partition substitution, and non-finite diagnostics must fail closed.

Boundary

This closes only the one-window predictive candidate-selection gate. It does not complete #680: multiple rolling-origin windows, true-parameter/true-K temporal-relational recovery across replications, RMSE/bias/failure denominator/MCSE acceptance, deliberate future-leak rejection/audit evidence, Evidence source/event-time authentication (#527/#658/#671), Membership provenance (#605), relation activation (#675), backend parity, or immutable release remain separate requirements.

Refs #639 #680 #684 #685 #686.

Activity

  1. seonghobae commented on Sep 22, 2026

    @seonghobae
    ContributorAuthor

    Ordinary-forward source lineage is now on Draft #639:

    • source/API RED e843ba043222e1af96443b35ed421b6b466b55e5: actual CPU f64 K=2/K=3 training fits over one admitted rolling-origin partition require a public predictive candidate-K selector; empty and duplicate-K failure contracts are included. The RED head is source-level only; no hosted failing run is claimed.
    • typed duplicate-dimension failure fe58c615f4bb563a1dc90ca498c9ecb47bab8f9e adds ModelSelectionError::DuplicateCandidateK.
    • causal selector 7df02402ad286640df080b642065ca97e145d235 derives K from each owner-issued fitted topic dimension, scores only through scientific(selection): bind predictive rows to admitted rolling-origin partition #686's partition-bound predictive consumer, rejects duplicate dimensions, selects the highest finite score, and uses smaller K for exact ties.
    • public export 07355a3a399aae350e35ca1fd0a61b355c0e44c4.
    • CHANGELOG/current source head dde2d464ddc0bded649f1b7abb35132125f4411d records the claim boundary.

    The existing in-sample Schwarz selector is unchanged. This still does not complete #680: one admitted window can now choose K from predictive evidence, but multi-window realistic temporal/relational true-K recovery, deliberate future-leak rejection/audit evidence, recovery RMSE/bias/failure/MCSE acceptance, owner provenance dependencies, backend parity, and immutable release remain outstanding. Keep #687 open until exact-head hosted gates and normal landing settle.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions