Stability monitoring: PSI, CSI, and score-distribution drift

[orgoca]

Problem

The validation framework checks an artifact at a point in time. Model risk management is about ongoing monitoring, and CompileML currently offers nothing for it. There are no mentions of population stability index, characteristic stability, or drift anywhere in the codebase.

This is the gap that matters most in practice: the questions a risk committee asks monthly are "has the applicant population moved?", "is the score distribution still where it was at development?", and "is any band's bad rate diverging from expectation?"

Why CompileML can answer this unusually well

Because the artifact is hashed and the model provably cannot change between runs, any movement in a monitored metric is attributable to the population rather than the tooling. That is a cleaner attribution than most monitoring stacks can offer, and we should say so.

Proposed approach

A new compileml.monitor module (compile-side, numpy allowed):

• population_stability(reference_latents, current_latents, edges) → PSI per band and total, with the conventional 0.10 / 0.25 interpretation bands
• characteristic_stability(reference_X, current_X, feature_names) → CSI per feature
• band_drift(artifact, X, y, reference_band_rates) → observed vs expected bad rate per band, with confidence intervals so small bands do not raise false alarms
• Optional: a monitor CLI subcommand producing a JSON report suitable for a scheduled job

Acceptance criteria

• PSI computed on the deployed integer band assignment, not a re-derived float binning
• Small-sample honesty: intervals or minimum-volume guards, consistent with how band_efficiency already handles this
• Reference distribution can be captured at compile time and stored in artifact metadata (design question — see below)
• Documented in a docs/howto/monitor.md

Design question for the implementer

Should the development-time reference distribution live inside the artifact (travels with it, but changes the hash on any refresh) or beside it as a separate reference file? Worth deciding explicitly before coding.

Files

New src/compileml/monitor/, plus src/compileml/cli.py and docs.

Scope

Medium. Self-contained, no spec change required if the reference lives outside the artifact.