Problem
We report one retention number: 97.9% of the teacher's Gini. That is a portfolio average, and distillation loss is almost never uniform.
If retention is 99% on thick-file applicants and 88% on thin-file, we have concentrated the compression cost on a specific population and reported a number that conceals it. In lending, that is a fair-lending exposure, not just a modelling detail.
Proposed approach
- Add a
segments= parameter to sweep_whitebox (an array of group labels) that reports retention per segment alongside the global number
- Add retention-by-band reporting: how well does the artifact track the teacher within each risk band, especially near the cutoff where decisions are most sensitive
- Surface the worst segment prominently, in the same spirit as
band_efficiency's worst_band
Acceptance criteria
Files
src/compileml/tune/sweeps.py, docs/howto/tuning.md
Scope
Small to medium. Good entry point for someone who wants to work on the credit-risk side without touching the artifact contract.
Problem
We report one retention number: 97.9% of the teacher's Gini. That is a portfolio average, and distillation loss is almost never uniform.
If retention is 99% on thick-file applicants and 88% on thin-file, we have concentrated the compression cost on a specific population and reported a number that conceals it. In lending, that is a fair-lending exposure, not just a modelling detail.
Proposed approach
segments=parameter tosweep_whitebox(an array of group labels) that reports retention per segment alongside the global numberband_efficiency'sworst_bandAcceptance criteria
sweep_whitebox(..., segments=labels)returns per-segment Gini retention and Spearmandocs/howto/tuning.mdwith an explicit note on why the average can misleadFiles
src/compileml/tune/sweeps.py,docs/howto/tuning.mdScope
Small to medium. Good entry point for someone who wants to work on the credit-risk side without touching the artifact contract.