Skip to content

Report distillation retention by segment, not just globally #12

Description

@orgoca

Problem

We report one retention number: 97.9% of the teacher's Gini. That is a portfolio average, and distillation loss is almost never uniform.

If retention is 99% on thick-file applicants and 88% on thin-file, we have concentrated the compression cost on a specific population and reported a number that conceals it. In lending, that is a fair-lending exposure, not just a modelling detail.

Proposed approach

  • Add a segments= parameter to sweep_whitebox (an array of group labels) that reports retention per segment alongside the global number
  • Add retention-by-band reporting: how well does the artifact track the teacher within each risk band, especially near the cutoff where decisions are most sensitive
  • Surface the worst segment prominently, in the same spirit as band_efficiency's worst_band

Acceptance criteria

  • sweep_whitebox(..., segments=labels) returns per-segment Gini retention and Spearman
  • A helper reports retention by band for a fitted artifact
  • Documented in docs/howto/tuning.md with an explicit note on why the average can mislead

Files

src/compileml/tune/sweeps.py, docs/howto/tuning.md

Scope

Small to medium. Good entry point for someone who wants to work on the credit-risk side without touching the artifact contract.

Metadata

Metadata

Assignees

No one assigned

    Labels

    credit-riskDomain requirement from regulated lending practiceenhancementNew feature or requesthelp wantedExtra attention is needed

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions