Fair lending: disparate impact testing and attribution-level disparity analysis

[orgoca]

Problem

For US consumer lending, disparate impact testing is not optional (ECOA / Regulation B). CompileML currently has no fairness capability at all — it is a roadmap bullet only. A bank cannot take a model to committee without it.

Why this is a natural fit here, not a bolt-on

CompileML already computes exact per-decision attributions. That enables something most fairness tooling does poorly: disparity analysis at the driver level, not just the outcome level.

• Outcome level: "the approval rate ratio between groups is 0.78"
• Attribution level: "62% of the mean score gap between groups is contributed by INQUIRIES_6M"

The second is what turns a fair-lending finding into a remediation plan, and it is exactly the kind of statement a proxy analysis needs. Because our attributions are exact and reconcile to the decision, the decomposition is defensible in a way a SHAP-based one is not.

Proposed approach

A compileml.fairness module:

• approval_rate_disparity(decisions, protected, threshold_band) → adverse impact ratio with confidence intervals
• score_distribution_disparity(decisions, protected) → group score/PD distributions, KS
• attribution_disparity(decisions, protected) → the differentiator: decomposition of the mean score gap by feature, using the exact contributions
• calibration_by_group(decisions, y, protected) → is the PD equally well calibrated per group

Acceptance criteria

• All metrics computed from decide() payloads — never recomputing the model (same design rule as compileml.viz)
• Attribution decomposition sums to the total score gap, exactly, in integer units
• Confidence intervals on every ratio; small groups must not produce confident nonsense
• docs/howto/fairness.md stating plainly what the tool does and does not certify

Important scoping note

This must not claim to certify compliance. Consistent with the rest of the project: it produces evidence a validator can inspect, not a verdict. That boundary belongs in the module docstring and the docs page.

Files

New src/compileml/fairness/, docs, tests.

Scope

Substantial, but splittable — the outcome-level metrics and the attribution decomposition can land as separate PRs.