Champion/challenger: diff two artifacts at the band and decision level

[orgoca]

Problem

Replacing a model in production requires answering: who moves, by how much, and why? Today you would score a population through both artifacts and diff by hand.

Because both artifacts are deterministic and hashed, this comparison can be exact and fully reproducible — a real advantage over comparing two float scorers.

Proposed approach

compileml.compare.diff_artifacts(artifact_a, artifact_b, X) returning:

• band migration matrix (how many accounts move from each band to each other band)
• share of decisions unchanged, upgraded, downgraded
• PD delta distribution
• for the movers: which features' contributions changed most (both artifacts produce exact attributions, so this decomposes cleanly)
• both artifact hashes recorded in the report

A compileml diff a.json b.json --csv population.csv CLI subcommand.

Acceptance criteria

• Migration matrix computed on the deployed integer band path
• Report includes both hashes so the comparison is itself reproducible
• Works when the two artifacts have different band counts or different feature sets (report the mismatch honestly rather than failing silently)

Files

New src/compileml/compare/, src/compileml/cli.py

Scope

Medium. Self-contained, no spec change, and immediately useful to anyone running a model replacement.