Default decile impacts to household net income and person-weighted groups

[anth-volk]

Fixes #485

Summary

• default DecileImpact.income_variable and calculate_decile_impacts to household_net_income
• replace the unweighted pandas.qcut fallback with shared microdf weighted ranking for both DecileImpact and IntraDecileImpact
• form computed household groups with household_weight * household_count_people, while other entities use their entity survey weights
• clamp computed groups to 1..quantiles; exclude negative, missing, and non-finite computed ranking values under the documented rules
• pin microdf_python==1.3.0 exactly so tied values retain the reviewed max-rank behavior
• preserve explicit income concepts and precomputed grouping variables, including out-of-range sentinel values that intentionally exclude rows
• make UK economic-impact analysis pass its income concept explicitly so future shared-default changes do not alter country analysis silently
• calculate DecileImpact statistics with entity survey weights and IntraDecileImpact proportions with effective population weights
• calculate relative changes from weighted group means and the overall intra-decile row directly from the included population
• add an approved behavior specification and tests covering weighting, ties, exclusions, invalid inputs, empty groups, alignment, and overall results

Breaking change

This intentionally changes default public decile results and is classified with a .breaking.md Towncrier fragment. The default income concept changes for callers that previously relied on equivalised HBAI income, and computed groups change from equal raw survey rows to person-weighted population groups.

The UK .py economic-impact bundle changes both its income concept and its grouping. The US bundle already selected household_net_income explicitly, so its income concept does not change, but its computed grouping also becomes person-weighted.

Root cause

The original UK report used equivalised HBAI income to rank households while calculating several separate outcomes within those deciles. When the generic DecileImpact output was extracted, the ranking variable and measured outcome were collapsed into one UK-specific default.

Separately, the generic fallback passed a weighted MicroSeries to pandas.qcut. Pandas ignored the survey weights, so the groups represented equal raw rows rather than approximately equal populations. The prior output code also used unweighted means/counts, averaged observation-level relative changes, and constructed the overall intra-decile row by averaging decile proportions.

Implementation

A shared _PreparedDecileAnalysis path now resolves the target entity, maps income values when needed, validates baseline/reform alignment and weights, creates baseline groups once, and supplies the two output types with their distinct analysis weights. Empty and excluded populations are represented explicitly rather than being reported as no-change populations.

The behavior is documented in docs/decile-impact-spec.md, including indivisible survey rows, tied-income max-rank grouping, negative and non-finite ranking values, zero weights, precomputed exclusion sentinels, and null results for empty populations.

User impact

Default decile results now measure household net income, including effects represented in household_tax, and computed household groups are person-weighted. Consumers can still request equivalised HBAI income or pass a precomputed grouping variable explicitly. Household-level decile means and outcome counts remain household-survey-weighted, while intra-decile proportions are person-weighted.

Validation

• make format — passed
• make lint — passed
• focused and adjacent regression suite — 61 passed
• the complete suite was also sampled through 28%; all decile tests passed, while four unrelated extra-variable tests failed because the available validation environment has policyengine-us==1.729.0 instead of the repository manifest's 1.764.6
• mypy reports no diagnostics in the four changed output modules; the repository-wide run still has pre-existing diagnostics in imported modules

CI note

Before the test-first checkpoint, Python 3.13 and 3.14 full test jobs passed, as did docs, changelog, mypy, bundle metadata, and every smoke-import version. The prior GitHub Lint failure installed an unbounded newer Ruff that flagged existing Markdown files across the repository; the repository-locked Ruff checks pass locally.

[vahid-ahmadi]

Reviewing from the downstream side — UK Chat PR PolicyEngine/policyengine-uk-chat#217 is pinned to this commit, so we care about both the semantics and the release timing.

The diagnosis in the description is convincing, and both halves of the fix look right to me: separating the ranking variable from the measured outcome, and replacing pandas.qcut (which silently discarded the MicroSeries weights) with weighted ranking. Extracting calculate_decile_groups so DecileImpact and IntraDecileImpact cannot drift apart is the right shape, and making UK economic_impact_analysis pass its income concept explicitly is a good guard.

Substantive

• calculate_decile_groups clamps the top but not the bottom:

  groups = np.minimum(np.ceil(percentile_ranks * quantiles), quantiles).astype(int)

  rank(pct=True) appears to be cumulative-inclusive (the [10, 20, 30, 40] / person-weights [4, 1, 1, 1] fixture landing on [6, 8, 9, 10] implies 4/7 → 0.571). So the smallest rank is w_first / total, which is 0 whenever the lowest-income rows carry zero weight — and zero-weight households do occur in some datasets. Those rows then get group 0, which falls outside the range(1, quantiles + 1) loop in calculate_decile_impacts and disappears from every decile without an error. A matching np.maximum(..., 1) would close that off. A fixture with a leading zero-weight row would pin the behaviour either way.

• Consider framing this as a breaking change rather than .fixed. Default decile results move for every consumer of calculate_decile_impacts and compute_intra_decile_impacts, not only UK Chat — both the income concept and the grouping change, and published decile charts will shift. A heads-up to the US side and policyengine-api before release seems worth it.

• Cost: each DecileImpact.run() recomputes the full grouping, and calculate_decile_impacts constructs ten of them, so the weighted rank runs ten times over the whole frame. That was already true of qcut, so it is not a regression, but MicroSeries.rank is meaningfully more expensive and this now runs on FRS-sized data on every UK Chat decile request. Hoisting the grouping into calculate_decile_impacts and passing it down would be a straightforward follow-up if it shows up in profiles.

Minor

• microdf_python>=1.3.0 is a floor, not a pin, and the tie-handling behaviour the fixtures rely on is version-sensitive. Worth confirming the lockfile bump is enough for consumers installing outside the lock.
• The decile_variable branch returns baseline_data[decile_variable] unvalidated, so a precomputed variable outside 1..quantiles fails the same silent-drop way. Pre-existing, and out of scope here, but it is the same failure mode as the clamp point above.

Release timing

Whenever this lands, could you drop a note on the release version? UK Chat #217 is currently carrying an immutable GitHub-archive pin to a289cc4 in backend/requirements.txt, which we would rather not merge to production — we want to swap it for the first release containing this commit.