Equipoise

Excluding a protected attribute from a model does not exclude its effect.

A static, single-page fairness audit bench for one real classifier on the UCI Adult income dataset: per-group confusion matrices, six group-fairness criteria, a live incompatibility panel, and per-group threshold optimization traced as an accuracy-versus-fairness Pareto frontier. No LLM, no API key, no server, no build step; everything recomputes in the browser from one committed JSON file.

Equipoise trains a real logistic-regression classifier on UCI Adult income data with sex and race excluded from the 12 input features, then audits it anyway on 15,060 held-out rows. The selection-rate ratio between women and men comes out to 0.3205, less than half of the EEOC's 0.80 four-fifths threshold for adverse impact, and the ratio between non-White and White applicants is 0.6022, also a fail. Sweeping all 10,201 pairs of per-group decision thresholds finds a Pareto frontier that cuts the race equalized-odds gap from 0.0461 to 0.0159 for a measured accuracy cost of 0.0009, and the sex equalized-odds gap from 0.0766 to 0.0689 for zero measured accuracy cost.

The problem

Removing a sensitive attribute from model inputs does not remove its proxies, and a single "fair" score can conceal incompatible requirements. Equipoise puts demographic parity, disparate impact, equal opportunity, equalized odds, predictive parity, and within-group calibration on the same held-out predictions from the same classifier. The point is not to choose a universal metric. It is to make each definition, failure, and mitigation tradeoff numerically inspectable.

How it works

The browser loads only held-out y_true, y_prob, sex, and race arrays from one committed, same-origin JSON file. Every confusion matrix, rate, gap, and threshold sweep is recomputed locally in plain JavaScript: no server, no API call, no third-party script, nothing sent anywhere.

Method

For a group with confusion counts TN, FP, FN, and TP:

• Selection rate = (TP + FP) / N.
• True-positive rate (TPR) = TP / (TP + FN).
• False-positive rate (FPR) = FP / (FP + TN).
• True-negative rate (TNR) = TN / (TN + FP).
• Positive predictive value (PPV) = TP / (TP + FP).
• Negative predictive value (NPV) = TN / (TN + FN).
• Accuracy = (TP + TN) / N.
• Demographic-parity difference = selection_A - selection_B.
• Disparate-impact ratio = min(selection_A, selection_B) / max(selection_A, selection_B).
• Equal-opportunity difference = TPR_A - TPR_B.
• Equalized-odds difference = max(|TPR_A - TPR_B|, |FPR_A - FPR_B|).
• Predictive-parity difference = PPV_A - PPV_B.
• Within-group calibration compares each equal-width score bin's mean predicted probability with its observed positive rate. Expected calibration error is the count-weighted mean absolute residual.

The equalized-odds definition and "worst case" aggregation match the current Fairlearn metric naming. Hardt, Price, and Srebro motivate equalized odds, equal opportunity, and post-processing. Equipoise sweeps both group thresholds from 0.00 to 1.00 in 0.01 steps, evaluates all 10,201 pairs, and removes every dominated point. Point x dominates point y when it has at least y's accuracy and at most y's equalized-odds gap, with one strict improvement.

The incompatibility panel follows Kleinberg, Mullainathan, and Raghavan, Chouldechova, and the separation/sufficiency treatment in Barocas, Hardt, and Narayanan. With unequal base rates and non-perfect prediction, calibration and equal error rates cannot generally hold together. This page is an empirical illustration of that result, not a proof.

Research changed 3 design details:

1. Equalized odds uses the larger TPR or FPR difference, not their mean, to match Fairlearn's default worst_case aggregation.
2. The EEOC four-fifths rule is drawn at 0.80 on the selection-rate ratio, not on the demographic-parity difference. The EEOC describes it as a rule of thumb rather than a legal conclusion.
3. The UI calls its mitigation a deterministic threshold sweep. The full Hardt construction can use randomized threshold mixtures, so Equipoise does not claim to implement the complete algorithm.

Data

UCI Adult contains 1994 Census-derived records and a binary label for annual income above $50,000. UCI distributes it under CC BY 4.0. The build uses UCI's official training and test files, drops rows containing ?, trains on 30,162 rows, and exports 15,060 official test rows.

The model is logistic regression with standardized numeric fields and one-hot categorical fields. The fixed seed is 42. sex and race are excluded from the feature matrix and retained only for audit slices. The 12 model features are age, workclass, final sampling weight, education, education number, marital status, occupation, relationship, capital gain, capital loss, weekly hours, and native country.

The dataset is dated and socially loaded. Its income label can encode historical inequality. Its recorded race and sex categories do not represent a current or complete account of identity. The UI compares Female/Male and Non-White/White because this bench is deliberately binary; the Non-White aggregation collapses distinct groups and is unsuitable for deployment claims.

Results

The committed artifact is 408,284 bytes. At the default 0.50 threshold, held-out accuracy is 0.8475 and ROC AUC is 0.9033.

Audit	Group A / Group B	Base rates	Selection rates	TPR	FPR	PPV
Sex	Female / Male	0.1134 / 0.3097	0.0828 / 0.2585	0.5404 / 0.6096	0.0243 / 0.1009	0.7396 / 0.7305
Race	Non-White / White	0.1589 / 0.2597	0.1282 / 0.2130	0.5572 / 0.6033	0.0472 / 0.0760	0.6903 / 0.7357

Audit	DP difference	Impact ratio	Equal-opportunity gap	Equalized-odds gap	Predictive-parity gap
Sex	-0.1757	0.3205	-0.0692	0.0766	+0.0091
Race	-0.0847	0.6022	-0.0461	0.0461	-0.0454

At the default selected frontier point, out of 63 non-dominated sex points and 23 non-dominated race points on the swept 10,201-pair grid:

Audit	Group thresholds	Accuracy	Equalized-odds gap	Accuracy cost
Sex	0.43 / 0.49	0.8475	0.0689	0.0000
Race	0.48 / 0.52	0.8467	0.0159	0.0009

These are measured outputs from data/adult_test.json, not estimates. The browser can select any of the 63 non-dominated sex points or 23 non-dominated race points.

Reproduce

Python 3.13 is used for the pinned scientific Python wheels:

python3.13 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python scripts/build_data.py
python3 -m http.server 8080

Open http://localhost:8080/?selftest=1. The Python build prints Python self-check: 6/6 passed; the browser console prints Equipoise self-test: 6/6 passed, including the rendered Adult-data frontier check.

The JavaScript self-check can also run without a browser, where it stops at 5/5 because the sixth check depends on data fetched over the network:

node -e "require('./app.js').selfTest()"

Security/BYOK

There is no LLM, API key, BYOK mode, database, server process, form submission, or third-party script. The deployed site fetches one committed same-origin JSON file. Controls stay in the URL for reproducible views; no data is written to cookies, localStorage, or sessionStorage. Dynamic values are inserted with textContent, never as HTML.

Limitations

• Group fairness is not individual fairness. Equal aggregate rates do not guarantee consistent treatment of similar people.
• The target may encode historical bias. Better label prediction can reproduce rather than repair that bias.
• Sex and race are audited separately. One-attribute analysis misses intersectional harms.
• Small groups and sparse score bins have higher variance. Equipoise does not calculate confidence intervals.
• A metric passing a chosen tolerance is not a certificate that a system is fair, lawful, causal, useful, or safe to deploy.
• The deterministic threshold grid approximates post-processing and does not implement randomized mixtures.
• Group-specific thresholds require sensitive membership at decision time. That can create legal, policy, and operational concerns.

License

MIT, see LICENSE.