SOCP: Self-Organized Conformal Prediction

This repository is the official code for Self-Organized Conformal Prediction: Reducing Regional Coverage Gaps with Unsupervised Group Discovery (Berthier et al., NeurIPS 2026).

Paper | Reproducing the experiments | Citation

---

Overview

Split conformal prediction (SCP) guarantees $1-\alpha$ coverage on average. But one pooled threshold can undercover some regions of the feature space and overcover others.

SOCP keeps the predictor and the nonconformity score and changes only the calibration. A self-organizing map (SOM), trained on training inputs before calibration, splits the feature space into cells. Each test point is then calibrated with the calibration scores retrieved around its best-matching unit (BMU). The result is a trade-off: lower regional coverage gaps than pooled calibration of the same score, for a small increase in output size.

Key results

Across ten regression and classification benchmarks, with ten paired seeds:

Comparison	Result
SO-SCP vs pooled SCP	Lower mean weighted coverage gap (WCovGap) on all ten datasets: -14.8% on average, for a +3.0% mean change in output size
Five scores per task	SO calibration lowers the ten-seed mean WCovGap in 43 of 50 score and dataset comparisons
Other calibrators: LCP, RLCP, Mondrian k-means, Classwise, Clustered	On Bike Sharing, Bio, and California Housing, only SO-SCP and pooled SCP keep every interval finite. Mondrian k-means lowers WCovGap more than SO-SCP on the five classification datasets, but enlarges sets more on four of them
External k-means partitions with 10, 25, and 50 groups	SO-SCP lowers WCovGap for every dataset at all three granularities
Diagnostics that need no feature-space partition	Held-out WSC target error improves on all nine datasets with a value (-29.6% on average), $L_1$ ERT on 9 of 10 (-11.1%), and the SSC gap on 6 of 10

---

How it works

1. Fit the predictor and train a SOM on the training inputs. The map never sees calibration data.
2. Send each calibration point to its BMU cell.
3. For a test point, retrieve the calibration scores in a buffer around its BMU and use their conformal quantile as the threshold.

The buffer defines three regimes. Writing $k^\ast$ for the test point's BMU, the paper proves these coverage guarantees:

Regime	Calibration buffer	Conditioning event	Coverage lower bound	Status
1: the BMU cell	$\mathcal{I}_k$	$k^\ast = k$	$1-\alpha$	exact
2: a fixed grid neighborhood	$\mathcal{I}_{N(k,r)}$	$k^\ast \in N(k,r)$	$1-\alpha$	exact
		$k^\ast = k$	$1-\alpha-\varepsilon_k$	approximate
		none	$1-\alpha-\bar\varepsilon$	approximate
3: the neighborhood plus cells in prototype-distance order	$\mathcal{I}_{R_{L^\star}(k)}$	$k^\ast \in R_{L^\star}(k)$	$1-\alpha$	exact
		$k^\ast = k$	$1-\alpha-\varepsilon_k(R_{L^\star}(k))$	approximate

Here $\mathcal{I}_A$ holds the calibration points whose BMU lies in the cell set $A$, $N(k,r)$ the cells within grid distance $r$ of cell $k$, and $R_{L^\star}(k)$ that neighborhood plus the next $L^\star$ cells by prototype distance.
$\varepsilon_k(A)$ is the Kolmogorov-Smirnov bias of cell $k$ over $A$, with $\varepsilon_k = \varepsilon_k(N(k,r))$, and $\bar\varepsilon$ averages $\varepsilon_k$ over cells, weighted by their mass.
Rows conditioned on a cell set hold for the threshold of a fixed reference cell $k$, and the other rows hold for the deployed threshold.

Regime 3 is for sparse calibration sets. Its global cell budget $L^\star$ is chosen from training cell occupancies and the planned calibration size, before any calibration score is seen.

---

Installation

You need Python 3.11 or newer. We use uv:

git clone https://github.com/LouisTier/SOCP.git
cd SOCP
uv sync

pip install -e . works too. Both install TorchSOM from commit 020ec68 (tag socp-neurips2026), which adds the fixed-cell retrieval used by Regime 3.

---

Repository layout

socp/conformal/   scores, localizers, calibration mechanisms, and their composition
socp/models/      base predictors needed by the scores
socp/data/        dataset loaders
socp/evaluation/  coverage metrics and paired statistics
experiments/      config.yaml, every pipeline step, and two helper scripts (see its README)

---

Reproducing the experiments

1. Check the installation

make smoke   # Synthetic data through prepare, run, and aggregate, in under a minute.

2. Get the data

make download DATASET=all

This downloads the nine public datasets. ImageNet-1k can't be downloaded automatically. With access to the ILSVRC2012 validation images and the torchvision ResNet-50 IMAGENET1K_V2 weights file, build the cache it needs:

uv run python experiments/scripts/imagenet_cache.py --imagenet-root <root containing val/> \
    --weights resnet50-11ad3fa6.pth --out data/raw/imagenet_val_cache.npz

The cache holds the 50,000 labels, the classifier probabilities, and the 2,048-dimensional penultimate representations. The protocol splits it into 8,000 representation-training, 2,000 selection, 20,000 calibration, and 20,000 test rows.

3. Run the experiments

make prepare     # Splits, fitted predictors, and representations    -> data/prepared
make run         # Test predictions and metrics for every method      -> results/runs
make aggregate   # Seed summaries and paired tests                   -> results/aggregate

The full profile covers ten datasets, ten seeds, and all 25 regression and 35 classification score and calibrator pairs declared in experiments/config.yaml.
On the paper's Intel Xeon Gold 6134 CPUs, the run step took about 31 minutes per regression dataset and seed, and about 6.5 hours per classification dataset and seed.
experiments/scripts/run_parallel.sh runs the dataset and seed pairs in parallel, one CPU each. To run a single pair:

make prepare DATASETS=concrete SEEDS=42
make run DATASETS=concrete SEEDS=42

Each method writes predictions.csv (coverage, size, and cutoff for each test row), metrics.csv, and metadata.json under results/runs/<dataset>/seed_<seed>/<score>__<calibrator>/.
make aggregate then combines them into results/aggregate/:

File	Contents
seed_metrics.csv	The metrics of every method, dataset, and seed
summary.csv	Their means over seeds
paired_tests.csv	Wilcoxon tests of SO-SCP against SCP on WCovGap, with Holm's correction across datasets
regime3.csv	SCP, Regime 2, and Regime 3 on Concrete and Auto MPG, by calibration size
sensitivity.csv	SO-SCP on California Housing across SOM grid sizes and neighborhood radii

---

Methods

Kind	Method	Reference
Regression score	Absolute residual	Lei et al., 2018
Regression score	CQR	Romano et al., 2019
Regression score	CHR	Sesia and Romano, 2021
Regression score	NFCP	Colombo, 2024
Regression score	RCP	Plassier et al., 2025
Classification score	LAC	Sadinle et al., 2019
Classification score	APS	Romano et al., 2020
Classification score	RAPS	Angelopoulos et al., 2021
Classification score	SAPS, rank-only	Huang et al., 2024
Calibration	Pooled SCP	Vovk et al., 2005
Calibration	LCP	Guan, 2023
Calibration	RLCP	Hore and Barber, 2025
Calibration	Mondrian on k-means groups	Vovk et al., 2003
Calibration	Classwise, Clustered	Ding et al., 2023
Calibration	SOCP	This paper

---

Citation

@inproceedings{berthier2026socp,
  title     = {Self-Organized Conformal Prediction: Reducing Regional Coverage Gaps with Unsupervised Group Discovery},
  author    = {Berthier, Louis and Shokry, Ahmed and Moreaud, Maxime and Ramelet, Guillaume and Dieuleveut, Aymeric},
  booktitle = {Advances in Neural Information Processing Systems},
  year      = {2026},
  url       = {https://arxiv.org/abs/2606.29403}
}

See also CITATION.cff.

---

Acknowledgments

• Centre de Mathématiques Appliquées (CMAP), École Polytechnique
• Manufacture Française des Pneumatiques Michelin
• Aymeric Dieuleveut is supported by French State aid managed by the Agence Nationale de la Recherche under the France 2030 program, reference ANR-23-IACL-0005 (Hi! PARIS).

---

License

SOCP is released under the Apache License 2.0.