position-bias

Seven decision models, measured as the candidate list grows, the option order changes, and the wrong answers stop being obvious.

[NeurIPS'2026] Position Bias in Mamba and Hybrid Language Models | Evidence Position Bias Across Sequence Mixers in Long-Context Question Answering

Do LLM judges know when they're wrong? Three open-weight judges (Qwen2.5-7B, kev-8b, auto-j-13b) graded against MT-Bench human votes: calibration, position and padding attacks, and what auto-accepting confident verdicts would cost. Interactive site included.

ACL 2026 | CapCal: content-agnostic probability calibration for de-biasing listwise rerankers.

position bias in LLM judges is worse than people think: ask a small instruct model to choose 1 of 2 items. 81% of the answer will be based on which slot the item was in, not which it was. swap the order and the answer flips 78% of the time. an inconclusive cognitive dissonance experiment

Position-bias-aware ranking: estimating and correcting position bias to optimize Earnings Per Visitor (EPV) · simulation study · IPW & propensity modeling · Python

Blind side-by-side human eval of two model responses on a weighted rubric — randomized panes to kill position bias, bootstrap CIs on the margin, inter-rater reliability. An eval you can't audit is a vote, not a measurement.

Independent research on AI evaluation validity, LLM-as-a-judge reliability, benchmark comparability and experimental assurance.

Learning-to-rank lab: listwise models, position bias, calibration, and NDCG evaluation

Position bias, length bias and self-uncertainty of a small LLM judge on MT-Bench (Qwen2.5-7B)

SSIT: a label-free, gold-free test for whether LLM-judge position x verbosity bias corrections actually compose. No human labels, no model of the judge. Code, 7-judge/6-family pilot data, and pre-registered protocol for the NeurIPS 2026 JUDGe workshop paper.

Measure position/verbosity/assertiveness bias in an LLM-as-judge (Claude) by judging pairs in both orders. Finding: no position bias, but 75% verbosity bias and 100% assertiveness bias — judge scores gameable by length + tone.

Two kinds of saturation: why LLM-judge order bias is hard to measure — essay, Lean proofs (0 sorry), and a reproducible dispersion measurement.

Open-source implementation of the typed-decision pattern popularised by TypeSafe's Jev: read a decision out of a small language model's logits, in the browser. Library, benchmarks and paper. Not affiliated with TypeSafe.

Official reproduction code for 'Efficient LLM Preference Classification Through Position Bias Mitigation and Architectural Symmetry' (Applied Intelligence APIN-D-26-05359)

Counterfactual learning-to-rank for marketplace search logs in PySpark: position-bias estimation, IPS-weighted training, NDCG evaluation against known ground truth

Ranking evaluation with error bars: NDCG, MRR and MAP with confidence intervals, plus position-bias correction for click logs. No dependencies.

A frozen probe set and a harness for auditing LLM judges for bias. No audit published yet.