Skip to content

Adds independent forced-choice scoring (0.13.0) - #18

Merged
aaronstevenwhite merged 3 commits into
mainfrom
feat/forced-choice-scalar-scoring
Nov 6, 2026
Merged

aaronstevenwhite merged 3 commits into
mainfrom
feat/forced-choice-scalar-scoring

Conversation

@aaronstevenwhite

Copy link
Copy Markdown
Collaborator

Description

Adds ForcedChoiceModelConfig.scoring_mode="independent" for dual encoders. Each alternative receives a score from the same scalar head, and softmax converts those scores into choice probabilities. The existing joint mode remains the default; single encoding with independent scoring is rejected.

Supports fixed, random-intercept, and participant-specific scoring heads, preserves unselected alternatives, validates option names, and persists the scoring mode in checkpoints. Fixes dual-encoder training, participant parameter optimization, and early-stopping restoration. Updates the gallery configuration interface and API documentation, and bumps the minor version to 0.13.0 with a changelog entry.

Motivation

Forced-choice models previously predicted logits from combined option representations. Independent scoring provides a shared acceptability function over individual alternatives, including the Bradley-Terry choice rule for two alternatives.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactoring only
  • Tests

Checklist

  • Read the contributing guidelines and followed repository conventions.
  • Ran Ruff lint and formatting checks.
  • Ran Pyright with zero errors and warnings.
  • Added tests for probabilities, invariances, gradients, training, participant effects, early stopping, checkpoint compatibility, and invalid configurations.
  • Updated documentation and changelog.

Testing

  • Full Python suite before rebasing onto current main: 3,553 passed, 16 skipped, 257 deselected with -m 'not slow_model_training' --no-cov; two existing dependency warnings.
  • After rebasing: 48 focused feature and version tests passed; Ruff lint, formatting, Pyright, lockfile validation, and git diff --check passed.
  • GitHub CI will validate the full suite against current main, including documentation code blocks and TypeScript checks.

@aaronstevenwhite
aaronstevenwhite merged commit 8af5182 into main Nov 6, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant