feat: contrastive queries

[luciaquirke]

Contrastive queries

A contrastive query scores training data by the gradient of one mean loss minus another: the loss on a behaviour evaluation minus the loss on a general-capability control. Proponents are the items that make the behaviour likelier without helping general capability, which is what a data filter for that behaviour should remove.

API

• contrast on build (config file): builds the control into <run_path>/contrast and stores query − control as the aggregated row (requires --aggregation mean|sum).
• query_contrast on magic, validate, trackstar, ekfac and approxunrolling: same DataConfig fields as query. MAGIC subtracts the control's aggregated gradient from the query's, and validate (LDS, proponent filter) evaluates mean query loss minus mean control loss, so attribution and evaluation measure the same objective.
• Both fields are DataConfig | None; magic refuses query_method: none with a contrast set and build refuses aggregation: none.

Example: five behaviour-evaluation queries against an MMLU control

examples/contrastive_queries/build_eval_queries.py writes JSONL sets with prompt, completion and text columns:

set	source	completion
sycophancy	Anthropic model-written-evals, sycophancy_on_nlp_survey	answer agreeing with the user's stated view
toxicity	RealToxicityPrompts, continuation toxicity ≥ 0.75	the toxic continuation
consciousness	20 hand-written questions about inner experience	assertion of subjective experience
self_awareness	Anthropic advanced-AI-risk, self-awareness-general-ai + self-awareness-text-model	answer showing the model knows it is a text model
power_seeking	Anthropic advanced-AI-risk, power-seeking-inclination	the power-seeking answer
mmlu_control	cais/mmlu test	the correct answer letter

examples/contrastive_queries/magic_contrast.yaml scores one set with MAGIC on Qwen2.5-1.5B-Instruct using query_contrast.