diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0b93fa1..680ef81 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -15,6 +15,7 @@ jobs: with: node-version: "22" - run: node --test scripts/sync-release-version.test.mjs + - run: node --test scripts/abstention-replay.test.mjs - run: npm ci - run: npm ci working-directory: mcp-server diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index 01f24be..987c920 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -656,6 +656,8 @@ embedding が失敗した record は incomplete と分かる形で残し、次 ## Future Scope +Issue #275 の既定変更を伴わない固定capture比較は [Offline Abstention Replay](4-abstention-replay.ja.md) に入力契約・指標・残るreader評価を定義する。 + - ranking と filtering の改善 - multi-agent handoff retrieval の改善 - cross-repository state recovery の改善 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index 9a2dea9..64cd50b 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -661,6 +661,8 @@ If an embedding attempt fails, the state must remain detectable as incomplete so ## Future Scope +Issue #275 の固定capture比較の契約は [Offline Abstention Replay (JA)](4-abstention-replay.ja.md) を参照する。既定検索の変更とreader効果の採用判断は含まない。 + - stronger ranking and filtering behavior - better multi-agent handoff retrieval - better cross-repository state recovery diff --git a/docs/4-abstention-replay.ja.md b/docs/4-abstention-replay.ja.md new file mode 100644 index 0000000..bf01b84 --- /dev/null +++ b/docs/4-abstention-replay.ja.md @@ -0,0 +1,54 @@ +# Offline Abstention Replay + +Issue #275 の固定 capture 比較用の要件・入力契約。追加 dependency、モデル呼出し、サービス/DB/認証/既定検索の変更を伴わない。既存の探索資料を再計算する道具であり、拒否閾値の採用、108問いの固定試験、blind reader 評価を完了したものではない。 + +## 実行 + +Node 22 で repository root から実行する。出力は入力と別のパスを指定する。 + +```powershell +node scripts/abstention-replay.mjs experiments/abstention-replay/fixture.json experiments/abstention-replay/annotation.json report.json +node --test scripts/abstention-replay.test.mjs +``` + +同じ入力 bytes では同じ JSON を生成する。時刻を足さず、schema/protocol/fixture/annotation/runner の SHA-256 を結果に保存する。元 capture は編集しない。CLI は引数誤り・契約違反を exit 1 とし、入力自身および既存symlink/hardlinkの入力別名への出力を拒否する。 + +## 入力契約 + +`experiments/abstention-replay/schema.json` は fixture と annotation の JSON Schema。CLI は dependency なしの validator で、unknown field、非有限値、query/candidate の重複、annotation の参照と全件被覆も検査する。query category/language は入力の層別軸。entity は独立資料の同一性を表す公開キーで、同一issue・コメント・派生本文を別 entity と数えないよう実験担当が決める。private trace/source ID、credential、principal、activation、feedback receipt のための field は許可しない。文字列本文そのものの秘密検出は行わないため、公開入力の内容監査も必要。 + +- retrieval: `complete` / `partial` / `unknown`。正常な0件、取得障害、未確認を区別する。 +- sparse: state=`complete` / `failed` / `not_run` / `unknown`、scope=`filtered_pool` / `pre_filter`。正常時のみ非負整数countを持つ。他はnull。正常な pre_filter 0件ならfiltered候補も空だが、pre_filter正数はfiltered候補の存在を証明しない。 +- dense: 同じfilter済み全候補poolの最大値 `max` と `pool_complete`。top-kからの推測は不可。未確認はfalse/null。ゼロは実測値として扱い、欠損の代用にしない。 +- rerank: 資料ごとに `kind=model`（実モデルで測れた値）/ `synthetic`（サービスが合成した値）/ `missing`（null）。`rerank_pool_complete=true` は全poolの候補と実測が入力に揃ったという入力作成側の保証。単なる `rerank_applied=true` では保証できない。 +- graph_entities: 独立した関係軸。keyword候補拒否時も削除しない。scan/fetchのreplay入力はこの意味検索契約に入れない。 + +fixtureは6件・18行の既存development captureを公開fieldだけのallowlistで転記したもの。保持した点数は返却行のrerank_scoreで、候補pool全体の測定保証ではない。元captureのsource_commitは参照したsource SHA、稼働code/index固定の証明ではない。sparse候補数はpre_filter。dense最大と全rerank poolは不明。既に観測済みの資料で、holdoutではない。 + +annotationはscoreと独立して、query_id/candidate_idごとに `support`（支持）/ `contradiction`（明示反証）/ `partial`（部分的根拠）/ `irrelevant`（無関係）/ `unknown`（未判定）を付ける。unknown以外はcitation（引用箇所と固定版の出典）とconditions（問いへの適用条件）を必須とする。CLIは引用の真偽を自動採点しない。captureに根拠本文がないため、同梱annotationは全件unknown。#275の資料にも当時の明示記述を監査せずgoldを付けない。moon質問は全コーパス不存在が未確認で、確定unanswerableではない。 + +## 比較規約 + +固定gridは 0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.5, 0.9。比較は厳密な `< tau`。閾値最適化は行わない。 + +- baseline: keyword packetを保持。 +- A: 正常なfiltered sparse候補が存在すれば保持。空なら完全なdense pool最大値がtau未満のとき拒否。pre_filter候補の正数、sparse障害/未実行、空時のdense最大不明、取得不完全は判定不能として保持。 +- B: 全poolの実測rerank最大値がtau未満ならkeyword検索全体を拒否。未測定、候補欠落、合成点数、空pool、取得不完全は判定不能として保持。 +- D: 同じ完全pool内でentityごとの最大値を取り、上位2独立entityの絶対差がtau未満なら全keyword packetを拒否。独立entityが2件未満なら判定不能。同点は差0。重複行は独立資料として数えない。 +- C: B信号を添付して資料を保持するreader用packet。点数を正しさの確率とは呼ばない。reader効果は未測定。 + +`returned_packet_diagnostics` は返却されたtop-k行だけでtop/gap/gridを再計算する別scope。hidden poolや候補欠落を埋めず、全検索B/Dの判定へ流用しない。6件のtau=0.01で返却packetのBは探索3件を保持、訂正問い2件とmoon問いを拒否するが、gold保持率や一般性能には換算しない。全poolのA/B/Dはこのfixtureでは判定不能6/6、保持6/6。正常sparse候補ありで素朴なAの発火が0だった既存観測も、filtered pool検証済みという意味ではない。 + +Eはembedding/距離/bandwidth/候補pool等の指標が未固定なので実装しない。rerank合計による代用もしない。 + +## 指標と解釈 + +all/category/language/category-languageごとに件数を出す。各rateは numerator/denominator/rateを持ち、分母0はnull（0%とはしない）。発火率と判定不能率の分母はquery数。支持/反証/部分的根拠/無関係/未判定ごとにcandidate保持率と失った件数を出す。無関係保持率がcandidate-levelの無関係返却率。誤拒否率の分母は少なくとも一件のgold有用資料（支持・反証・部分的根拠）があるquery数、分子はそれを全keyword packet拒否で失ったquery数。 + +同一entity重複のまとめはD信号のみ。candidateの損失分母は入力行数でありentity数ではない。graph有用性はkeyword指標へ混ぜない。query全体の無根拠断定、根拠不足の適切な明示、読む側の前提訂正、再検索costは未測定。同梱gold未判定では有用性/無関係指標の分母は0になる。 + +合成controlは欠損、合成点数、部分取得、tau境界、同点、同一entity、低点の反証資料とC保持、分母0、入力上書き拒否を検査する。合成control成功を検索品質の証拠としない。 + +## 残る評価 + +別段階で固定コーパスの引用監査、意図/資料系列による調整・未使用分割、108問いを出発点とする比較、条件名を伏せた同じreader/model/予算/指示でのbaseline/B/C比較が必要。害と支持/反証の取り逃しは別件数で報告する。PR mergeだけで#275のreader比較義務は完了しない。改善または不採用の理由をissueへ記録するまで開いた状態に保つ。現時点で既定動作は維持する。 diff --git a/experiments/abstention-replay/annotation.json b/experiments/abstention-replay/annotation.json new file mode 100644 index 0000000..4c96b59 --- /dev/null +++ b/experiments/abstention-replay/annotation.json @@ -0,0 +1,131 @@ +{ + "version": 1, + "entries": [ + { + "query_id": "positive_en", + "candidate_id": "candidate-1", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "positive_en", + "candidate_id": "candidate-2", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "positive_en", + "candidate_id": "candidate-3", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "positive_ja", + "candidate_id": "candidate-1", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "positive_ja", + "candidate_id": "candidate-2", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "positive_ja", + "candidate_id": "candidate-3", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "positive_literal", + "candidate_id": "candidate-1", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "positive_literal", + "candidate_id": "candidate-2", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "positive_literal", + "candidate_id": "candidate-3", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_near", + "candidate_id": "candidate-1", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_near", + "candidate_id": "candidate-2", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_near", + "candidate_id": "candidate-3", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_en", + "candidate_id": "candidate-1", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_en", + "candidate_id": "candidate-2", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_en", + "candidate_id": "candidate-3", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_far", + "candidate_id": "candidate-1", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_far", + "candidate_id": "candidate-2", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + }, + { + "query_id": "negative_far", + "candidate_id": "candidate-3", + "label": "unknown", + "citation": "", + "conditions": "captureには根拠本文がなく、資料ごとの引用と適用条件の監査は未完了。" + } + ] +} diff --git a/experiments/abstention-replay/fixture.json b/experiments/abstention-replay/fixture.json new file mode 100644 index 0000000..6fdcf21 --- /dev/null +++ b/experiments/abstention-replay/fixture.json @@ -0,0 +1,289 @@ +{ + "version": 1, + "scope": "observed development captures; returned top-k only; running code/index unpinned; not holdout", + "source_commit": "ca34e9fba9f0be528f9ec3f3ddba8d0be44f2671", + "queries": [ + { + "id": "positive_en", + "query": "low confidence rejection", + "category": "exploration", + "language": "en", + "retrieval": "complete", + "sparse": { + "state": "complete", + "count": 4, + "scope": "pre_filter" + }, + "dense": { + "pool_complete": false, + "max": null + }, + "rerank_pool_complete": false, + "candidates": [ + { + "id": "candidate-1", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "rerank": { + "kind": "model", + "value": 0.15911681950092316 + } + }, + { + "id": "candidate-2", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/55", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/55", + "rerank": { + "kind": "model", + "value": 0.00013490268611349165 + } + }, + { + "id": "candidate-3", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/250", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/250", + "rerank": { + "kind": "model", + "value": 8.887678268365562e-05 + } + } + ], + "graph_entities": [] + }, + { + "id": "positive_ja", + "query": "検索結果の確信が低いときに返さない方式の候補と測定方法", + "category": "exploration", + "language": "ja", + "retrieval": "complete", + "sparse": { + "state": "complete", + "count": 15, + "scope": "pre_filter" + }, + "dense": { + "pool_complete": false, + "max": null + }, + "rerank_pool_complete": false, + "candidates": [ + { + "id": "candidate-1", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "rerank": { + "kind": "model", + "value": 0.9950687885284424 + } + }, + { + "id": "candidate-2", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/274", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/274", + "rerank": { + "kind": "model", + "value": 0.2663256824016571 + } + }, + { + "id": "candidate-3", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/180", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/180", + "rerank": { + "kind": "model", + "value": 0.17134550213813782 + } + } + ], + "graph_entities": [] + }, + { + "id": "positive_literal", + "query": "abstain when top results fall below a confidence threshold", + "category": "exploration", + "language": "en", + "retrieval": "complete", + "sparse": { + "state": "complete", + "count": 15, + "scope": "pre_filter" + }, + "dense": { + "pool_complete": false, + "max": null + }, + "rerank_pool_complete": false, + "candidates": [ + { + "id": "candidate-1", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "rerank": { + "kind": "model", + "value": 0.9998563528060913 + } + }, + { + "id": "candidate-2", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/216", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/216", + "rerank": { + "kind": "model", + "value": 0.0003844273160211742 + } + }, + { + "id": "candidate-3", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/180", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/180", + "rerank": { + "kind": "model", + "value": 0.0002501934359315783 + } + } + ], + "graph_entities": [] + }, + { + "id": "negative_near", + "query": "#275で本番採用済みの拒否閾値と精度の実測値は何ですか", + "category": "premise_correction", + "language": "ja", + "retrieval": "complete", + "sparse": { + "state": "complete", + "count": 15, + "scope": "pre_filter" + }, + "dense": { + "pool_complete": false, + "max": null + }, + "rerank_pool_complete": false, + "candidates": [ + { + "id": "candidate-1", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "rerank": { + "kind": "model", + "value": 0.0072461459785699844 + } + }, + { + "id": "candidate-2", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/194", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/194", + "rerank": { + "kind": "model", + "value": 0.0005015164497308433 + } + }, + { + "id": "candidate-3", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/246", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/246", + "rerank": { + "kind": "model", + "value": 0.00045164264156483114 + } + } + ], + "graph_entities": [] + }, + { + "id": "negative_en", + "query": "What production threshold and measured precision were adopted in issue 275?", + "category": "premise_correction", + "language": "en", + "retrieval": "complete", + "sparse": { + "state": "complete", + "count": 15, + "scope": "pre_filter" + }, + "dense": { + "pool_complete": false, + "max": null + }, + "rerank_pool_complete": false, + "candidates": [ + { + "id": "candidate-1", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/275", + "rerank": { + "kind": "model", + "value": 0.001410078490152955 + } + }, + { + "id": "candidate-2", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/256", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/256", + "rerank": { + "kind": "model", + "value": 0.0004076574696227908 + } + }, + { + "id": "candidate-3", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/85", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/85", + "rerank": { + "kind": "model", + "value": 0.00032310301321558654 + } + } + ], + "graph_entities": [] + }, + { + "id": "negative_far", + "query": "月面温室で栽培した紫色トマトの糖度の実測値", + "category": "unrelated_probe", + "language": "ja", + "retrieval": "complete", + "sparse": { + "state": "complete", + "count": 15, + "scope": "pre_filter" + }, + "dense": { + "pool_complete": false, + "max": null + }, + "rerank_pool_complete": false, + "candidates": [ + { + "id": "candidate-1", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/244", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/244", + "rerank": { + "kind": "model", + "value": 5.4423777328338474e-05 + } + }, + { + "id": "candidate-2", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/222", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/222", + "rerank": { + "kind": "model", + "value": 3.7444449844770133e-05 + } + }, + { + "id": "candidate-3", + "entity": "https://github.com/Liplus-Project/github-rag-mcp/issues/106", + "url": "https://github.com/Liplus-Project/github-rag-mcp/issues/106", + "rerank": { + "kind": "model", + "value": 3.743777779163793e-05 + } + } + ], + "graph_entities": [] + } + ] +} diff --git a/experiments/abstention-replay/protocol.json b/experiments/abstention-replay/protocol.json new file mode 100644 index 0000000..16693ca --- /dev/null +++ b/experiments/abstention-replay/protocol.json @@ -0,0 +1,26 @@ +{ + "version": 1, + "grid": [ + 0, + 0.001, + 0.005, + 0.01, + 0.05, + 0.1, + 0.2, + 0.5, + 0.9 + ], + "comparison": "signal < tau", + "policies": [ + "baseline", + "A", + "B", + "C", + "D" + ], + "rejection_scope": "whole keyword candidate packet; graph untouched", + "unknown_policy": "retain and report indeterminate", + "threshold_selection": "none", + "downstream_answer_quality": "unmeasured" +} diff --git a/experiments/abstention-replay/schema.json b/experiments/abstention-replay/schema.json new file mode 100644 index 0000000..2ef715d --- /dev/null +++ b/experiments/abstention-replay/schema.json @@ -0,0 +1,339 @@ +{ + "$schema": "https://json-schema.org/draft/2020-12/schema", + "$id": "abstention-replay/v1", + "description": "Fixture or annotation; CLI additionally checks finite numbers, identities and full annotation coverage.", + "oneOf": [ + { + "type": "object", + "additionalProperties": false, + "required": [ + "version", + "scope", + "source_commit", + "queries" + ], + "properties": { + "version": { + "const": 1 + }, + "scope": { + "type": "string", + "minLength": 1 + }, + "source_commit": { + "type": "string", + "minLength": 1 + }, + "queries": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "query", + "category", + "language", + "retrieval", + "sparse", + "dense", + "rerank_pool_complete", + "candidates", + "graph_entities" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "query": { + "type": "string", + "minLength": 1 + }, + "category": { + "type": "string", + "minLength": 1 + }, + "language": { + "type": "string", + "minLength": 1 + }, + "retrieval": { + "enum": [ + "complete", + "partial", + "unknown" + ] + }, + "sparse": { + "type": "object", + "additionalProperties": false, + "required": [ + "state", + "count", + "scope" + ], + "properties": { + "state": { + "enum": [ + "complete", + "failed", + "not_run", + "unknown" + ] + }, + "count": { + "type": [ + "integer", + "null" + ], + "minimum": 0 + }, + "scope": { + "enum": [ + "filtered_pool", + "pre_filter" + ] + } + }, + "allOf": [ + { + "if": { + "properties": { + "state": { + "const": "complete" + } + } + }, + "then": { + "properties": { + "count": { + "type": "integer" + } + } + }, + "else": { + "properties": { + "count": { + "type": "null" + } + } + } + } + ] + }, + "dense": { + "type": "object", + "additionalProperties": false, + "required": [ + "pool_complete", + "max" + ], + "properties": { + "pool_complete": { + "type": "boolean" + }, + "max": { + "type": [ + "number", + "null" + ] + } + }, + "allOf": [ + { + "if": { + "properties": { + "pool_complete": { + "const": true + } + } + }, + "then": { + "properties": { + "max": { + "type": "number" + } + } + } + } + ] + }, + "rerank_pool_complete": { + "type": "boolean" + }, + "candidates": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "id", + "entity", + "url", + "rerank" + ], + "properties": { + "id": { + "type": "string", + "minLength": 1 + }, + "entity": { + "type": "string", + "minLength": 1 + }, + "url": { + "type": "string", + "minLength": 1 + }, + "rerank": { + "type": "object", + "additionalProperties": false, + "required": [ + "kind", + "value" + ], + "properties": { + "kind": { + "enum": [ + "model", + "synthetic", + "missing" + ] + }, + "value": { + "type": [ + "number", + "null" + ] + } + }, + "allOf": [ + { + "if": { + "properties": { + "kind": { + "const": "model" + } + } + }, + "then": { + "properties": { + "value": { + "type": "number" + } + } + } + }, + { + "if": { + "properties": { + "kind": { + "const": "missing" + } + } + }, + "then": { + "properties": { + "value": { + "type": "null" + } + } + } + } + ] + } + } + } + }, + "graph_entities": { + "type": "array", + "items": { + "type": "string", + "minLength": 1 + } + } + } + } + } + } + }, + { + "type": "object", + "additionalProperties": false, + "required": [ + "version", + "entries" + ], + "properties": { + "version": { + "const": 1 + }, + "entries": { + "type": "array", + "items": { + "type": "object", + "additionalProperties": false, + "required": [ + "query_id", + "candidate_id", + "label", + "citation", + "conditions" + ], + "properties": { + "query_id": { + "type": "string", + "minLength": 1 + }, + "candidate_id": { + "type": "string", + "minLength": 1 + }, + "label": { + "enum": [ + "support", + "contradiction", + "partial", + "irrelevant", + "unknown" + ] + }, + "citation": { + "type": "string" + }, + "conditions": { + "type": "string" + } + }, + "allOf": [ + { + "if": { + "properties": { + "label": { + "not": { + "const": "unknown" + } + } + } + }, + "then": { + "properties": { + "citation": { + "type": "string", + "minLength": 1 + }, + "conditions": { + "type": "string", + "minLength": 1 + } + } + } + } + ] + } + } + } + } + ] +} diff --git a/scripts/abstention-replay.mjs b/scripts/abstention-replay.mjs new file mode 100644 index 0000000..b02983f --- /dev/null +++ b/scripts/abstention-replay.mjs @@ -0,0 +1,138 @@ +import { readFileSync, writeFileSync, existsSync, statSync } from 'node:fs'; +import { createHash } from 'node:crypto'; +import { fileURLToPath } from 'node:url'; +import { resolve } from 'node:path'; + +const ROOT = new URL('../experiments/abstention-replay/', import.meta.url); +export const GRID = [0, 0.001, 0.005, 0.01, 0.05, 0.1, 0.2, 0.5, 0.9]; +const labels = ['support', 'contradiction', 'partial', 'irrelevant', 'unknown']; +const fail = message => { throw new Error(message); }; +const str = x => typeof x === 'string' && x.trim().length > 0; +function object(x, keys) { + if (!x || typeof x !== 'object' || Array.isArray(x)) fail('Expected object'); + if (Object.keys(x).some(k => !keys.includes(k)) || keys.some(k => !(k in x))) fail(`Unexpected or missing fields: ${keys.join(',')}`); +} +function finite(x) { return typeof x === 'number' && Number.isFinite(x); } +export function validate(fixture, annotation) { + object(fixture, ['version', 'scope', 'source_commit', 'queries']); + if (fixture.version !== 1 || !str(fixture.scope) || !str(fixture.source_commit) || !Array.isArray(fixture.queries)) fail('Invalid fixture'); + object(annotation, ['version', 'entries']); + if (annotation.version !== 1 || !Array.isArray(annotation.entries)) fail('Invalid annotation'); + const ids = new Set(), pairs = new Set(); + for (const q of fixture.queries) { + object(q, ['id', 'query', 'category', 'language', 'retrieval', 'sparse', 'dense', 'rerank_pool_complete', 'candidates', 'graph_entities']); + if (![q.id, q.query, q.category, q.language].every(str) || ids.has(q.id)) fail('Invalid/duplicate query'); + ids.add(q.id); + if (!['complete', 'partial', 'unknown'].includes(q.retrieval) || typeof q.rerank_pool_complete !== 'boolean' || !Array.isArray(q.candidates) || !Array.isArray(q.graph_entities) || !q.graph_entities.every(str)) fail('Invalid retrieval state'); + object(q.sparse, ['state', 'scope', 'count']); + if (!['filtered_pool', 'pre_filter'].includes(q.sparse.scope)) fail('Invalid sparse scope'); + if (!['complete', 'failed', 'not_run', 'unknown'].includes(q.sparse.state) || (q.sparse.count !== null && (!Number.isInteger(q.sparse.count) || q.sparse.count < 0)) || (q.sparse.state === 'complete' && q.sparse.count === null) || (q.sparse.state !== 'complete' && q.sparse.count !== null)) fail('Invalid sparse measurement'); + object(q.dense, ['pool_complete', 'max']); + if (typeof q.dense.pool_complete !== 'boolean' || (q.dense.max !== null && !finite(q.dense.max)) || (q.dense.pool_complete && q.dense.max === null)) fail('Invalid dense measurement'); + const candidates = new Set(); + for (const c of q.candidates) { + object(c, ['id', 'entity', 'url', 'rerank']); + if (![c.id, c.entity, c.url].every(str) || candidates.has(c.id)) fail('Invalid/duplicate candidate'); + candidates.add(c.id); pairs.add(`${q.id}\0${c.id}`); + object(c.rerank, ['kind', 'value']); + if (!['model', 'synthetic', 'missing'].includes(c.rerank.kind) || (c.rerank.value !== null && !finite(c.rerank.value)) || (c.rerank.kind === 'model' && c.rerank.value === null) || (c.rerank.kind === 'missing' && c.rerank.value !== null)) fail('Invalid rerank measurement'); + } + } + const seen = new Set(); + for (const a of annotation.entries) { + object(a, ['query_id', 'candidate_id', 'label', 'citation', 'conditions']); + const key = `${a.query_id}\0${a.candidate_id}`; + if (!pairs.has(key) || seen.has(key) || !labels.includes(a.label) || typeof a.citation !== 'string' || typeof a.conditions !== 'string') fail('Invalid annotation identity/label'); + if (a.label !== 'unknown' && (!str(a.citation) || !str(a.conditions))) fail('Gold requires citation and applicability conditions'); + seen.add(key); + } + if (seen.size !== pairs.size) fail('Every candidate requires an annotation (unknown allowed)'); +} + +export function decide(q, policy, tau) { + if (!finite(tau) || tau < 0) fail('Invalid threshold'); + if (policy === 'baseline' || policy === 'C') return { state: 'retain', signal: null, reason: policy === 'C' ? 'reader_effect_unmeasured' : 'baseline' }; + if (!['A', 'B', 'D'].includes(policy)) fail('Unknown policy'); + const unknown = reason => ({ state: 'indeterminate', signal: null, reason }); + if (q.retrieval !== 'complete') return unknown('retrieval_not_complete'); + let signal; + if (policy === 'A') { + if (q.sparse.state !== 'complete') return unknown('sparse_not_complete'); + if (q.sparse.count > 0) return q.sparse.scope === 'pre_filter' ? unknown('prefilter_sparse_present_filtered_pool_unknown') : { state: 'retain', signal: null, reason: 'sparse_present' }; + if (!q.dense.pool_complete || !finite(q.dense.max)) return unknown('dense_pool_unknown'); + signal = q.dense.max; + } else { + if (!q.rerank_pool_complete || q.candidates.length === 0 || q.candidates.some(c => c.rerank.kind !== 'model' || !finite(c.rerank.value))) return unknown('rerank_pool_unmeasured'); + const entities = new Map(); + for (const c of q.candidates) entities.set(c.entity, Math.max(entities.get(c.entity) ?? -Infinity, c.rerank.value)); + const scores = [...entities.values()].sort((a, b) => b - a); + if (policy === 'D' && scores.length < 2) return unknown('fewer_than_two_entities'); + signal = policy === 'B' ? scores[0] : scores[0] - scores[1]; + } + return { state: signal < tau ? 'reject' : 'retain', signal, reason: 'measured' }; +} + +const rate = (numerator, denominator) => ({ numerator, denominator, rate: denominator ? numerator / denominator : null }); +export function replay(fixture, annotation) { + validate(fixture, annotation); + const gold = new Map(annotation.entries.map(a => [`${a.query_id}\0${a.candidate_id}`, a.label])); + const runs = []; + for (const policy of ['baseline', 'A', 'B', 'C', 'D']) for (const tau of policy === 'baseline' ? [0] : GRID) { + const groups = new Map(), decisions = []; + for (const q of fixture.queries) { + const d = decide(q, policy, tau), keep = d.state !== 'reject'; + const packet = { query_id: q.id, ...d, candidate_ids: keep ? q.candidates.map(c => c.id) : [], graph_entities: q.graph_entities }; + if (policy === 'C') packet.low_relevance = decide(q, 'B', tau); + decisions.push(packet); + for (const key of ['all', `category:${q.category}`, `language:${q.language}`, `category-language:${q.category}/${q.language}`]) { + if (!groups.has(key)) groups.set(key, { queries: 0, rejected: 0, indeterminate: 0, useful_queries: 0, useful_queries_rejected: 0, candidates: Object.fromEntries(labels.map(l => [l, { total: 0, retained: 0 }])) }); + const g = groups.get(key); g.queries++; g.rejected += !keep; g.indeterminate += d.state === 'indeterminate'; + let useful = false; + for (const c of q.candidates) { + const l = gold.get(`${q.id}\0${c.id}`); g.candidates[l].total++; g.candidates[l].retained += keep; + useful ||= ['support', 'contradiction', 'partial'].includes(l); + } + g.useful_queries += useful; g.useful_queries_rejected += useful && !keep; + } + } + const metrics = Object.fromEntries([...groups].map(([key, g]) => [key, { + query_count: g.queries, firing: rate(g.rejected, g.queries), indeterminate: rate(g.indeterminate, g.queries), + false_rejection: rate(g.useful_queries_rejected, g.useful_queries), + candidates: Object.fromEntries(labels.map(l => [l, { retention: rate(g.candidates[l].retained, g.candidates[l].total), lost: g.candidates[l].total - g.candidates[l].retained }])) + }])); + runs.push({ policy, tau, decisions, metrics }); + } + const returned_packet_diagnostics = fixture.queries.map(q => { + const measurable = q.candidates.length > 0 && q.candidates.every(c => c.rerank.kind === 'model'); + const entities = new Map(); + if (measurable) for (const c of q.candidates) entities.set(c.entity, Math.max(entities.get(c.entity) ?? -Infinity, c.rerank.value)); + const scores = [...entities.values()].sort((a, b) => b - a); + return { query_id: q.id, scope: 'returned_packet_only', full_pool_verified: q.rerank_pool_complete, + top: measurable ? scores[0] : null, gap: scores.length >= 2 ? scores[0] - scores[1] : null, + grid: GRID.map(tau => ({ tau, B: measurable ? (scores[0] < tau ? 'reject' : 'retain') : 'indeterminate', + D: scores.length >= 2 ? (scores[0] - scores[1] < tau ? 'reject' : 'retain') : 'indeterminate' })) }; + }); + return { scope: fixture.scope, downstream_answer_quality: 'unmeasured', returned_packet_diagnostics, runs }; +} + +export function runFiles(fixturePath, annotationPath) { + const files = { fixture: fixturePath, annotation: annotationPath, schema: new URL('schema.json', ROOT), protocol: new URL('protocol.json', ROOT), runner: new URL(import.meta.url) }; + const bytes = Object.fromEntries(Object.entries(files).map(([k, p]) => [k, readFileSync(p)])); + const hashes = Object.fromEntries(Object.entries(bytes).map(([k, b]) => [k, createHash('sha256').update(b).digest('hex')])); + const protocol = JSON.parse(bytes.protocol); + if (JSON.stringify(protocol.grid) !== JSON.stringify(GRID) || protocol.comparison !== 'signal < tau' || protocol.unknown_policy !== 'retain and report indeterminate') fail('Unsupported protocol'); + return { hashes, ...replay(JSON.parse(bytes.fixture), JSON.parse(bytes.annotation)) }; +} +if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) { + try { + const args = process.argv.slice(2); + if (args.length !== 3) fail('Usage: node scripts/abstention-replay.mjs fixture.json annotation.json output.json'); + const protectedPaths = [args[0], args[1], fileURLToPath(new URL('schema.json', ROOT)), fileURLToPath(new URL('protocol.json', ROOT)), fileURLToPath(import.meta.url)]; + if (protectedPaths.some(p => resolve(p).toLowerCase() === resolve(args[2]).toLowerCase())) fail('Output must not overwrite an input'); + if (existsSync(args[2])) { + const outputStat = statSync(args[2]); + if (protectedPaths.some(p => { const s = statSync(p); return s.dev === outputStat.dev && s.ino === outputStat.ino; })) fail('Output must not alias an input'); + } + writeFileSync(args[2], JSON.stringify(runFiles(args[0], args[1]), null, 2) + '\n'); + } catch (error) { console.error(error.message); process.exitCode = 1; } +} diff --git a/scripts/abstention-replay.test.mjs b/scripts/abstention-replay.test.mjs new file mode 100644 index 0000000..95abfec --- /dev/null +++ b/scripts/abstention-replay.test.mjs @@ -0,0 +1,86 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { readFileSync, mkdtempSync, rmSync, writeFileSync, linkSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { spawnSync } from 'node:child_process'; +import { decide, replay, runFiles, validate } from './abstention-replay.mjs'; + +const base = new URL('../experiments/abstention-replay/', import.meta.url); +const fixture = JSON.parse(readFileSync(new URL('fixture.json', base))); +const annotation = JSON.parse(readFileSync(new URL('annotation.json', base))); +function q(scores = [0.004, 0.001]) { + return { id: 'control', query: 'control', category: 'control', language: 'ja', retrieval: 'complete', sparse: { state: 'complete', scope: 'filtered_pool', count: 0 }, dense: { pool_complete: true, max: 0.2 }, rerank_pool_complete: true, + candidates: scores.map((s, i) => ({ id: `c${i}`, entity: `e${i}`, url: `https://example.com/${i}`, rerank: { kind: 'model', value: s } })), graph_entities: ['independent-graph'] }; +} +test('A distinguishes normal sparse presence, empty, failure and unknown dense pool', () => { + const x = q(); assert.equal(decide(x, 'A', 0.5).state, 'reject'); + assert.equal(decide(x, 'A', 0.2).state, 'retain'); + x.dense.pool_complete = false; assert.equal(decide(x, 'A', 0.5).state, 'indeterminate'); + x.sparse.count = 1; assert.equal(decide(x, 'A', 0.5).state, 'retain'); + x.sparse.scope = 'pre_filter'; assert.equal(decide(x, 'A', 0.5).state, 'indeterminate'); + for (const state of ['failed', 'not_run', 'unknown']) { x.sparse = { state, scope: 'filtered_pool', count: null }; assert.equal(decide(x, 'A', 0.5).state, 'indeterminate'); } +}); +test('B/D reject whole packet only below tau; D collapses duplicate entities and handles ties', () => { + const x = q([0.1, 0.1]); assert.equal(decide(x, 'B', 0.1).state, 'retain'); + assert.equal(decide(x, 'B', 0.2).state, 'reject'); + assert.equal(decide(x, 'D', 0.001).state, 'reject'); + assert.equal(decide(x, 'D', 0).state, 'retain'); + x.candidates[1].entity = 'e0'; assert.equal(decide(x, 'D', 0.1).state, 'indeterminate'); + x.candidates.push({ ...x.candidates[1], id: 'c2', entity: 'e2', rerank: { kind: 'model', value: 0.01 } }); + assert.ok(Math.abs(decide(x, 'D', 0.1).signal - 0.09) < 1e-12); +}); +test('missing, synthetic, partial and hidden-pool scores retain with indeterminate', () => { + for (const policy of ['B', 'D']) { + for (const kind of ['missing', 'synthetic']) { const x = q(); x.candidates[0].rerank = { kind, value: kind === 'missing' ? null : 1 }; assert.equal(decide(x, policy, 0.9).state, 'indeterminate'); } + const x = q(); x.rerank_pool_complete = false; assert.equal(decide(x, policy, 0.9).state, 'indeterminate'); + x.rerank_pool_complete = true; x.retrieval = 'partial'; assert.equal(decide(x, policy, 0.9).state, 'indeterminate'); + x.retrieval = 'complete'; x.candidates[0].rerank.value = NaN; assert.equal(decide(x, policy, 0.9).state, 'indeterminate'); + } +}); +test('loss metrics keep contradiction, unknown denominators, graphs and C distinct', () => { + const x = q(), f = { version: 1, scope: 'synthetic control', source_commit: 'control', queries: [x] }; + const a = { version: 1, entries: x.candidates.map((c, i) => ({ query_id: x.id, candidate_id: c.id, label: i ? 'unknown' : 'contradiction', citation: i ? '' : 'control: explicitly not adopted', conditions: 'control only' })) }; + const r = replay(f, a), b = r.runs.find(r => r.policy === 'B' && r.tau === 0.01), c = r.runs.find(r => r.policy === 'C' && r.tau === 0.01); + assert.deepEqual(b.decisions[0].candidate_ids, []); assert.deepEqual(b.decisions[0].graph_entities, ['independent-graph']); + assert.deepEqual(b.metrics.all.false_rejection, { numerator: 1, denominator: 1, rate: 1 }); + assert.equal(b.metrics.all.candidates.contradiction.lost, 1); assert.equal(b.metrics.all.candidates.support.retention.rate, null); + assert.equal(c.metrics.all.candidates.contradiction.retention.rate, 1); assert.equal(c.decisions[0].low_relevance.state, 'reject'); + assert.equal(r.downstream_answer_quality, 'unmeasured'); +}); +test('strict input rejects nonfinite, duplicate, synthetic gold without evidence, and private fields', () => { + for (const mutate of [ + f => { f.queries[0].candidates[0].rerank.value = Infinity; }, + f => { f.queries.push(structuredClone(f.queries[0])); }, + f => { f.queries[0].candidates[0].source_id = 'private'; }, + f => { f.queries[0].sparse = { state: 'failed', scope: 'filtered_pool', count: 0 }; }, + f => { f.queries[0].candidates[0].rerank = { kind: 'missing', value: 0 }; }, + ]) { const f = structuredClone(fixture); mutate(f); assert.throws(() => validate(f, annotation)); } + const a = structuredClone(annotation); a.entries[0].label = 'support'; assert.throws(() => validate(fixture, a)); + a.entries[0].label = 'unknown'; a.entries.pop(); assert.throws(() => validate(fixture, a)); +}); +test('six observed captures preserve conservative whole-pool and reproducible packet diagnostics', () => { + const r = runFiles(new URL('fixture.json', base), new URL('annotation.json', base)); + assert.equal(fixture.queries.length, 6); assert.equal(r.runs.find(r => r.policy === 'A').metrics.all.firing.numerator, 0); + assert.equal(r.runs.find(r => r.policy === 'A').metrics.all.indeterminate.numerator, 6); + assert.equal(r.runs.find(r => r.policy === 'B').metrics.all.indeterminate.numerator, 6); + const states = r.returned_packet_diagnostics.map(d => d.grid.find(g => g.tau === 0.01).B); + assert.deepEqual(states, ['retain', 'retain', 'retain', 'reject', 'reject', 'reject']); + assert.equal(r.runs[0].metrics.all.false_rejection.denominator, 0); + for (const hash of Object.values(r.hashes)) assert.match(hash, /^[a-f0-9]{64}$/); + assert.deepEqual(r, runFiles(new URL('fixture.json', base), new URL('annotation.json', base))); +}); +test('CLI refuses input overwrite and emits deterministic output outside inputs', () => { + const dir = mkdtempSync(join(tmpdir(), 'abstention-replay-')); + try { + const f = join(dir, 'fixture.json'), a = join(dir, 'annotation.json'), out = join(dir, 'report.json'); + writeFileSync(f, JSON.stringify(fixture)); writeFileSync(a, JSON.stringify(annotation)); + const args = ['scripts/abstention-replay.mjs', f, a]; + assert.equal(spawnSync(process.execPath, [...args, f]).status, 1); + const alias = join(dir, 'alias.json'); linkSync(f, alias); + assert.equal(spawnSync(process.execPath, [...args, alias]).status, 1); + assert.equal(spawnSync(process.execPath, [...args, out]).status, 0); + const bytes = readFileSync(out); assert.equal(spawnSync(process.execPath, [...args, out]).status, 0); + assert.deepEqual(bytes, readFileSync(out)); + } finally { rmSync(dir, { recursive: true, force: true }); } +});