lipluscodex · Oct 12, 2026 · Oct 12, 2026 · Oct 12, 2026
diff --git a/.gitattributes b/.gitattributes
@@ -0,0 +1,4 @@
+# Hash-pinned experiment inputs must survive Windows checkout byte-exactly.
+experiments/document-tail/fixture.json -text whitespace=cr-at-eol
+experiments/document-tail/provenance.json -text whitespace=cr-at-eol
+experiments/document-tail/source-* -text whitespace=cr-at-eol
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -16,6 +16,11 @@ jobs:
           node-version: "22"
       - run: node --test scripts/sync-release-version.test.mjs
       - run: node --test scripts/abstention-replay.test.mjs
+      - run: node --experimental-strip-types --test scripts/document-tail.test.mjs
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: python -m unittest discover -s scripts -p 'document_tail_test.py'
       - run: npm ci
       - run: npm ci
         working-directory: mcp-server

diff --git a/.gitignore b/.gitignore
@@ -2,3 +2,7 @@ node_modules/
 dist/
 .wrangler/
 .dev.vars
+
+# Offline experiment databases / Python caches
+*.sqlite
+__pycache__/
diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md
@@ -679,3 +679,7 @@ Issue #263 の調査で再現した issue_comment.deleted の部分削除: Vecto
 両 CD packaging job は bridge package、lock root、MCP Bundle manifest、MCP registry の npm version を同じ検証済み release tag から同期し、pack / publish 前に一致を assert する。依存版、schema、runtime 条件、Worker の挙動を保持する。対象 field と検証 command は [版数と公開成果物](installation.ja.md#versioning-and-published-artifacts) に定義する。commit 済み版は placeholder のままとし、trusted publishing は別変更とする。
 
 source の `docs/_Sidebar.md` は既存 Feedback Memory EN/JA ページを参照し、release 後の Wiki navigation に含める。
+
+### Offline document-tail comparison (#279)
+
+固定公開文書の後半発見を検討する opt-in sparse-only 実験は [Document Tail Replay](5-document-tail-replay.ja.md) の契約に従う。prefix8000 / fullbody / nonoverlap8000 の比較、入力・source hash、named target と exact literal source の別採点、静的rerank予算、節長分布を記録する。既定の検索・embedding・reranker・live索引は変更しない。
diff --git a/docs/0-requirements.md b/docs/0-requirements.md
@@ -684,3 +684,7 @@ Issue #259 version contract: live inline doc/wiki uses the returned-text SHA-256
 Both CD packaging jobs must derive the bridge package, lock root, MCP Bundle manifest and MCP registry npm versions from the same validated release tag and assert agreement before packing or publishing. Preserve dependency versions, schemas, runtime constraints and Worker behavior. [Versioning and published artifacts](installation.md#versioning-and-published-artifacts) owns the affected fields and validation command. Committed versions remain placeholders; trusted publishing is a separate change.
 
 The source `docs/_Sidebar.md` must link the existing Feedback Memory EN/JA pages so release wiki navigation includes them.
+
+### Offline document-tail comparison (#279)
+
+固定公開文書の後半発見を検討する opt-in sparse-only 実験は [Document Tail Replay](5-document-tail-replay.ja.md) の契約に従う。prefix8000 / fullbody / nonoverlap8000 の比較、入力・source hash、named target と exact literal source の別採点、静的rerank予算、節長分布を記録する。既定の検索・embedding・reranker・live索引は変更しない。
diff --git a/docs/5-document-tail-replay.ja.md b/docs/5-document-tail-replay.ja.md
@@ -0,0 +1,50 @@
+# Offline Document Tail Replay
+
+長い資料の後半を発見する候補の比較を、固定公開入力から再計算する opt-in CLI。既定検索、embedding、reranker、live DB、デプロイは変更しない。全文索引や分割方式を本番に採用する変更ではない。
+
+```sh
+# Node >=22.6 (TypeScript stripping), Python >=3.10 with SQLite FTS5
+python scripts/document-tail-replay.py --out /tmp/document-tail-new
+node --experimental-strip-types --test scripts/document-tail.test.mjs
+python -m unittest discover -s scripts -p 'document_tail_test.py'
+```
+
+`--out` は存在しない新規ディレクトリを指定する。CLIは原子的に作成し、既存出力・入力・DBを上書きしない。既存のsymlink先も拒否する。ネットワーク、認証、Workers、NGR共有DB、モデルを使わない。失敗時の部分出力も再利用しない。異なる新規dirで実行する。`prepared.json`、3 SQLite DB、`results.json`、`sections.json`、`REPORT.md` をそのdirに生成する。DBやpreparedをコミットしない。
+
+## 固定入力と再現契約
+
+`experiments/document-tail/fixture.json` は2026-10-13に独立captureした11公開文書 / 7事前queryの新規fixture。frozen corpus、WSLC、`corpora/` 配下は入力に含めない。公開文書中でcorpusに言及していても、そのcorpus自体は読み込まない。`provenance.json` は各文書のcommit、Git blob SHA、本文SHA256、immutable URLを保持し、parentがGitHub Contentsの固定refと照合した記録。runnerはネットワーク再取得せず、その本文hashとblobをローカル検証する。
+
+`protocol.json` はfixture/provenanceとsource snapshotのSHA256を固定する。sourceは github-rag-mcp `da23db3f077a2975e11fc4ab16cfab1556098045` のbyte-exact写し。segmentForFtsをその写しから実行し、escapeFtsQueryは同じ写しからTypeScript注釈のみを除いて抽出する。source snapshotは実行中のsrcとは別物。runnerは入力と全snapshot hash、各本文version hashを照合する。
+
+- prefix: `(path + '\n\n' + body).slice(0,8000)`、JavaScript UTF16 units。production snapshotと同じ切断。
+- full: 同じ入力全体。
+- chunks: 同じ入力の固定非重複 <=8000 units。surrogate pairを割る境界だけ1unit戻す。pathはchunk0のみ、重複・title再挿入なし。結合が元入力に一致することを検証する。
+- natural FTS5: `porter unicode61 remove_diacritics 2`、単一external-content列。strict ANDを先に、relaxed ORを後に採る。各tier内BM25昇順、rowidでtie break、全候補列挙後にbest rowでentity dedup。
+- 順位付けが使うのはqueryとoptional exact repo filterのみ。target/eligible-sourceは順位付け後の採点にだけ使用する。
+
+環境version（Node/ICU/Python/SQLite/compile options）、bytes/rows、build時間、prepare時間、1warmup +31queryのsample/median、全候補・snippet・match windowsを出す。ローカル時間・DB bytesは本番CPU/RAM/費用の保証ではない。Node/ICU/SQLite versionが変わるとtokenや順位が変わりうる。CIの固定順位検査はそのfixture契約であり汎用FTS品質を保証しない。
+
+## 指標と観測された範囲
+
+named target hit@1/@3は事前指定pathの順位。literal-source entity hitは同repoの原本文にexact query文字列がある任意entityの順位で、意味的goldではない。eligible entityがprefix由来で偶然rankすることもあるため、**top1のindex row自体にphraseがあるか**も別に記録する。negativeはこのfixture中のliteral不在を検査するだけで、productionで答えられないことを意味しない。`ngr_prefix` の `judgment relation` は個々のtokenがprefix内にあるgeneral-query controlであり、phrase-in-prefix controlではない。
+
+固定fixtureでのtail3件の再計算:
+
+| Arm | named target @1/@3 | literal-source entity @1/@3 | top1 index rowにliteral |
+|---|---|---|---|
+| prefix | 0/3, 1/3 | 1/3, 1/3 | 0/3 |
+| full | 3/3, 3/3 | 3/3, 3/3 | 3/3 |
+| chunks | 2/3, 3/3 | 3/3, 3/3 | 3/3 |
+
+webhook chunkの1位は別資料にも同じ根拠literalがあるため、named-target差からfullの品質優位を導かない。prefix/fullはproduction natural topK3の各branch cap、outer6、entity dedupとの14結果の一致を別監査する。chunkは全列挙/dedupでありproduction candidate capとは異なる。コード/trigram/dense/graph/RRFを含めた本番の検索経路は測定しない。
+
+## 静的根拠露出と節長
+
+現在のsnapshotのrerank内容予算は `1700-min(query.length,200)` UTF16 units。full入力と採点targetのwinning8000 chunkのprefixには3tailのphraseがどれも収まらない。見出し節候補はpath+改行2つ+節（見出しmarkup含む）の仮想prefixで、2/3が見える。NGRは19454-unitの節の後半にあり隠れる。これは節retrieval/rerankingを実行した結果ではない。
+
+節解析はfence外のATX h1-h6 / single-line setext h1/h2間を区切り、親子集約しない。非空preambleを含み、空preambleを別集計。indented code、blockquote、listのsetext候補を除外する簡易parserで、完全なCommonMark対応を主張しない。UTF16 nearest-rank percentilesを記録する。tail対象root3は74節、59節が<=1700。多数の短節があってもsection chunkだけで十分とは証明しない。
+
+## 残課題
+
+端-to-端モデル正答率、誤陽性のrelevance注釈、広いcorpusでの評価は未測定。採用前にはunchanged-hash backfill、deletion/replacement、provenance、chunk ID/offset、entity dedup、candidate budget、rerank evidence slice、return/fetch-source semantics、D1 statement/size/CPU制約を決める必要がある。chunk境界でphrase/tokenが分割される影響も未評価。
diff --git a/experiments/document-tail/fixture.json b/experiments/document-tail/fixture.json
diff --git a/experiments/document-tail/protocol.json b/experiments/document-tail/protocol.json
@@ -0,0 +1,41 @@
+{
+  "scope": "Isolated offline development candidate-discovery pilot, 11 public document fixtures / 7 previously declared queries; not holdout or frozen evaluation",
+  "queries_origin": "Fixed public fixture.json; seven predeclared queries and target labels retained from the 2026-10-13 pilot",
+  "source_sha": "da23db3f077a2975e11fc4ab16cfab1556098045",
+  "algorithm": {
+    "prefix": "Exact current prepareEmbeddingInput: (path + two newline characters + body).slice(0,8000), JavaScript UTF16 units",
+    "full": "Same path + two newlines + body, no clipping",
+    "chunks": "Same full sequence, fixed nonoverlap <=8000 UTF16 units, boundary moved back one unit when it bisects a surrogate pair; path only in first chunk, no overlap/no repeated title",
+    "segmentation": "Byte-exact pinned source-src-segment.ts, imported using Node TypeScript stripping; escapeFtsQuery extracted from pinned source with TypeScript annotations removed",
+    "tokenizer": "porter unicode61 remove_diacritics 2, external-content single natural FTS5 column; no code rows",
+    "rank": "Strict AND tier0 before relaxed OR tier1, bm25 ascending; exhaust branches, dedup document entity using best row, top3 scoring; deterministic rowid tie break",
+    "label_use": "Target paths used only after ranking; retrieval uses query text and optional exact repository filter",
+    "timing": "One warm query then31 repeated local queries; median includes snippet/highlight window extraction, sorting and dedup; DB build includes insertion/commit but excludes integrity check; preparation separate",
+    "negative": "Only literal absence from this fixture, never production unanswerability"
+  },
+  "variations_from_production": [
+    "Local SQLite/Python and Node ICU are not Cloudflare D1/workerd parity proof",
+    "Exhaustive branch collection instead of production per-branch topK/outer cap to prevent chunk rows consuming entity slots; prefix/full cap-equivalence audit separately",
+    "Added rowid tie break; production equal-score ordering unspecified",
+    "No dense, graph, RRF, crossencoder, abstention or end-to-end packet assessment"
+  ],
+  "unknowns": [
+    "Broader corpus/false-positive relevance annotation",
+    "Phrase/token split at chunk boundaries, no overlap",
+    "Deletion/replacement/provenance contracts and unchanged-hash backfill",
+    "Full sparse source vs embedding prefix vs rerank evidence slice/API fetch-source semantics",
+    "D1 size/statement/CPU constraints and runtime segmentation compatibility",
+    "Chunk source offsets, stable IDs/entity dedup and rerank/return window provenance"
+  ],
+  "fixture_sha256": "b6c44c1d505e1c3a5235495c20f071d0c49f2c93204deda07b31a952a817fea3",
+  "source_hashes": {
+    "source-migrations-0006_fts5_segmented_nat_index.sql": "3e2b53ffc1e947e1987141bd8feb7fcd9adbcf508244bc31d91d0ab636f38d59",
+    "source-src-fts.ts": "fd54209ecb0e3b6875a663dcb8eafb351f3d88b7305dc095ffcf6c5cffe3a0c7",
+    "source-src-pipeline-embed-doc.ts": "f1f8ec2a1efd4bb44921551801168c75aa359dcb75a75380d2a8a1836ff45947",
+    "source-src-pipeline-embedding.ts": "0e717d0bc0a457f298a00402cd379b977ff3276a84e04595ea0bb22a52471cf3",
+    "source-src-pipeline-hash.ts": "ab8595fa440f7c2a86ecac0e8684116a378e3a41722e90d7529d81a25e629956",
+    "source-src-rerank.ts": "e9693945311759e896fec6f6ce7d5eb575d77135546987366bdd1044aefc8148",
+    "source-src-segment.ts": "e5268f5d338d933ba0818d02b38648b848548ca1925266381f26bb3b5c33592a"
+  },
+  "provenance_sha256": "29c2f7b9c712dd02794790e1c9af3c7af889f0dad8b112a4943527518c4c78f8"
+}
diff --git a/experiments/document-tail/provenance.json b/experiments/document-tail/provenance.json
@@ -0,0 +1,101 @@
+[
+  {
+    "repo": "Liplus-Project/neuron-graph-rag",
+    "path": "docs/requirements.md",
+    "commit": "e51627243b7d30915bf0b5b8fd6e6c1704b8e2b8",
+    "blob_sha": "2fe74a27d8e1d0ab913b2f3c797ef70eb94adde5",
+    "content_sha256": "4e3bf59075021a65f46e2f2e9f09dfc654e7e0a489682b701fa80dad758eb277",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/neuron-graph-rag/blob/e51627243b7d30915bf0b5b8fd6e6c1704b8e2b8/docs/requirements.md"
+  },
+  {
+    "repo": "Liplus-Project/neuron-graph-rag",
+    "path": "docs/specifications/runtime-wheel.md",
+    "commit": "e51627243b7d30915bf0b5b8fd6e6c1704b8e2b8",
+    "blob_sha": "35efe37ae74afa699402a3eced6a21e23ce372dd",
+    "content_sha256": "5e51153c9e8c75fefaadaa3d3b4b67bbafca8a70d690cd75ce922d4cf285328e",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/neuron-graph-rag/blob/e51627243b7d30915bf0b5b8fd6e6c1704b8e2b8/docs/specifications/runtime-wheel.md"
+  },
+  {
+    "repo": "Liplus-Project/github-rag-mcp",
+    "path": "docs/0-requirements.md",
+    "commit": "da23db3f077a2975e11fc4ab16cfab1556098045",
+    "blob_sha": "9a58ecc8b1fbf0096bd90886f475f0a6df595503",
+    "content_sha256": "2fbb9191ea766fe7bd09eed5dc3f0d88aefdac0f26e6e72d5f90e7631f91f29a",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-rag-mcp/blob/da23db3f077a2975e11fc4ab16cfab1556098045/docs/0-requirements.md"
+  },
+  {
+    "repo": "Liplus-Project/github-rag-mcp",
+    "path": "docs/0-requirements.ja.md",
+    "commit": "da23db3f077a2975e11fc4ab16cfab1556098045",
+    "blob_sha": "987c92065275e73793e86cc80213c401e13b6b24",
+    "content_sha256": "86ab0352e8b15b04679cb045f56b6da1b3d0e21535c287bcdc7f42e7b6156420",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-rag-mcp/blob/da23db3f077a2975e11fc4ab16cfab1556098045/docs/0-requirements.ja.md"
+  },
+  {
+    "repo": "Liplus-Project/github-rag-mcp",
+    "path": "docs/2-feedback-memory.md",
+    "commit": "da23db3f077a2975e11fc4ab16cfab1556098045",
+    "blob_sha": "8597a23b9facbbbd8888df46cf063db89f312885",
+    "content_sha256": "02dc1d8f0c95eea85db64dd23f2e2d491a399262316e8535dbb59d21ba750c1b",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-rag-mcp/blob/da23db3f077a2975e11fc4ab16cfab1556098045/docs/2-feedback-memory.md"
+  },
+  {
+    "repo": "Liplus-Project/github-webhook-mcp",
+    "path": "docs/0-requirements.ja.md",
+    "commit": "347d8604a600e086ff0789bb01f49448c64cad41",
+    "blob_sha": "fb8311e564cd3e69f9860a7e6bbc5130ed546583",
+    "content_sha256": "905146f7c5d566a712471205cae8f5eb4234bf3f0fe0dc7589b6ac00e7cb755c",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-webhook-mcp/blob/347d8604a600e086ff0789bb01f49448c64cad41/docs/0-requirements.ja.md"
+  },
+  {
+    "repo": "Liplus-Project/github-webhook-mcp",
+    "path": "docs/0-requirements.md",
+    "commit": "347d8604a600e086ff0789bb01f49448c64cad41",
+    "blob_sha": "965a20dcf03273056a78fe23df18b45e904b519d",
+    "content_sha256": "0bcd2a072539fc801df6690784e0127cf1b9df4a8dd50122378a8b1224584fe6",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-webhook-mcp/blob/347d8604a600e086ff0789bb01f49448c64cad41/docs/0-requirements.md"
+  },
+  {
+    "repo": "Liplus-Project/github-webhook-mcp",
+    "path": "docs/_Sidebar.md",
+    "commit": "347d8604a600e086ff0789bb01f49448c64cad41",
+    "blob_sha": "d8a18c50a87d0a9403ad994c380c2cc55050dd80",
+    "content_sha256": "a7dd46d4e83426198e588f194c229d430a7832fbac5955f50e79bf7527a709d9",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-webhook-mcp/blob/347d8604a600e086ff0789bb01f49448c64cad41/docs/_Sidebar.md"
+  },
+  {
+    "repo": "Liplus-Project/github-rag-mcp",
+    "path": "docs/2-feedback-memory.ja.md",
+    "commit": "da23db3f077a2975e11fc4ab16cfab1556098045",
+    "blob_sha": "522fd9cb952e2756be6bec796ac2b51722cc1d93",
+    "content_sha256": "359e9af4f291b732d6b09b106bea6f1c5a4c4e40c9c651a836b626774ca8e847",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-rag-mcp/blob/da23db3f077a2975e11fc4ab16cfab1556098045/docs/2-feedback-memory.ja.md"
+  },
+  {
+    "repo": "Liplus-Project/github-rag-mcp",
+    "path": "docs/4-abstention-replay.ja.md",
+    "commit": "da23db3f077a2975e11fc4ab16cfab1556098045",
+    "blob_sha": "bf01b842970baad23a2092b60752d82e48cc00c5",
+    "content_sha256": "cac7b8f42ca4d280a2b745f26a56130559a104c66c11c9cc1bab01d64af851ff",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-rag-mcp/blob/da23db3f077a2975e11fc4ab16cfab1556098045/docs/4-abstention-replay.ja.md"
+  },
+  {
+    "repo": "Liplus-Project/github-webhook-mcp",
+    "path": "README.md",
+    "commit": "347d8604a600e086ff0789bb01f49448c64cad41",
+    "blob_sha": "d7f80a07e0cfeddce7bb6ed907bb35fd7fc95186",
+    "content_sha256": "10b1ce65da5b661f6ef689dedfe5066f175b5d0690103485e847ff649ef50e20",
+    "matches_live_capture": true,
+    "url": "https://github.com/Liplus-Project/github-webhook-mcp/blob/347d8604a600e086ff0789bb01f49448c64cad41/README.md"
+  }
+]