diff --git a/README.ja.md b/README.ja.md index 39c0900..b0f55ff 100644 --- a/README.ja.md +++ b/README.ja.md @@ -131,6 +131,7 @@ bot (`sender.login` が `[bot]` で終わる) と trim 後 10 文字未満の bo | `query` | string (省略可) | 自然言語クエリ。省略または空文字で scan モード。 | | `repo` | string | repository で絞り込み。フルスラッグ (`owner/repo`) の完全一致。短いリポジトリ名は1件もマッチせず、search モードはそれをレスポンスの `filters_unmatched` に `"repo"` として報告します。 | | `path_prefix` | string | repository-relative directory prefix で doc を絞り込み。`type: "doc"`、末尾 `/`、UTF-8で64 byte以内が必須。 | +| `doc_context` | `prefix` / `matched` | 既定 prefix。matched は非空 query と doc/wiki_doc が必須、scan/vector_ids はエラー。準備済み全文を検索し、rerank:false でも検証済みの有界 matched_context を返す。coverage は語一致前の scope を数え、未準備の後半は検索できない。live include_content は別欄。[契約](docs/7-matched-context.ja.md)。 | | `state` | `"open"` / `"closed"` / `"all"` | state で絞り込み (既定 `all`)。 | | `labels` | string[] | label 名で AND 絞り込み。 | | `milestone` | string | milestone title で絞り込み。 | diff --git a/README.md b/README.md index f0e54b9..ef682c6 100644 --- a/README.md +++ b/README.md @@ -131,6 +131,7 @@ Bot-authored comments (`sender.login` ending in `[bot]`) and comments shorter th | `query` | string (optional) | Natural-language query. Omit or empty = scan mode. | | `repo` | string | Filter by repository — full slug (`owner/repo`), exact match. A bare repository name matches nothing; search mode reports that as `"repo"` in the response's `filters_unmatched`. Pass it whenever the question is about a known repository — without it, a near-duplicate from another repository can rank first; leave it unset only when the question spans repositories. | | `path_prefix` | string | Filter docs by repository-relative directory prefix. Requires `type: "doc"`, a trailing `/`, and at most 64 UTF-8 bytes. | +| `doc_context` | `prefix` / `matched` | Default `prefix`. Explicit `matched` requires nonempty query and `type: doc` or `wiki_doc`, rejects scan/vector_ids, and searches prepared indexed full sources. Returns bounded verified `matched_context` even with `rerank: false`. Coverage counts the filtered indexed scope before term matching; unprepared tails remain unsearchable. Live `include_content` stays separate. | | `state` | `"open"` \| `"closed"` \| `"all"` | Filter by state (default `all`). | | `labels` | string[] | Filter by label names (AND). | | `milestone` | string | Filter by milestone title. | diff --git a/docs/0-requirements.ja.md b/docs/0-requirements.ja.md index ebe04e1..85df0d9 100644 --- a/docs/0-requirements.ja.md +++ b/docs/0-requirements.ja.md @@ -690,3 +690,12 @@ source の `docs/_Sidebar.md` は既存 Feedback Memory EN/JA ページを参照 #284 は既定 off の保存・operator 基盤を追加する。[全文 sparse source 契約](6-full-sparse-sources.ja.md)に従い prefix corpus、既定検索、graph、vector 数を維持する。matched 検索は #281 の別作業。 sparse operator の 2 経路は既存 Worker secret と `GITHUB_TOKEN` ヘッダの完全一致で認証し、secret が未設定・空なら401。専用 token と Authorization bearer は受け入れない。GitHub source 取得は従来の `GITHUB_TOKEN` を使う。既定 off の更新を含め、prefix revision CAS により版修復と owner 保存を同一 D1 batch に結ぶ。 +# 明示的な文書一致コンテキスト (#281) + +legacy code/trigram prefix 証拠は、未準備と準備済み owner の両方に共通 scope を LIMIT 前に適用して保持する。独立した全文 code storage/index は追加しない。code strict arm を他 arm と同じ順位定数で統合し、一資料一票を維持する。全文 nat の位置がない code-only prefix 候補は `code_only_prefix` 理由で原文抜粋をせず既存本文を返す。coverage は先行 SELECT の観測値であり、後続候補 batch と同一 snapshot を保証しない。 + +coverage の total=0 は metadata scope に対象なし。full=total による status=complete でも全 repo の準備完了を意味しない。total>0 で候補0件なら対象資料はあるが検索語に一致しない状態であり、両者は件数から区別する。 + +`doc_context: matched` は非空 query と doc/wiki_doc に限定する。省略および prefix は従来契約を維持する。検索は search_docs の共通 scope 内で、有効な full owner の全文 FTS と、それ以外の prefix FTS を LIMIT 前に分割する。strict を relaxed より先にし、各母集団内の順位を同じ RRF 定数で合成する。同一 vector は一票だけとする。coverage は語との一致前の対象全体を数え、未準備の後半は検索できない。全文読取り失敗時には unavailable を明示し、準備済みを除外しない従来 prefix 候補へ戻る。 + +抜粋は postfilter と base 本文供給、非空判定後の最大50件に限定する。狭い snippet/版/bytes を先に読み、keyword 順位と vector_id 順で raw 合計1MiB、資料 raw256KiB/segmented512KiB/owner1MiB を割り当てる。raw は最大5件ずつ取得し、位置 map は1件ずつ検証する。原文 path+2改行+body の連続1400 UTF16 units 以下を、版・hash・保存 segmentation 完全一致・一意 fragment・surrogate 境界検証後にモデルと matched_context の両方へ渡す。不成立は資料を残して prefix と理由を返す。indexed 抜粋と live include_content は別欄とする。予算は工学的初期値であり Cloud CPU/heap 限界の証明ではない。取り込みフラグ、保存/backfill、graph と frozen corpus は変更しない。 diff --git a/docs/0-requirements.md b/docs/0-requirements.md index 10624b5..9fa4d81 100644 --- a/docs/0-requirements.md +++ b/docs/0-requirements.md @@ -695,3 +695,12 @@ The source `docs/_Sidebar.md` must link the existing Feedback Memory EN/JA pages Issue #284 adds a default-off storage/operator foundation; its contract is [Full sparse sources](6-full-sparse-sources.ja.md). It preserves the prefix corpus, default retrieval, graph and vector count. Matched retrieval remains #281. The two sparse operator endpoints require an exact match between the `GITHUB_TOKEN` header and the existing Worker `GITHUB_TOKEN` secret, and return 401 when the secret is unset or empty. Dedicated tokens and Authorization bearer credentials are not accepted; GitHub source fetching retains `GITHUB_TOKEN`. Version repair and owner writes share a D1 batch guarded by the prefix revision, including default-off updates. +# Explicit matched document context (#281) + +Legacy code/trigram prefix evidence is retained for both unprepared and prepared owners, with the same canonical metadata scope before LIMIT. No full code storage/index is added. The strict code arm uses the same rank constant and deduplicates to one vote per vector. Code-only prefix hits without a full natural-language position keep their base text with `code_only_prefix` rather than claiming a matched excerpt. Coverage is an earlier SELECT observation, not a guaranteed shared snapshot with the subsequent candidate batch. + +Coverage total=0 means the metadata scope is empty; its status=complete (full=total) does not imply repository-wide preparation. A nonempty scope with no candidates instead means no term matches. Counts distinguish these cases. + +`doc_context: matched` requires a nonempty query and doc/wiki_doc type; prefix and omission preserve existing retrieval. The canonical metadata scope separates current full owners from prefix-only owners before LIMIT. Strict precedes relaxed; independent corpus ranks use the same RRF constant and each vector contributes once. Coverage counts the scope before term matching. Unprepared tails cannot be searched; unavailable full retrieval falls back to every legacy prefix owner. + +After postfilters, base text supply and nonempty checks, at most 50 candidates may resolve excerpts. Narrow snippets/version/bytes precede raw reads, with a 1 MiB raw request budget, 256 KiB raw / 512 KiB segmented / 1 MiB owner per source, batches of five and one position map at a time. Verified excerpts are contiguous original slices of at most 1400 UTF16 units, with exact path, source version, body/source/raw/excerpt hashes, saved segmentation equality, unique fragments and safe surrogate boundaries. The model and matched_context receive the same text; live include_content remains separate. Failed validation retains the prefix candidate and supplies a reason. Budgets are initial engineering limits, not measured Cloud CPU/heap guarantees. Storage, backfill, ingest enablement, graph and frozen corpora remain unchanged. diff --git a/docs/6-full-sparse-sources.ja.md b/docs/6-full-sparse-sources.ja.md index 32a6a92..2ebd585 100644 --- a/docs/6-full-sparse-sources.ja.md +++ b/docs/6-full-sparse-sources.ja.md @@ -1,5 +1,7 @@ # Doc/wiki 全文 sparse source 基盤 +明示的な `doc_context: matched` 読取りは [一致コンテキスト](7-matched-context.ja.md) が所有する。保存フラグ false でも準備済み source を読める。省略/prefix の既定検索は以下の旧経路を維持する。 + Issue #284 の保存・運用契約。通常検索への接続は #281 の別作業。 `DOC_SPARSE_CONTEXT_ENABLED` は省略時 false。false では ingest は全文の生成・取得・追加 D1 呼出しを行わない。8000 UTF-16 unit の prefix、通常 nat FTS corpus/IDF、1 資料 1 vector、fetch、graph は維持する。既定 off の ingest と検索は migration 0009 の新 schema にアクセスしない。migration 0009 は operator 経路と enable より先に適用する。検索だけの旧経路は全文テーブルを読まない。 diff --git a/docs/7-matched-context.ja.md b/docs/7-matched-context.ja.md new file mode 100644 index 0000000..a9f17df --- /dev/null +++ b/docs/7-matched-context.ja.md @@ -0,0 +1,19 @@ +# 文書の一致コンテキスト + +`doc_context: matched` は `query` と `type: doc|wiki_doc` を明示する。`rerank: false` や単一候補でも一致抜粋を返す。省略/`prefix` は従来の検索経路を使う。既存の全文保存フラグは取り込みの制御であり、この読取りの条件にはしない。準備済み少数 source でも利用できる。 + +`doc_context_coverage` の total/full/prefix_only/unprepared/out_of_range/stale は query 語との一致前の metadata scope 件数。status は complete/partial/unprepared/unavailable。全文索引読取り失敗時は unavailable とし、準備済みも含む従来 prefix 候補を返す。未準備資料の後半は検索できない。 + +coverage は候補 batch の直前に別 SELECT で読むため、同時更新時には候補と件数の観測時点が少し異なる。候補の全文/prefix 分割は同一 batch で読み、raw を再読した版は再検証する。 + +total=0 は metadata scope の対象なし。status=complete は full=total の条件なので、空 scope でも complete になるが全 repo 準備完了の意味ではない。total>0 で結果0件なら対象資料は存在し、語との一致がない状態。件数で区別する。 + +legacy code/trigram prefix arm は、未準備/準備済み owner のどちらも検索する補助証拠。共通 scope を LIMIT 前に適用し、strict の順位として統合して一資料一票にする。全文 code index は追加しない。全文 nat の位置を得られない code-only prefix hit は `code_only_prefix` と既存本文を返し、matched_context を生成しない。 + +metadata 正本 search_docs の repo/type/state/milestone/path_prefix/labels/assignee/since/until を共通 SQL で適用する。full owner は版・取得時点・revision・index_version の一致が必要。それ以外を prefix 系統に含める。strict/relaxed 各 tier 内の順位を 1/(60+rank) とし、異なる corpus の BM25 生点数を比較せず、同点は vector_id、同一 vector は一票。全 sparse を internalTopK 最大50へ絞り、既存 dense/sparse 融合へ渡す。 + +`matched_context.text` は indexed 原文 path+2改行+body の連続 slice。start/end はその UTF16 座標、最大1400 units、source_version と body/source/raw/excerpt hash を伴う。同じ text がモデル入力になる。`include_content` の最新 GitHub 本文とは別欄。graph_results、stored fetch は従来の所有を維持する。 + +候補の postfilter/base 本文/非空判定後に最大50件を選び、keyword 順位で狭い source 情報を検査する。raw 合計1MiB、資料 raw256KiB/segmented512KiB/owner1MiB、raw batch 最大5、位置 map 同時1。大きい候補は skip し後続を扱う。予算は初期工学的制限であり、実 heap/GC/Cloud CPU 適合の証明ではない。 + +版や hash、path、byte 数、保存 segmentation と再計算が完全一致し、一意 fragment が安全な surrogate 境界に写像される場合だけ抜粋を返す。曖昧fragment・更新/削除競合・未準備・範囲外・dense-only/位置なし・予算・読取り失敗等では matched_context_fallback と既存本文を維持し、資料を捨てない。位置 map は保存しない。取り込みフラグ変更、全 corpus backfill、secret、production 操作をこの検索機能へ含めない。 diff --git a/mcp-server/server/search-schema.json b/mcp-server/server/search-schema.json index f3103cf..c875790 100644 --- a/mcp-server/server/search-schema.json +++ b/mcp-server/server/search-schema.json @@ -3,6 +3,15 @@ "$schema": "https://json-schema.org/draft/2020-12/schema", "type": "object", "properties": { + "doc_context": { + "default": "prefix", + "description": "Document context. Default prefix preserves existing search. matched requires a nonempty query and type=doc or wiki_doc; scan and vector_ids are rejected. Searches prepared indexed full sources and returns a verified bounded matched_context separately from live include_content, even with rerank=false. doc_context_coverage reports the filtered indexed population before term matching; total=0 means an empty metadata scope, distinct from no term matches. Coverage is read before the candidate batch and may differ under concurrent updates. Unprepared tails cannot be searched. Legacy code-only prefix evidence is retained with code_only_prefix fallback when no full natural-language position exists. Failures retain prefix candidates with explicit fallback reasons.", + "type": "string", + "enum": [ + "prefix", + "matched" + ] + }, "query": { "description": "Natural language search query. When omitted or empty, the tool switches to metadata-only scan mode and results are ordered by the timestamp implied by sort (default sort=\"updated_desc\" for empty query).", "type": "string", diff --git a/src/fts.ts b/src/fts.ts index b73ae9f..f771e07 100644 --- a/src/fts.ts +++ b/src/fts.ts @@ -384,6 +384,41 @@ export interface FtsFilter { state?: "open" | "closed" | "published" | "active"; milestone?: string; pathPrefix?: string; + labels?: string[]; + assignee?: string; + since?: string; + until?: string; +} + +/** Shared canonical metadata scope; matched retrieval also pushes postfilters before LIMIT. */ +export function ftsScope(filter?: FtsFilter): { sql: string; params: (string | number)[] } { + const clauses: string[] = []; + const params: (string | number)[] = []; + for (const [key, column] of [['repo', 'repo'], ['type', 'type'], ['state', 'state'], ['milestone', 'milestone']] as const) { + if (filter?.[key]) { clauses.push(`d.${column} = ?`); params.push(filter[key]!); } + } + if (filter?.pathPrefix) { + const { lower, upper } = pathPrefixRange(filter.pathPrefix); + clauses.push('d.doc_path >= ? AND d.doc_path < ?'); params.push(lower, upper); + } + for (const label of filter?.labels ?? []) { + clauses.push(csvContains('labels')); + params.push(label); + } + if (filter?.assignee) { + clauses.push(csvContains('assignees')); + params.push(filter.assignee); + } + if (filter?.since) { clauses.push('d.updated_at >= ?'); params.push(filter.since); } + if (filter?.until) { clauses.push('d.updated_at < ?'); params.push(filter.until); } + return { sql: clauses.length ? ` AND ${clauses.join(' AND ')}` : '', params }; +} + +function csvContains(column: 'labels' | 'assignees'): string { + return `EXISTS (WITH RECURSIVE parts(value,rest) AS ( + SELECT '',d.${column} || ',' UNION ALL + SELECT substr(rest,1,instr(rest,',')-1),substr(rest,instr(rest,',')+1) FROM parts WHERE rest<>'' + ) SELECT 1 FROM parts WHERE trim(value)=?)`; } /** @@ -421,31 +456,7 @@ export async function queryFts( // Build dynamic WHERE clause for metadata filters. Shared by both branches; only // the leading MATCH parameter differs, so it is bound per branch below. - const whereClauses: string[] = []; - const params: (string | number)[] = []; - if (filter?.repo) { - whereClauses.push("d.repo = ?"); - params.push(filter.repo); - } - if (filter?.type) { - whereClauses.push("d.type = ?"); - params.push(filter.type); - } - if (filter?.state) { - whereClauses.push("d.state = ?"); - params.push(filter.state); - } - if (filter?.milestone) { - whereClauses.push("d.milestone = ?"); - params.push(filter.milestone); - } - if (filter?.pathPrefix) { - const { lower, upper } = pathPrefixRange(filter.pathPrefix); - whereClauses.push("d.doc_path >= ? AND d.doc_path < ?"); - params.push(lower, upper); - } - const whereSql = - whereClauses.length > 0 ? ` AND ${whereClauses.join(" AND ")}` : ""; + const { sql: whereSql, params } = ftsScope(filter); // UNION ALL arms so each tokenizer contributes hits. The nat arms read the v3 index // created by migration 0006 (external content = the segmented `content_fts` column); diff --git a/src/matched-context.ts b/src/matched-context.ts new file mode 100644 index 0000000..a77b18b --- /dev/null +++ b/src/matched-context.ts @@ -0,0 +1,216 @@ +import { escapeFtsQuery, ftsScope, queryFts, type FtsFilter, type FtsHit } from './fts.js'; +import { DOC_SPARSE_INDEX_VERSION, MAX_RAW_BYTES, MAX_SEGMENTED_BYTES, MAX_ROW_BYTES, utf8Bytes, gitBlobSha } from './doc-sparse.js'; +import { segmentForFts, needsSegmentation, getSegmenter } from './segment.js'; +import { sha256Hex } from './pipeline/hash.js'; + +export const RAW_REQUEST_BUDGET = 1024 * 1024; +export const RAW_BATCH_SIZE = 5; +export const MATCHED_MAX_CANDIDATES = 50; +export const EXCERPT_UNITS = 1400; +const ELLIPSIS = '\u0001'; +type Row = Record; +export interface Coverage { + status: 'complete' | 'partial' | 'unprepared' | 'unavailable'; + total: number | null; full: number | null; prefix_only: number | null; + unprepared: number | null; out_of_range: number | null; stale: number | null; + reason?: string; +} +export interface MatchedContext { + text: string; start: number; end: number; coordinate_system: 'path_newlines_body_utf16'; + source_version: string; body_hash: string; source_hash: string; raw_hash: string; excerpt_hash: string; +} +export interface ContextOutcome { matched_context?: MatchedContext; matched_context_fallback?: string } +export interface MatchedSearch { hits: FtsHit[]; sources: Map; coverage: Coverage } +const ownerCurrent = `s.source_type=d.type AND s.source_version=d.source_version + AND s.prefix_indexed_at=d.indexed_at AND s.prefix_revision=d.source_revision`; +const fullOwner = `${ownerCurrent} AND s.coverage='full' AND s.index_version='${DOC_SPARSE_INDEX_VERSION}'`; +const validFullExists = `EXISTS (SELECT 1 FROM doc_sparse_sources s WHERE s.vector_id=d.vector_id AND ${fullOwner})`; +export const unavailableCoverage = (): Coverage => ({ status:'unavailable', total:null, full:null, + prefix_only:null, unprepared:null, out_of_range:null, stale:null, reason:'full_index_unavailable' }); + +export async function matchedCoverage(db: D1Database, filter: FtsFilter): Promise { + const scope = ftsScope(filter); + const r = (await db.prepare(`SELECT COUNT(*) AS total, + COALESCE(SUM(CASE WHEN ${fullOwner} THEN 1 ELSE 0 END),0) AS full, + COALESCE(SUM(CASE WHEN s.vector_id IS NULL THEN 1 ELSE 0 END),0) AS unprepared, + COALESCE(SUM(CASE WHEN ${ownerCurrent} AND s.index_version='${DOC_SPARSE_INDEX_VERSION}' AND s.coverage='out_of_range' THEN 1 ELSE 0 END),0) AS out_of_range + FROM search_docs d LEFT JOIN doc_sparse_sources s ON s.vector_id=d.vector_id + WHERE d.type IN ('doc','wiki_doc')${scope.sql}`).bind(...scope.params).first())!; + const total=Number(r.total), full=Number(r.full), unprepared=Number(r.unprepared), out_of_range=Number(r.out_of_range); + return { status: full === total ? 'complete' : full > 0 ? 'partial' : 'unprepared', + total, full, prefix_only:total-full, unprepared, out_of_range, stale:total-full-unprepared-out_of_range }; +} + +function hit(r: Row): FtsHit { + return { vectorId:String(r.vector_id), repo:String(r.repo), type:String(r.type), state:String(r.state), + labels:String(r.labels), milestone:String(r.milestone), assignees:String(r.assignees), updatedAt:String(r.updated_at), + number:Number(r.number), tagName:String(r.tag_name), docPath:String(r.doc_path), commitSha:String(r.commit_sha), + filePath:String(r.file_path), fileStatus:String(r.file_status), commitDate:String(r.commit_date), + commitAuthor:String(r.commit_author), commentId:Number(r.comment_id), reviewId:Number(r.review_id), + content:String(r.content), score:Number(r.score) }; +} + +/** Separate corpora never compare BM25 magnitudes. One tier-rank vote per vector. */ +export function mergeSparseTiers(arms: { tier:number; rows:Row[] }[], topK:number): Row[] { + const votes = new Map(); + for (const arm of arms) arm.rows.forEach((row, i) => { + const id=String(row.vector_id), vote=1/(60+i+1), old=votes.get(id); + if (!old || arm.tier < old.tier || (arm.tier === old.tier && vote > old.vote)) votes.set(id,{tier:arm.tier,vote,row}); + }); + return [...votes.values()].sort((a,b)=>a.tier-b.tier || b.vote-a.vote || + (String(a.row.vector_id)v.row); +} + +export async function queryMatchedFts(db:D1Database, query:string, topK:number, filter:FtsFilter, denseOnly=false):Promise { + try { + const coverage=await matchedCoverage(db,filter), sources=new Map(); + if (denseOnly) return { hits:[], sources, coverage }; + const scope=ftsScope(filter), text=segmentForFts(query), strict=escapeFtsQuery(text), relaxed=escapeFtsQuery(text,'any'); + const matches=strict===relaxed?[strict]:[strict,relaxed]; + const arms:{tier:number;rows:Row[]}[]=[]; + // Candidate arms share a batch snapshot. Coverage is a separate, earlier SELECT observation. + const statements=matches.flatMap(match=>[true,false].map(full=>{ + const table=full?'doc_sparse_fts_v1':'search_docs_nat_fts_v3'; + return db.prepare(`SELECT d.*, bm25(${table}) AS score${full ? `, + snippet(${table},0,'','','${ELLIPSIS}',32) AS fragment, + s.source_version AS owner_version,s.prefix_indexed_at,s.prefix_revision,s.index_version, + s.raw_hash,s.body_hash,s.source_hash,s.raw_bytes,s.segmented_bytes,s.row_bytes` : ''} + FROM ${table} f ${full ? 'JOIN doc_sparse_sources s ON s.rowid=f.rowid JOIN search_docs d ON d.vector_id=s.vector_id' : 'JOIN search_docs d ON d.rowid=f.rowid'} + WHERE ${table} MATCH ? AND d.type IN ('doc','wiki_doc')${scope.sql} + AND ${full?fullOwner:`NOT ${validFullExists}`} + ORDER BY score ASC,d.vector_id ASC LIMIT ?`).bind(match,...scope.params,topK); + })); + // Legacy code-prefix evidence can exist on either owner population. It supplies + // no full-source position and never creates a second vote for an existing vector. + const codeArm=statements.length; + statements.push(db.prepare(`SELECT d.*,bm25(search_docs_code_fts_v2) AS score + FROM search_docs_code_fts_v2 f JOIN search_docs d ON d.rowid=f.rowid + WHERE search_docs_code_fts_v2 MATCH ? AND d.type IN ('doc','wiki_doc')${scope.sql} + ORDER BY score ASC,d.vector_id ASC LIMIT ?`).bind(escapeFtsQuery(query),...scope.params,topK)); + const results=await db.batch(statements); + results.forEach((r,i)=>{ const rows=r.results ?? []; arms.push({tier:i===codeArm?0:Math.floor(i/2),rows}); + if(i===codeArm) return; + if(i%2===0) for(const row of rows) if(!sources.has(String(row.vector_id))) sources.set(String(row.vector_id),row); + }); + const merged=mergeSparseTiers(arms,topK); + const codeIds=new Set((results[codeArm].results ?? []).map(row=>String(row.vector_id))); + merged.forEach((row,i)=>{ + const id=String(row.vector_id); + if(!sources.has(id) && codeIds.has(id)) sources.set(id,{position_kind:'code_only_prefix'}); + const source=sources.get(id);if(source) source.keyword_rank=i+1; + }); + return { hits:merged.map(hit), sources, coverage }; + } catch { + // Include all legacy prefix owners on failure, including prepared ones. + return { hits:denseOnly?[]:await queryFts(db,query,topK,filter), sources:new Map(), coverage:unavailableCoverage() }; + } +} + +function safeBoundary(raw:string,index:number):boolean { + return !(index>0 && indexs.length>0); + if(fragments.length!==1) return 'ambiguous_fragment'; + const fragment=fragments[0], at=stored.indexOf(fragment); + if(at<0 || stored.lastIndexOf(fragment)!==at) return 'ambiguous_fragment'; + const spans:{a:number;b:number;raw:number}[]=[]; + if(!needsSegmentation(raw)) spans.push({a:0,b:raw.length,raw:0}); + else { + let pos=0; + for(const part of getSegmenter().segment(raw)) { + const text=part.segment.trim(); if(!text) continue; + if(spans.length) pos++; + spans.push({a:pos,b:pos+text.length,raw:part.index+part.segment.indexOf(text)}); pos+=text.length; + } + } + const first=spans.find(s=>at>=s.a && atendAt>s.a && endAt<=s.b); + if(!first || !last) return 'invalid_span'; + const lo=first.raw+at-first.a, hi=last.raw+endAt-last.a; + if(hi<=lo || hi-lo>EXCERPT_UNITS || !safeBoundary(raw,lo) || !safeBoundary(raw,hi)) return 'invalid_span'; + let start=Math.max(0,lo-Math.floor((EXCERPT_UNITS-(hi-lo))/2)); + if(!safeBoundary(raw,start)) start++; + let end=Math.min(raw.length,start+EXCERPT_UNITS); + if(!safeBoundary(raw,end)) end--; + if(end { + for(const key of ['vector_id','repo','type','doc_path','source_version','indexed_at','source_revision','owner_version','prefix_indexed_at','prefix_revision','index_version','raw_hash','body_hash','source_hash','raw_bytes','segmented_bytes','row_bytes']) { + if(narrow[key]!==row[key]) return 'version_mismatch'; + } + const raw=row.raw, segmented=row.segmented; + if(typeof raw!=='string' || typeof segmented!=='string') return 'missing_source'; + if(utf8Bytes(raw)!==Number(row.raw_bytes) || utf8Bytes(segmented)!==Number(row.segmented_bytes)) return 'byte_mismatch'; + const prefix=`${row.doc_path}\n\n`; + if(!raw.startsWith(prefix)) return 'path_mismatch'; + const body=raw.slice(prefix.length), bodyHash=await sha256Hex(body), rawHash=await sha256Hex(raw); + const sourceHash=row.type==='doc'?await gitBlobSha(body):bodyHash; + if(bodyHash!==row.body_hash || rawHash!==row.raw_hash || sourceHash!==row.source_hash || sourceHash!==row.source_version) return 'hash_mismatch'; + const bounds=excerptBounds(raw,segmented,String(narrow.fragment)); + if(typeof bounds==='string') return bounds; + const text=raw.slice(bounds.start,bounds.end); + return {text,...bounds,coordinate_system:'path_newlines_body_utf16',source_version:sourceHash, + body_hash:bodyHash,source_hash:sourceHash,raw_hash:rawHash,excerpt_hash:await sha256Hex(text)}; +} + +/** Narrow snapshots come from sparse search. Only selected, nonempty base candidates read raw. */ +export async function resolveMatchedContexts(db:D1Database,ids:string[],sources:Map,unavailable=false):Promise> { + const outcomes=new Map(), selected=ids.slice(0,MATCHED_MAX_CANDIDATES); + const fallback=(id:string,reason:string)=>outcomes.set(id,{matched_context_fallback:reason}); + const noPosition = new Map(); + if(!unavailable && selected.some(id=>!sources.has(id))) { + try { + const missing=selected.filter(id=>!sources.has(id)); + const rows=await db.prepare(`SELECT d.vector_id, CASE WHEN s.vector_id IS NULL THEN 'unprepared' + WHEN NOT (${ownerCurrent}) OR s.index_version<>'${DOC_SPARSE_INDEX_VERSION}' THEN 'stale_source' + WHEN s.coverage='out_of_range' THEN 'out_of_range' ELSE 'no_full_sparse_position' END AS reason + FROM search_docs d LEFT JOIN doc_sparse_sources s ON s.vector_id=d.vector_id + WHERE d.vector_id IN (${missing.map(()=>'?').join(',')})`).bind(...missing).all(); + for(const row of rows.results ?? []) noPosition.set(String(row.vector_id),String(row.reason)); + } catch { for(const id of selected) if(!sources.has(id)) noPosition.set(id,'source_read_failed'); } + } + let remaining=RAW_REQUEST_BUDGET; + const admitted:string[]=[]; + const budgetOrder=[...selected].sort((a,b)=>Number(sources.get(a)?.keyword_rank ?? Infinity)-Number(sources.get(b)?.keyword_rank ?? Infinity) || (ab?1:0)); + for(const id of budgetOrder) { + const s=sources.get(id); + if(s?.position_kind==='code_only_prefix') {fallback(id,'code_only_prefix');continue;} + if(!s) { fallback(id,unavailable?'full_index_unavailable':noPosition.get(id) ?? 'missing_source'); continue; } + const raw=Number(s.raw_bytes), segmented=Number(s.segmented_bytes), owner=Number(s.row_bytes); + if(!Number.isSafeInteger(raw)||!Number.isSafeInteger(segmented)||!Number.isSafeInteger(owner)|| + raw<0||segmented<0||owner<0||raw>MAX_RAW_BYTES||segmented>MAX_SEGMENTED_BYTES||owner>MAX_ROW_BYTES) {fallback(id,'source_budget');continue;} + if(raw>remaining) {fallback(id,'request_budget');continue;} + remaining-=raw; admitted.push(id); + } + for(let offset=0;offset{ + const n=sources.get(id)!; + return db.prepare(`SELECT d.vector_id,d.repo,d.type,d.doc_path,d.source_version,d.indexed_at,d.source_revision, + s.source_version AS owner_version,s.prefix_indexed_at,s.prefix_revision,s.index_version,s.raw_hash,s.body_hash,s.source_hash, + s.raw_bytes,s.segmented_bytes,s.row_bytes,s.raw,s.segmented FROM search_docs d JOIN doc_sparse_sources s ON s.vector_id=d.vector_id + WHERE d.vector_id=? AND ${fullOwner} AND s.raw_bytes=? AND s.segmented_bytes=? AND s.row_bytes=? + AND s.raw_hash=? AND s.source_version=? AND s.prefix_revision=? + AND length(CAST(s.raw AS BLOB))<=? AND length(CAST(s.segmented AS BLOB))<=?`).bind( + id,Number(n.raw_bytes),Number(n.segmented_bytes),Number(n.row_bytes),String(n.raw_hash),String(n.owner_version),Number(n.prefix_revision),Number(n.raw_bytes),Number(n.segmented_bytes)); + }); + const results=await db.batch(statements); + for(let i=0;i applyD1Migrations(env.DB_FTS, env.TEST_MIGRATIONS)); +async function seed(repo:string,id:string,body:string,full=true,path='docs/a.md',type:'doc'|'wiki_doc'='doc',extra:Partial={}) { + const version=type==='doc'?await gitBlobSha(body):await sha256Hex(body); + await upsertFtsRow(env.DB_FTS,{repo,vectorId:id,type,state:'active',labels:'',milestone:'',assignees:'',updatedAt:'2026-10-01',docPath:path, + content:prepareEmbeddingInput(path,body),sourceVersion:version,...extra}); + if(full) { + const p=(await env.DB_FTS.prepare('SELECT * FROM search_docs WHERE vector_id=?').bind(id).first())!; + await sparseSourceStatement(env.DB_FTS,await prepareSparseSource(id,type,path,body,version,p.indexed_at),p).run(); + } +} +async function client(ai: (...args:any[])=>Promise = vi.fn(async()=>({response:[{id:0,score:0.9},{id:1,score:0.5}]})),vectors:unknown[]=[]) { + const bindings={...env,DOC_SPARSE_CONTEXT_ENABLED:'false',AI:{run:ai},VECTORIZE:{query:vi.fn(async()=>({matches:vectors}))}} as unknown as Env; + const handler=createMcpHandler(()=>createRagMcpServer(bindings),{route:'/mcp',legacy:'reject'}); + const remote=createRemoteClient({workerUrl:'https://synthetic.example',clientVersion:'test',fetch:(async(input:any,init:any)=>{ + const r=input instanceof Request?input:new Request(input,init); const headers=new Headers(r.headers);headers.set('host',new URL(r.url).host); + return handler(new Request(r,{headers}),bindings,{props:{githubUserId:9281,accessToken:'synthetic-test-only'}} as any); + }) as typeof fetch}); + return {call:async(args:unknown)=>{const r=await remote.callTool('search',args) as any; + return {error:r.isError??false,data:r.isError?r.content[0].text:JSON.parse(r.content[0].text)};},reset:()=>remote.reset()}; +} + +describe('matched D1 retrieval and ownership',()=>{ + it('finds the tail with flag false, preserves prefix/default and returns exact indexed UTF16 slice',async()=>{ + const repo='matched/tail',body='intro '.repeat(1600)+'unique-tailneedle 😀 日本語説明'; + await seed(repo,'tail-full',body); + expect(await queryFts(env.DB_FTS,'tailneedle',5,{repo,type:'doc'})).toEqual([]); + const found=await queryMatchedFts(env.DB_FTS,'tailneedle',5,{repo,type:'doc'}); + expect(found.hits.map(h=>h.vectorId)).toEqual(['tail-full']);expect(found.coverage).toMatchObject({total:1,full:1,status:'complete'}); + const resolved=await resolveMatchedContexts(env.DB_FTS,['tail-full'],found.sources); + const context=resolved.get('tail-full')!.matched_context!; + expect(context.text).toContain('tailneedle');expect(context.text).toBe(('docs/a.md\n\n'+body).slice(context.start,context.end)); + expect(context.excerpt_hash).toBe(await sha256Hex(context.text));expect(context.end-context.start).toBeLessThanOrEqual(1400); + const c=await client(); + for(const extra of [{},{doc_context:'prefix'}]) expect((await c.call({query:'tailneedle',repo,type:'doc',fusion:'sparse_only',rerank:false,...extra})).data.count).toBe(0); + const r=(await c.call({query:'tailneedle',repo,type:'doc',fusion:'sparse_only',rerank:false,doc_context:'matched'})).data; + expect(r.results[0].matched_context).toEqual(context);expect(r.rerank_applied).toBe(false);expect(r.doc_context_coverage.full).toBe(1); + expect(r.results[0].content).toBeUndefined();await c.reset(); + }); + it('splits partial preparation, scopes metadata before LIMIT, and reports all scope rows before term matching',async()=>{ + const repo='matched/scope'; + await seed(repo,'scope-full','needle alpha',true,'docs/a.md','doc',{labels:' a, b ',assignees:' x, y ',milestone:'M'}); + await seed(repo,'scope-prefix','needle alpha',false,'docs/b.md','doc',{labels:'a,b',assignees:'y',milestone:'M'}); + await seed(repo,'scope-unmatched','irrelevant',false,'docs/c.md','doc',{labels:'a,b',assignees:'y',milestone:'M'}); + await seed(repo,'scope-excluded','needle alpha '.repeat(100),true,'other/a.md'); + const filter={repo,type:'doc' as const,pathPrefix:'docs/',labels:['a','b'],assignee:'y',milestone:'M',since:'2026-09',until:'2026-11'}; + const r=await queryMatchedFts(env.DB_FTS,'needle alpha',2,filter); + expect(r.hits.map(h=>h.vectorId)).toEqual(['scope-full','scope-prefix']); + expect(r.coverage).toEqual({status:'partial',total:3,full:1,prefix_only:2,unprepared:2,out_of_range:0,stale:0}); + expect(await queryMatchedFts(env.DB_FTS,'needle',1,{...filter,state:'closed'})).toMatchObject({hits:[],coverage:{total:0,full:0,status:'complete'}}); + expect(await queryMatchedFts(env.DB_FTS,'unmatchedqueryword',1,filter)).toMatchObject({hits:[],coverage:{total:3,full:1,status:'partial'}}); + }); + it('ranks strict before relaxed across both corpora, with stable ties and one sparse vote',async()=>{ + const repo='matched/tiers'; + await seed(repo,'tiers-z','needle alpha',true);await seed(repo,'tiers-a','needle alpha',false); + await seed(repo,'tiers-relaxed','needle',true); + expect((await queryMatchedFts(env.DB_FTS,'needle alpha',5,{repo,type:'doc'})).hits.map(h=>h.vectorId)).toEqual(['tiers-a','tiers-z','tiers-relaxed']); + expect(mergeSparseTiers([{tier:0,rows:[{vector_id:'x',score:-100}]},{tier:0,rows:[{vector_id:'x',score:-1}]},{tier:1,rows:[{vector_id:'a'}]}],50).map(r=>r.vector_id)).toEqual(['x','a']); + }); + it('retains legacy trigram prefix evidence for both prepared and unprepared doc owners',async()=>{ + const repo='matched/legacy-code'; + await seed(repo,'legacy-code-full','DOC_SPARSE_ADMIN_TOKEN',true,'docs/a.md'); + await seed(repo,'legacy-code-prefix','DOC_SPARSE_ADMIN_TOKEN',false,'docs/b.md'); + await seed(repo,'legacy-code-excluded','DOC_SPARSE_ADMIN_TOKEN '.repeat(100),false,'other/c.md'); + for(const id of ['legacy-code-full','legacy-code-prefix','legacy-code-excluded']) + await env.DB_FTS.prepare("UPDATE search_docs SET tokenizer_kind='code' WHERE vector_id=?").bind(id).run(); + const filter={repo,type:'doc' as const,pathPrefix:'docs/'}; + const legacy=await queryFts(env.DB_FTS,'SPARSE_ADM',5,filter); + expect(new Set(legacy.map(h=>h.vectorId))).toEqual(new Set(['legacy-code-full','legacy-code-prefix'])); + const matched=await queryMatchedFts(env.DB_FTS,'SPARSE_ADM',5,filter); + expect(new Set(matched.hits.map(h=>h.vectorId))).toEqual(new Set(legacy.map(h=>h.vectorId))); + expect(matched.coverage).toMatchObject({full:1,prefix_only:1,total:2}); + expect((await queryMatchedFts(env.DB_FTS,'SPARSE_ADM',1,filter)).hits.map(h=>h.vectorId)).toEqual(['legacy-code-full']); + const contexts=await resolveMatchedContexts(env.DB_FTS,matched.hits.map(h=>h.vectorId),matched.sources); + for(const id of ['legacy-code-full','legacy-code-prefix']) expect(contexts.get(id)).toEqual({matched_context_fallback:'code_only_prefix'}); + const both=await queryMatchedFts(env.DB_FTS,'SPARSE',5,filter); + expect(both.hits).toHaveLength(2);expect(both.hits.filter(h=>h.vectorId==='legacy-code-full')).toHaveLength(1); + expect((await resolveMatchedContexts(env.DB_FTS,['legacy-code-full'],both.sources)).get('legacy-code-full')?.matched_context).toBeDefined(); + }); + it('keeps every prefix owner if full query/schema fails and exposes unavailable',async()=>{ + const repo='matched/unavailable';await seed(repo,'unavailable-full','fallbackneedle',true);await seed(repo,'unavailable-prefix','fallbackneedle',false); + const broken={prepare:(sql:string)=>{if(sql.includes('doc_sparse_fts_v1'))throw new Error('synthetic full failure');return env.DB_FTS.prepare(sql);},batch:env.DB_FTS.batch.bind(env.DB_FTS)} as D1Database; + const r=await queryMatchedFts(broken,'fallbackneedle',5,{repo,type:'doc'}); + expect(new Set(r.hits.map(h=>h.vectorId))).toEqual(new Set(['unavailable-full','unavailable-prefix']));expect(r.coverage.status).toBe('unavailable'); + await applyD1Migrations(env.DB_FTS_DOC_LEGACY,env.TEST_MIGRATIONS.filter(m=>!m.name.startsWith('0009'))); + await upsertFtsRow(env.DB_FTS_DOC_LEGACY,{vectorId:'matched-legacy',repo,type:'doc',state:'active',labels:'',milestone:'',assignees:'',updatedAt:'2026-10-01',docPath:'a.md',content:'fallbackneedle'}); + expect(await queryMatchedFts(env.DB_FTS_DOC_LEGACY,'fallbackneedle',5,{repo,type:'doc'})).toMatchObject({hits:[{vectorId:'matched-legacy'}],coverage:{status:'unavailable'}}); + }); + it('rejects update/delete races, unknown position, stale owners and raw read failure without losing candidates',async()=>{ + const repo='matched/races';await seed(repo,'race-update','raceneedle',true);await seed(repo,'race-delete','raceneedle',true);await seed(repo,'race-stale','raceneedle',true); + const r=await queryMatchedFts(env.DB_FTS,'raceneedle',5,{repo,type:'doc'}); + await env.DB_FTS.prepare('UPDATE search_docs SET content=? WHERE vector_id=?').bind('new','race-update').run(); + await env.DB_FTS.prepare('DELETE FROM search_docs WHERE vector_id=?').bind('race-delete').run(); + await env.DB_FTS.prepare('UPDATE doc_sparse_sources SET prefix_revision=prefix_revision+1 WHERE vector_id=?').bind('race-stale').run(); + const out=await resolveMatchedContexts(env.DB_FTS,['race-update','race-delete'],r.sources); + expect(out.get('race-update')?.matched_context_fallback).toBe('source_changed_or_missing');expect(out.get('race-delete')?.matched_context_fallback).toBe('source_changed_or_missing'); + const fresh=await queryMatchedFts(env.DB_FTS,'raceneedle',5,{repo,type:'doc'}); + expect(fresh.coverage.stale).toBe(1);expect(fresh.hits.map(h=>h.vectorId)).toContain('race-stale'); + expect((await resolveMatchedContexts(env.DB_FTS,['race-stale'],fresh.sources)).get('race-stale')?.matched_context_fallback).toBe('stale_source'); + const fail={prepare:env.DB_FTS.prepare.bind(env.DB_FTS),batch:async()=>{throw new Error('synthetic read failure');}} as unknown as D1Database; + expect((await resolveMatchedContexts(fail,['race-stale'],r.sources)).get('race-stale')?.matched_context_fallback).toBe('source_read_failed'); + }); + it('validates Unicode, duplicate fragments, segmentation, path/hash/version and span guards',async()=>{ + const raw='p.md\n\n'+'😀 日本語 '.repeat(400)+'終端独自文'; const stored=segmentForFts(raw); + const b=excerptBounds(raw,stored,segmentForFts('終端独自文')); + expect(typeof b).toBe('object');if(typeof b==='object') expect(raw.slice(b.start,b.end)).toContain('終端独自文'); + expect(excerptBounds(raw,stored,'😀')).toBe('ambiguous_fragment');expect(excerptBounds(raw,stored+'x','x')).toBe('segmentation_mismatch'); + expect(excerptBounds('😀','😀','\uDE00')).toBe('invalid_span');expect(excerptBounds('xx','xx','\u0001x\u0001x')).toBe('ambiguous_fragment'); + await seed('matched/guards','guards','guardneedle',true); + const search=await queryMatchedFts(env.DB_FTS,'guardneedle',5,{repo:'matched/guards',type:'doc'}), narrow=search.sources.get('guards')!; + const owner=(await env.DB_FTS.prepare('SELECT raw,segmented FROM doc_sparse_sources WHERE vector_id=?').bind('guards').first>())!; + const row={...narrow,...owner}; + expect(typeof await verifyExcerpt(narrow,row)).toBe('object'); + expect(await verifyExcerpt(narrow,{...row,source_revision:999})).toBe('version_mismatch'); + expect(await verifyExcerpt(narrow,{...row,raw:String(row.raw).replace('guardneedle','wrongneedle')})).toBe('hash_mismatch'); + const bad={...row,body_hash:'wrong'};expect(await verifyExcerpt(bad,bad)).toBe('hash_mismatch'); + }); + it('skips large candidates, respects cumulative raw budget, batches five and caps selected raw reads at 50',async()=>{ + const repo='matched/budget';await seed(repo,'budget-source','budgetneedle',true); + const r=await queryMatchedFts(env.DB_FTS,'budgetneedle',5,{repo,type:'doc'}), n=r.sources.get('budget-source')!; + const sources=new Map>(), ids=Array.from({length:60},(_,i)=>`budget-${i}`); + for(const id of ids)sources.set(id,{...n,vector_id:id});sources.set(ids[0],{...n,vector_id:ids[0],raw_bytes:262145}); + const batches:number[]=[]; + const db={prepare:env.DB_FTS.prepare.bind(env.DB_FTS),batch:async(statements:D1PreparedStatement[])=>{batches.push(statements.length);return statements.map(()=>({results:[]}));}} as unknown as D1Database; + const out=await resolveMatchedContexts(db,ids,sources);expect(out.size).toBe(50);expect(out.get(ids[0])?.matched_context_fallback).toBe('source_budget'); + expect(batches.reduce((a,b)=>a+b,0)).toBe(49);expect(Math.max(...batches)).toBe(5); + for(const id of ids)sources.set(id,{...n,vector_id:id,raw_bytes:262144}); + batches.length=0;const limited=await resolveMatchedContexts(db,ids,sources); + expect(batches).toEqual([4]);expect(limited.get(ids[4])?.matched_context_fallback).toBe('request_budget'); + }); + it('supports wiki, dense-only fallback and exact model text, and rejects incompatible modes',async()=>{ + await seed('matched/wiki','wiki-full','head '.repeat(1800)+'wikitailneedle',true,'Home','wiki_doc'); + await seed('matched/wiki','wiki-prefix','wikitailneedle',false,'Other','wiki_doc'); + const ai=vi.fn(async()=>({response:[{id:0,score:0.9},{id:1,score:0.8}]})), c=await client(ai); + const r=(await c.call({query:'wikitailneedle',repo:'matched/wiki',type:'wiki_doc',fusion:'sparse_only',doc_context:'matched'})).data; + const model=ai.mock.calls[0] as unknown as [string,{contexts:{text:string}[]}]; + expect(model[1].contexts.some(v=>v.text===r.results.find((x:any)=>x.vector_id==='wiki-full').matched_context.text)).toBe(true); + expect(r.results.find((x:any)=>x.vector_id==='wiki-prefix').matched_context_fallback).toBe('unprepared'); + for(const args of [{query:'x',type:'all'},{type:'doc'},{query:'x',type:'doc',vector_ids:[]},{query:'x',type:'doc',vector_ids:['wiki-full']}]) expect((await c.call({...args,doc_context:'matched'})).error).toBe(true); + await c.reset(); + const denseAi=vi.fn(async()=>({data:[[0.1]]})); + const dense=await client(denseAi,[{id:'wiki-full',score:0.9,metadata:{repo:'matched/wiki',type:'wiki_doc',state:'active',wiki_path:'Home',updated_at:'2026-10-01'}}]); + const d=(await dense.call({query:'wikitailneedle',repo:'matched/wiki',type:'wiki_doc',fusion:'dense_only',rerank:false,doc_context:'matched'})).data; + expect(d.results[0].matched_context_fallback).toBe('no_full_sparse_position');expect(d.doc_context_coverage.full).toBe(1);await dense.reset(); + }); + it('allocates raw by sparse keyword rank and vector ties after the fusion candidate cap',async()=>{ + const sources=new Map>(); + for(const [id,rank] of [['dense-first',5],['tie-z',2],['tie-a',2],['sparse-first',1],['third',3]] as const) + sources.set(id,{keyword_rank:rank,raw_bytes:262144,segmented_bytes:262144,row_bytes:524544, + raw_hash:'synthetic',owner_version:'synthetic',prefix_revision:0}); + const readIds:string[]=[]; + const db={prepare:()=>({bind:(...params:unknown[])=>{readIds.push(String(params[0]));return {};}}), + batch:async(statements:unknown[])=>statements.map(()=>({results:[]}))} as unknown as D1Database; + const out=await resolveMatchedContexts(db,['dense-first','tie-z','third','tie-a','sparse-first'],sources); + expect(readIds).toEqual(['sparse-first','tie-a','tie-z','third']); + expect(out.get('dense-first')?.matched_context_fallback).toBe('request_budget'); + }); +}); diff --git a/src/mcp-stateless-contract.test.ts b/src/mcp-stateless-contract.test.ts index dba4722..de0a7aa 100644 --- a/src/mcp-stateless-contract.test.ts +++ b/src/mcp-stateless-contract.test.ts @@ -124,6 +124,7 @@ describe("worker <-> bridge stateless contract", () => { expect(Object.keys(props).sort()).toEqual( [ "assignee", + "doc_context", "fusion", "graph_expand", "graph_hops", diff --git a/src/mcp.ts b/src/mcp.ts index 53d83a6..7fa6de6 100644 --- a/src/mcp.ts +++ b/src/mcp.ts @@ -61,6 +61,7 @@ import { } from "./fetch.js"; import { entityKey, groupByEntity, markRecency } from "./aggregate.js"; import { pathPrefixRange, validateDocPathPrefix } from "./path-prefix.js"; +import { queryMatchedFts, resolveMatchedContexts, unavailableCoverage, type MatchedSearch, type ContextOutcome } from './matched-context.js'; const GITHUB_API = "https://api.github.com"; const USER_AGENT = "github-rag-mcp/0.1.0"; @@ -367,6 +368,7 @@ export function createRagMcpServer(env: Env): McpServer { top_k, fusion, rerank, + doc_context, sort, since, until, @@ -376,8 +378,12 @@ export function createRagMcpServer(env: Env): McpServer { graph_hops, use_memory, }) => { - const memoryRequest = { query, repo, path_prefix, state, labels, milestone, assignee, type, top_k, fusion, rerank, sort, since, until, include_content, vector_ids, graph_expand, graph_hops, use_memory }; + const memoryRequest = { query, repo, path_prefix, state, labels, milestone, assignee, type, top_k, fusion, rerank, ...(doc_context === 'matched' ? { doc_context } : {}), sort, since, until, include_content, vector_ids, graph_expand, graph_hops, use_memory }; const remember = (payload: unknown) => rememberResult(env, payload, memoryRequest); + const matched = doc_context === 'matched'; + if (matched && (!(query ?? '').trim() || vector_ids !== undefined || (type !== 'doc' && type !== 'wiki_doc'))) { + return { content: [{ type: 'text' as const, text: 'doc_context=matched requires a nonempty query, type=doc or wiki_doc, and no vector_ids' }], isError: true }; + } // ── Fetch mode (vector_ids): stored content for named rows ─── // First branch on purpose: a fetch call carries no query, so leaving it @@ -540,8 +546,9 @@ export function createRagMcpServer(env: Env): McpServer { // ── Sparse path: D1 FTS5 BM25 query ────────────────────── let retrievalIncomplete = false; + let matchedSearch: MatchedSearch | undefined; const sparsePromise: Promise = - fusionMode === "dense_only" + fusionMode === "dense_only" && !matched ? Promise.resolve([]) : (async () => { const ftsFilter: FtsFilter = {}; @@ -554,7 +561,12 @@ export function createRagMcpServer(env: Env): McpServer { } if (milestone) ftsFilter.milestone = milestone; if (path_prefix) ftsFilter.pathPrefix = path_prefix; + if (matched) Object.assign(ftsFilter, { labels, assignee, since, until }); try { + if (matched) { + matchedSearch = await queryMatchedFts(env.DB_FTS, trimmedQuery, internalTopK, ftsFilter, fusionMode === 'dense_only'); + return matchedSearch.hits; + } return await queryFts(env.DB_FTS, trimmedQuery, internalTopK, ftsFilter); } catch (err) { retrievalIncomplete = true; @@ -731,6 +743,24 @@ export function createRagMcpServer(env: Env): McpServer { // shape it returns null, in which case we keep the post-filter order. const rerankScores = new Map(); let rerankApplied = false; + const matchedBase = new Map(); + let matchedOutcomes = new Map(); + if (matched) { + filtered = [...filtered].sort((a,b)=>b.fusedScore-a.fusedScore || (a.vectorIdb.vectorId?1:0)); + const missing = filtered.filter(f => !(payload.get(f.vectorId)?.ftsRow?.content ?? '').trim()).map(f => f.vectorId); + try { + for (const [id, row] of await getDocsByVectorIds(env.DB_FTS, missing)) matchedBase.set(id, String(row.content ?? '')); + } catch { /* Preserve candidates when base content is unavailable. */ } + for (const f of filtered) { + const content=payload.get(f.vectorId)?.ftsRow?.content ?? ''; + if(content.trim()) matchedBase.set(f.vectorId,content); + } + const eligible=filtered.filter(f => (matchedBase.get(f.vectorId) ?? '').trim()).slice(0,RERANK_MAX_CANDIDATES); + matchedOutcomes=await resolveMatchedContexts(env.DB_FTS, eligible.map(f=>f.vectorId), matchedSearch?.sources ?? new Map(), + !matchedSearch || matchedSearch.coverage.status === 'unavailable'); + for(const f of filtered) if(!matchedOutcomes.has(f.vectorId)) matchedOutcomes.set(f.vectorId, + {matched_context_fallback:(matchedBase.get(f.vectorId) ?? '').trim() ? 'candidate_limit' : 'missing_base_content'}); + } if ( rerankEnabled && effectiveSort === "relevance" && @@ -743,7 +773,7 @@ export function createRagMcpServer(env: Env): McpServer { // the filter `rerankCandidates` applies, so a whitespace-only FTS row // is treated as missing here rather than silently dropped there. const missingContentIds = filtered - .filter((f) => (payload.get(f.vectorId)?.ftsRow?.content ?? "").trim() === "") + .filter((f) => !matched && (payload.get(f.vectorId)?.ftsRow?.content ?? "").trim() === "") .map((f) => f.vectorId); let backfilled: Map = new Map(); if (missingContentIds.length > 0) { @@ -765,7 +795,7 @@ export function createRagMcpServer(env: Env): McpServer { // dense-only hits (and for whitespace-only sparse rows). const content = sparseContent.trim() !== "" ? sparseContent : backfilled.get(f.vectorId) ?? ""; - return { id: f.vectorId, content }; + return { id: f.vectorId, content: matched ? matchedOutcomes.get(f.vectorId)?.matched_context?.text ?? matchedBase.get(f.vectorId) ?? '' : content }; }); const reranked = await rerankCandidates( @@ -988,6 +1018,7 @@ export function createRagMcpServer(env: Env): McpServer { // otherwise. `rerank_applied: true` guarantees at least one row in // the response carries a non-null score. rerank_score: rerankScores.get(f.vectorId) ?? null, + ...(matched ? matchedOutcomes.get(f.vectorId) : {}), url: buildResultUrl(r), updated_at: r.updatedAt, repo: r.repo, @@ -1167,6 +1198,7 @@ export function createRagMcpServer(env: Env): McpServer { sort: effectiveSort, dense_candidates: denseResult.hits.length, sparse_candidates: sparseHits.length, + ...(matched ? { doc_context_coverage: matchedSearch?.coverage ?? unavailableCoverage() } : {}), // Filters that matched no row in the index at all (issue #219). // Always present; `[]` means every applied filter matched // something, so `count: 0` is a genuine zero-hit result. A diff --git a/src/search-contract.ts b/src/search-contract.ts index 8911ec0..3bacb93 100644 --- a/src/search-contract.ts +++ b/src/search-contract.ts @@ -1,6 +1,11 @@ import { z } from "zod"; export const INCLUDE_CONTENT_MAX_DOCS = 5; export const searchInputSchema = z.object({ + doc_context: z.enum(['prefix', 'matched']).optional().default('prefix').describe( + 'Document context. Default prefix preserves existing search. matched requires a nonempty query and type=doc or wiki_doc; scan and vector_ids are rejected. ' + + 'Searches prepared indexed full sources and returns a verified bounded matched_context separately from live include_content, even with rerank=false. ' + + 'doc_context_coverage reports the filtered indexed population before term matching; total=0 means an empty metadata scope, distinct from no term matches. Coverage is read before the candidate batch and may differ under concurrent updates. ' + + 'Unprepared tails cannot be searched. Legacy code-only prefix evidence is retained with code_only_prefix fallback when no full natural-language position exists. Failures retain prefix candidates with explicit fallback reasons.'), query: z .string().max(4096) .optional() diff --git a/src/segment.ts b/src/segment.ts index 839e323..99df170 100644 --- a/src/segment.ts +++ b/src/segment.ts @@ -44,7 +44,7 @@ const UNSPACED_SCRIPT_RE = */ let segmenter: Intl.Segmenter | undefined; -function getSegmenter(): Intl.Segmenter { +export function getSegmenter(): Intl.Segmenter { if (segmenter === undefined) { segmenter = new Intl.Segmenter("ja", { granularity: "word" }); }