Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -302,4 +302,4 @@ wrangler.toml

Doc/wiki full sparse source preparation is a default-off operator foundation, independent of normal search. See the [storage contract and bounded plan/apply procedure](docs/6-full-sparse-sources.ja.md). Migration 0009 precedes operator preparation and enable; default-off ingest/search remains usable without it.

The two new sparse operator endpoints require the dedicated `DOC_SPARSE_ADMIN_TOKEN` secret and `Authorization: Bearer <token>`; they fail closed without it and do not accept GitHub credentials. Provision via `wrangler secret put DOC_SPARSE_ADMIN_TOKEN` after reviewed merge.
The two sparse operator endpoints require the existing Worker `GITHUB_TOKEN` secret in the `GITHUB_TOKEN` header (exact match). Unset or empty secrets return 401; dedicated tokens and Authorization bearer credentials are rejected. No additional secret provisioning is required.
2 changes: 1 addition & 1 deletion docs/0-requirements.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -689,4 +689,4 @@ source の `docs/_Sidebar.md` は既存 Feedback Memory EN/JA ページを参照

#284 は既定 off の保存・operator 基盤を追加する。[全文 sparse source 契約](6-full-sparse-sources.ja.md)に従い prefix corpus、既定検索、graph、vector 数を維持する。matched 検索は #281 の別作業。

新しい sparse operator の 2 経路は専用 `DOC_SPARSE_ADMIN_TOKEN` bearer secret で認証し、未設定・空は fail closed。GitHub source 取得は従来の `GITHUB_TOKEN` を使う。既定 off の更新を含め、prefix revision CAS により版修復と owner 保存を同一 D1 batch に結ぶ。
sparse operator の 2 経路は既存 Worker secret と `GITHUB_TOKEN` ヘッダの完全一致で認証し、secret が未設定・空なら401。専用 token と Authorization bearer は受け入れない。GitHub source 取得は従来の `GITHUB_TOKEN` を使う。既定 off の更新を含め、prefix revision CAS により版修復と owner 保存を同一 D1 batch に結ぶ。
2 changes: 1 addition & 1 deletion docs/0-requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -694,4 +694,4 @@ The source `docs/_Sidebar.md` must link the existing Feedback Memory EN/JA pages

Issue #284 adds a default-off storage/operator foundation; its contract is [Full sparse sources](6-full-sparse-sources.ja.md). It preserves the prefix corpus, default retrieval, graph and vector count. Matched retrieval remains #281.

The new sparse operator endpoints require the dedicated `DOC_SPARSE_ADMIN_TOKEN` bearer secret and fail closed without it; GitHub source fetching retains `GITHUB_TOKEN`. Version repair and owner writes share a D1 batch guarded by the prefix revision, including default-off updates.
The two sparse operator endpoints require an exact match between the `GITHUB_TOKEN` header and the existing Worker `GITHUB_TOKEN` secret, and return 401 when the secret is unset or empty. Dedicated tokens and Authorization bearer credentials are not accepted; GitHub source fetching retains `GITHUB_TOKEN`. Version repair and owner writes share a D1 batch guarded by the prefix revision, including default-off updates.
2 changes: 1 addition & 1 deletion docs/6-full-sparse-sources.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ migration の source_revision は prefix の source に関わる UPDATE ごと

## Operator

この 2 経路だけは専用 secret `DOC_SPARSE_ADMIN_TOKEN` を使い、`Authorization: Bearer <token>` が完全一致した時だけ認証する。未設定・空なら fail closed。`GITHUB_TOKEN` への fallback はなく、GitHub source の取得だけは従来の `GITHUB_TOKEN` を使う。既存の他 admin 経路は変更しない。reviewed merge 後、operator が `wrangler secret put DOC_SPARSE_ADMIN_TOKEN` で secret を provision する。値は [vars]・repository・chat に置かない。`POST /admin/backfill-doc-sparse?repo=owner/repo&type=doc|wiki_doc` は plan が既定。書込みは `apply=true` のみ。`limit` は既定 5、最大 10、`max_bytes` は既定/最大 1 MiB、`cursor` は search_docs rowid。1 回の source 取得も bounded reader で制限する。doc は DO の blobSHA の Git Blob API から取得・hash 検証し、wiki は DO の extension の raw 本文を取得して bodySHA を照合する。過去 wiki を hash から取得できるとは仮定しない。store と prefix の版が異なるものや検証できないものは conflict、再取得候補として返す。embedding/Vectorize を呼ばず、unchanged/ETag を回避して準備する。
この 2 経路は既存 Worker secret と `GITHUB_TOKEN` ヘッダの完全一致で認証する。secret が未設定・空なら401。専用 token と Authorization bearer は受け入れない。GitHub source の取得も従来の `GITHUB_TOKEN` を使う。既存の他 admin 経路は変更しない。追加の secret provision は不要。値は [vars]・repository・chat に置かない。`POST /admin/backfill-doc-sparse?repo=owner/repo&type=doc|wiki_doc` は plan が既定。書込みは `apply=true` のみ。`limit` は既定 5、最大 10、`max_bytes` は既定/最大 1 MiB、`cursor` は search_docs rowid。1 回の source 取得も bounded reader で制限する。doc は DO の blobSHA の Git Blob API から取得・hash 検証し、wiki は DO の extension の raw 本文を取得して bodySHA を照合する。過去 wiki を hash から取得できるとは仮定しない。store と prefix の版が異なるものや検証できないものは conflict、再取得候補として返す。embedding/Vectorize を呼ばず、unchanged/ETag を回避して準備する。

応答は対象 repo/type、取得時刻、各 source の版、coverage/bytes、成功・範囲外・欠損・競合・retryable error、次 cursor を持つ。retryable error または取得前に残り byte 予算が無い場合、cursor はその source を越えない。残り予算だけで取得が打ち切られた source は `byte_budget_deferred` として cursor を保持し、次の call で再試行する。call の初期予算でも取得上限で原文/hash を確認できなかった source は `source_byte_limit_unverified` conflict として進め、残りの取得を止める。`source_bytes` は取得・decode が完了した body の UTF-8 bytes、`budget_exhausted` は未取得分の上限到達も区別する。Git Blob JSON の network reader は残り decoded 予算 × 1.4 + 4096 bytes(base64 と envelope)で制限する。missing/conflict は明示報告して次へ進むため、競合解消後は cursor を戻して再実行する。plan は source を取得・検証するが保存しない。apply は同じ source の再実行で owner/FTS/vector を増やさない。

Expand Down
2 changes: 1 addition & 1 deletion docs/installation.ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -512,4 +512,4 @@ GitHub App の callback URL が worker 側 URL と完全一致していない。

operator 経路と enable より先に migration 0009 を適用し(既定 off の ingest・検索は新 schema 不要)、準備中は `DOC_SPARSE_CONTEXT_ENABLED="false"` を保つ。認証済み `POST /admin/backfill-doc-sparse?repo=owner/repo&type=doc` は plan が既定。計画と Time Travel bookmark と復旧の影響対象を確認した後(2026-10-13 の full export は FTS/virtual table 未対応で失敗、restore 未実行)だけ `apply=true` を指定する。1 回は最大 10 資料・decoded source 1 MiB、返る `next_cursor` を `cursor` に渡し、競合・error を確認する。wiki は `type=wiki_doc` で別処理。`POST /admin/doc-sparse-stats?repo=owner/repo&type=doc` で索引済み prefix owner の coverage を観察する。[完全な契約](6-full-sparse-sources.ja.md)に予算と rollout 境界を固定している。embedding と検索応答は変わらない。

reviewed merge 後、この 2 経路の呼出しより先に `wrangler secret put DOC_SPARSE_ADMIN_TOKEN` で専用 secret を provision し、`Authorization: Bearer <token>` を送る。未設定・空 secret、`GITHUB_TOKEN` ヘッダ、GitHub bearer credential は認証に使わない。source 取得だけは従来の `GITHUB_TOKEN` を使い、既存 admin 経路は変えない。secret の値を設定ファイルや chat に置かない。
この 2 経路には既存 Worker secret と同じ値を `GITHUB_TOKEN` ヘッダで送る。完全一致のみを受け入れ、secret が未設定・空なら401。専用 token と Authorization bearer は認証に使わない。追加の secret provision は不要。source 取得は従来の `GITHUB_TOKEN` を使い、既存 admin 経路は変えない。secret の値を設定ファイルや chat に置かない。
2 changes: 1 addition & 1 deletion docs/installation.md
Original file line number Diff line number Diff line change
Expand Up @@ -512,4 +512,4 @@ The GitHub App callback URL does not exactly match the worker callback URL.

Migration 0009 must precede these operator endpoints and enable. Default-off ingest and search need no new schema. Keep `DOC_SPARSE_CONTEXT_ENABLED="false"` while preparing. The authenticated `POST /admin/backfill-doc-sparse?repo=owner/repo&type=doc` endpoint defaults to plan; use `apply=true` only after reading the plan and confirming the Time Travel bookmark/recovery scope (full export failed on FTS/virtual tables in the 2026-10-13 operator check; no restore was executed). Process at most 10 sources and 1 MiB decoded source bytes per call, pass the returned `next_cursor` as `cursor`, and examine conflicts/errors before continuing. Use `type=wiki_doc` separately. `POST /admin/doc-sparse-stats?repo=owner/repo&type=doc` reports coverage of indexed prefix owners. See [the complete contract](6-full-sparse-sources.ja.md) for limits and rollout boundaries. Neither operation embeds or changes retrieval.

Provision `DOC_SPARSE_ADMIN_TOKEN` with `wrangler secret put DOC_SPARSE_ADMIN_TOKEN` after reviewed merge and before these operator calls. Send `Authorization: Bearer <token>` to both new endpoints. They reject unset/empty secrets, a `GITHUB_TOKEN` header, and GitHub bearer credentials. GitHub source acquisition still uses `GITHUB_TOKEN`; existing older admin authentication is unchanged. Keep secret values out of configuration files and chat.
Send the existing Worker secret value in the `GITHUB_TOKEN` header to both sparse operator endpoints. Authentication requires an exact match and returns 401 for an unset or empty secret. Dedicated tokens and Authorization bearer credentials are rejected. No additional secret provisioning is required. GitHub source acquisition still uses `GITHUB_TOKEN`; existing older admin authentication is unchanged. Keep secret values out of configuration files and chat.
4 changes: 2 additions & 2 deletions src/backfill-doc-sparse.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ import {

export const MAX_BACKFILL_BYTES = 1024 * 1024;
export async function handleDocSparseAdmin(request: Request, env: Env): Promise<Response> {
if (!env.DOC_SPARSE_ADMIN_TOKEN?.trim()
|| request.headers.get('Authorization') !== `Bearer ${env.DOC_SPARSE_ADMIN_TOKEN}`) {
if (!env.GITHUB_TOKEN?.trim()
|| request.headers.get('GITHUB_TOKEN') !== env.GITHUB_TOKEN) {
return new Response('Unauthorized', { status: 401 });
}
const url = new URL(request.url);
Expand Down
45 changes: 25 additions & 20 deletions src/doc-sparse.workers.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -287,38 +287,43 @@ describe('bounded operator backfill', () => {

it('requires operator authentication and rejects invalid/broad scopes', async () => {
const live = { ...runtime(), GITHUB_TOKEN: 'test-github-fetch', DOC_SPARSE_ADMIN_TOKEN: 'test-sparse-operator' } as Env;
const endpoint = 'http://test/admin/backfill-doc-sparse?repo=sparse/auth&type=doc';
expect((await handleDocSparseAdmin(new Request(endpoint, { method: 'POST' }), live)).status).toBe(401);
const invalidHeaders: Record<string, string>[] = [
{ GITHUB_TOKEN: 'test-github-fetch' },
{ Authorization: 'Bearer test-github-fetch' },
{ Authorization: 'Bearer wrong' },
{ Authorization: 'test-sparse-operator' },
{ Authorization: 'bearer test-sparse-operator' },
];
for (const headers of invalidHeaders) {
expect((await handleDocSparseAdmin(new Request(endpoint, { method: 'POST', headers }), live)).status).toBe(401);
}
for (const token of [undefined, '', ' ']) {
expect((await handleDocSparseAdmin(new Request(endpoint, { method: 'POST', headers: { Authorization: 'Bearer test-sparse-operator' } }),
{ ...live, DOC_SPARSE_ADMIN_TOKEN: token })).status).toBe(401);
const endpoints = ['backfill-doc-sparse', 'doc-sparse-stats'].map(path => `http://test/admin/${path}?repo=sparse/auth&type=doc`);
const endpoint = endpoints[0];
for (const endpoint of endpoints) {
expect((await handleDocSparseAdmin(new Request(endpoint, { method: 'POST' }), live)).status).toBe(401);
const invalidHeaders: Record<string, string>[] = [
{ GITHUB_TOKEN: 'wrong' },
{ Authorization: 'Bearer test-github-fetch' },
{ Authorization: 'Bearer wrong' },
{ Authorization: 'Bearer test-sparse-operator' },
{ GITHUB_TOKEN: 'test-sparse-operator' },
{ Authorization: 'test-sparse-operator' },
{ Authorization: 'bearer test-sparse-operator' },
];
for (const headers of invalidHeaders) {
expect((await handleDocSparseAdmin(new Request(endpoint, { method: 'POST', headers }), live)).status).toBe(401);
}
for (const token of [undefined, '', ' ']) {
expect((await handleDocSparseAdmin(new Request(endpoint, { method: 'POST', headers: { GITHUB_TOKEN: 'test-github-fetch' } }),
{ ...live, GITHUB_TOKEN: token } as unknown as Env)).status).toBe(401);
}
}
for (const suffix of ['&limit=11', '&max_bytes=1048577', '&cursor=-1', '&apply=yes']) {
expect((await handleDocSparseAdmin(new Request(endpoint + suffix, { method: 'POST', headers: { Authorization: 'Bearer test-sparse-operator' } }), live)).status).toBe(400);
expect((await handleDocSparseAdmin(new Request(endpoint + suffix, { method: 'POST', headers: { GITHUB_TOKEN: 'test-github-fetch' } }), live)).status).toBe(400);
}
const planned = await handleDocSparseAdmin(new Request(endpoint, { method: 'POST', headers: { Authorization: 'Bearer test-sparse-operator' } }), live);
const planned = await handleDocSparseAdmin(new Request(endpoint, { method: 'POST', headers: { GITHUB_TOKEN: 'test-github-fetch' } }), live);
expect(planned.status).toBe(200); expect(await planned.json()).toMatchObject({ mode: 'plan', scanned: 0 });
const stats = await handleDocSparseAdmin(new Request('http://test/admin/doc-sparse-stats?repo=sparse/auth&type=doc', {
method: 'POST', headers: { Authorization: 'Bearer test-sparse-operator' },
method: 'POST', headers: { GITHUB_TOKEN: 'test-github-fetch' },
}), live);
expect(stats.status).toBe(200);
expect((await handleDocSparseAdmin(new Request('http://test/admin/doc-sparse-stats?repo=sparse/auth&type=doc', {
method: 'POST', headers: { GITHUB_TOKEN: 'test-github-fetch' },
method: 'POST', headers: { Authorization: 'Bearer test-sparse-operator' },
}), live)).status).toBe(401);
const seeded = await seed('sparse/auth-fetch', 'sparse-auth-fetch', 'auth source');
mockBlob('auth source', seeded.record.blobSha);
const applied = await handleDocSparseAdmin(new Request('http://test/admin/backfill-doc-sparse?repo=sparse/auth-fetch&type=doc&apply=true', {
method: 'POST', headers: { Authorization: 'Bearer test-sparse-operator' },
method: 'POST', headers: { GITHUB_TOKEN: 'test-github-fetch' },
}), live);
expect(applied.status).toBe(200);
expect(await applied.json()).toMatchObject({ counts: { success: 1 } });
Expand Down
2 changes: 0 additions & 2 deletions src/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -228,8 +228,6 @@ export interface Env extends OAuthEnv {
DB_FTS: D1Database;
/** Full doc/wiki sparse ownership; enabled only by the literal "true", default off. */
DOC_SPARSE_CONTEXT_ENABLED?: string;
/** Dedicated operator bearer secret for full sparse plan/apply/statistics only. */
DOC_SPARSE_ADMIN_TOKEN?: string;
AI: Ai;
/** GitHub personal access token or installation token for API access (set via `wrangler secret put`) */
GITHUB_TOKEN: string;
Expand Down
5 changes: 1 addition & 4 deletions wrangler.toml
Original file line number Diff line number Diff line change
Expand Up @@ -64,10 +64,7 @@ crons = ["0 * * * *", "15 * * * *", "30 * * * *", "45 * * * *"]
# Required secrets (set via `wrangler secret put <NAME>`):
# GITHUB_CLIENT_ID — GitHub App OAuth client ID
# GITHUB_CLIENT_SECRET — GitHub App OAuth client secret
# GITHUB_TOKEN — GitHub token for API polling (PAT or installation token)
# DOC_SPARSE_ADMIN_TOKEN — Dedicated bearer secret for sparse plan/apply/stats only;
# provision after reviewed merge via wrangler secret put,
# never place its value in [vars]. Unset/empty fails closed.
# GITHUB_TOKEN — GitHub token for API polling and admin authentication (PAT or installation token)

# Environment variables
[vars]
Expand Down
Loading