diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index e26de7982..3ac963b86 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -2978,6 +2978,14 @@ Suggestion sidecars use `DataDirectorySecurity.ResolveSensitiveSidecarDirectoryF Before suggestion text is persisted, typed redaction replaces named credentials, AWS access keys, bearer values, known structured credential formats (including GitHub, Stripe, GitLab, Slack, and OpenAI prefixes), and opaque mixed-character tokens. The high-entropy fallback is identifier-aware: structured PascalCase and snake_case code/test identifiers, including embedded numeric components, leading-underscore identifiers, and slash/hyphen recipe IDs with multiple word boundaries remain available as reproducibility evidence. Single-block or alternating-case opaque tokens and known token formats remain redacted as `[REDACTED:high_entropy_token]`. +The same redactor preserves complete ordinary relative file references such as `artifacts/full-dogfood-20260912/FINDINGS.md#d01` in persistence, show, and JSON/Markdown/issue-draft export (#5345). This additional entropy exception validates spelling only; it never reads a referenced file or establishes that the file exists in the repository. It requires forward slashes, a filename with a 1–8 ASCII-letter extension, at most 260 path characters, at most 16 components, and at most 100 characters per component. Dot, hyphen, and underscore separate nonempty words within components. Words may be lowercase (up to 24 letters), uppercase (up to 12), or camel/Pascal words whose individual parts have 3–24 letters; a word can end with up to four digits, and a standalone numeric part can have up to eight digits for dates. Each component must also pass the sensitive-name check. The Markdown fragment remains separate and subject to normal secret detection. + +Known credential and assignment redaction runs before this exception. A preceding-context check of at most 128 characters also excludes values introduced by sensitive names such as case-insensitive `bearer` or `password:`, and declines the exception if that context budget is exhausted. Complete-token boundaries prevent accepting a suffix of an absolute/home path, URL, escaped path, or other token; empty/traversal components, backslashes, queries, controls within the path, and opaque components do not qualify. Paths outside this deliberately narrow subset retain the existing heuristic redaction behavior, as do the #4403/#4751 identifier controls. The 32,768-character input cap and regex-timeout fallback are unchanged. No historical redaction is reversed: an editable local draft can be repaired only by explicitly supplying the original text with `suggestions update --context "" --db `, preserving its ID and updating its revision. Submitted records retain the existing edit restrictions. + +Mixed-case word bodies of 16 or more letters additionally require a run of at least three lowercase letters, preserving the existing distinction between readable identifiers and repeated short alternating-case groups. + +Quoted credential names and quoted or collection-valued credentials are recognized before the path exception. Single, double, and backtick quotes, escaped characters, and directly concatenated value parts are consumed through a real value boundary. Existing credential markers remain stable under repeated redaction. Collection scanning allows at most 16 nested arrays/objects; malformed, unterminated, or deeper values conservatively consume the remainder of the already bounded field. YAML block, tag, and anchor/alias values after a credential name also consume that remainder; this recognizer does not parse YAML to recover following fields. A linear enclosing-token scan retains quoted or escaped whitespace and rejects URL/rooted-path prefixes before punctuation, with a 128-character prefix limit. Separate quoted JSON values use independent prefixes, while rooted prefixes cannot reset at a comma. Benign quoted evidence values still use the ordinary path checks. + ### Deduplication `SuggestionStore` first checks a normalized SHA256 content hash, then compares the candidate against the most recent suggestions in the same category and language using normalized-token Jaccard similarity. The default fuzzy threshold is `0.85`; `cdidx mcp --suggestion-dedup-threshold`, `CDIDX_SUGGESTION_DEDUP_THRESHOLD`, or `.cdidxrc.json` `suggestion_dedup_threshold` can override it with a value from `0` to `1`. Fuzzy matches are returned as duplicates before GitHub submission and log the matched immutable ID plus score to stderr for auditability. @@ -7432,6 +7440,14 @@ suggestion sidecar は `DataDirectorySecurity.ResolveSensitiveSidecarDirectoryFo 提案テキストを永続化する前に、型付き redaction が名前付き credential、AWS access key、bearer 値、GitHub、Stripe、GitLab、Slack、OpenAI の prefix を含む既知の構造化 credential 形式、および不透明な混合文字 token を置換します。high-entropy fallback は識別子を考慮し、複数の単語境界を持つ構造化された PascalCase / snake_case のコード・テスト識別子（埋め込み数値 component を含む）、先頭 underscore 付き識別子、slash / hyphen 形式の recipe ID は再現性の根拠として保持します。一方、単一ブロックまたは大文字・小文字が交互に並ぶ不透明 token と既知の token 形式は、引き続き `[REDACTED:high_entropy_token]` に置換します。 +同じ秘匿処理は、`artifacts/full-dogfood-20260912/FINDINGS.md#d01` のような通常の相対ファイル参照を、保存・詳細表示・JSON／Markdown／Issue draft のエクスポートで保持します（#5345）。追加したエントロピー判定の例外は文字列の構成だけを検証し、参照先を読み取ったり、リポジトリ内の実在を確認したりしません。区切りは `/`、ファイル名の拡張子は ASCII 英字1～8文字、パス全体は260文字以内、構成要素は16個以内、各構成要素は100文字以内とします。構成要素内では `.`・`-`・`_` で空でない単語を区切ります。単語は小文字24文字以内、大文字12文字以内、または各部分が3～24文字の camelCase／PascalCase とし、末尾の数字は4桁以内、日付などの数字だけの部分は8桁以内を許容します。各構成要素には機密名の検査も適用します。Markdown のフラグメントは別の文字列として通常の秘密情報検出を受けます。 + +既知の資格情報と代入値の秘匿処理を、この例外より先に実行します。直前の文脈も最大128文字まで検査し、大文字小文字を問わない `bearer` や `password:` などの機密名に続く値は例外対象外とします。この検査の上限に達した場合も例外を適用しません。文字列の境界を検証することで、絶対パス・ホーム相対パス・URL・エスケープを含むパス・別のトークンの末尾だけを受け入れることを防ぎます。空または親ディレクトリへの移動を表す構成要素、バックスラッシュ、クエリ、パス内の制御文字、不透明な構成要素は例外対象外です。この限定的な条件に合わないパスと #4403／#4751 の識別子には、既存の判定を維持します。入力32,768文字の上限と正規表現タイムアウト時の代替処理も変更しません。過去の伏字を推測で復元することはありません。編集可能なローカル draft は `suggestions update --context "<原文>" --db ` で原文を明示した場合だけ修復でき、ID を維持したまま revision を更新します。送信済みレコードの編集制限も維持します。 + +大文字と小文字が混在する単語の英字部分が16文字以上の場合は、小文字が3文字以上連続する箇所も必要とします。これにより、読みやすい識別子と短い大文字小文字の組が連続する文字列を区別する既存の判定を維持します。 + +引用符付きの機密名と、引用符・配列・オブジェクトに包まれた資格情報の値は、パスの例外より先に認識します。単一・二重引用符、バッククォート、エスケープ文字、直結した値の断片を、実際の値の区切りまで含めて秘匿します。既存の資格情報マーカーは繰り返し秘匿しても変化しません。配列・オブジェクトの入れ子は16段までとし、不正な区切り、閉じられていない値、上限を超える入れ子では、既に入力上限を適用したフィールドの残りを安全側に秘匿します。機密名に続く YAML のブロック・タグ・アンカー／エイリアス形式の値も、フィールドの残りを秘匿します。YAML を解析して後続フィールドを復元することはありません。周囲のトークンは文字列長に比例する処理で検査し、引用符内やエスケープされた空白を保持したうえで、括弧などの前に続く URL・絶対パスを除外します。接頭部分の上限は128文字です。別々の引用符付き JSON 値は接頭部分を独立して検証し、絶対パスの接頭部分はコンマでリセットしません。機密値ではない引用符付きの証拠参照には通常のパス検証を適用します。 + ### 重複排除とローカル保持 `SuggestionStore` はまず正規化した SHA256 content hash を確認し、その後、同じ category / language の直近提案と正規化 token の Jaccard 類似度で比較する。fuzzy しきい値の既定は `0.85` で、`cdidx mcp --suggestion-dedup-threshold`、`CDIDX_SUGGESTION_DEDUP_THRESHOLD`、または `.cdidxrc.json` の `suggestion_dedup_threshold` で `0` から `1` の値へ上書きできる。fuzzy match は GitHub 送信前に重複として返され、監査用に一致先の不変 ID と score を stderr に記録する。 diff --git a/TESTING_GUIDE.md b/TESTING_GUIDE.md index 88d6f77a5..252659dd8 100644 --- a/TESTING_GUIDE.md +++ b/TESTING_GUIDE.md @@ -979,6 +979,8 @@ Candidate-ordered parallel-index recovery tests must prove that the fatal result Workspace freshness and git metadata enrichment behavior. - `DiagnosticRedactorTests.cs`, `SuggestionStoreTests.cs` Local suggestion JSON storage: opaque/stable-ID allocation and legacy migration, all-editable-field revision conflicts, dedup hashing, submission-finalization races, persistence, corruption recovery, atomic writes. Keep suggestion-redaction cases table-driven with negative fixtures for structured PascalCase, snake_case, leading-underscore, and recipe identifiers plus positive fixtures for opaque mixed-character and known token formats; the persistence case should retain an identifier and redact a secret from the same context. + Evidence-link coverage (#5345) pairs the exact dated Markdown reference and representative source/document paths with credentials inside path components, sensitive names, traversal, rooted/URL/control inputs, and path/component/input bounds on net8/net9. The shared persistence fixture retains both the link and #4403/#4751 identifier while redacting its credential. Existing production-runtime add/update fixtures reopen the store through human/JSON show and JSON/Markdown/issue-draft export, verify that historical markers survive reads, and recover a stable-ID draft only from explicitly supplied original context. + Review regressions include quoted credential keys, long and concatenated quoted values, backticks, escapes, nested/malformed collections, YAML blocks/tags/anchors, repeated-redaction stability, multiple benign JSON evidence values, apostrophes in prose, and URL/absolute-path suffixes behind punctuation or quoted/escaped spaces. Keep complete-value redaction independent of the 128-character evidence-context budget; verify persistence and all full-text CLI exports with these controls. - `DataDirectorySecurityTests.cs`, `ProgramCliTests.cs` suggestion-sidecar coverage Shared-temp database routing, owner-only directory/file modes, colocated private-directory behavior, and structured CLI filesystem failures. Suggestion-history query coverage keeps NFKC/case-insensitive matching across every documented field, structured filters before deterministic pagination, aggregate omission reasons, compact redaction, progressing continuations, and whole-document UTF-8 byte budgets in the same production-runtime fixture. @@ -2312,6 +2314,8 @@ dotnet test --filter "FullyQualifiedName~GitHelperTests" ワークスペース鮮度と git メタデータ付与のテスト。 - `DiagnosticRedactorTests.cs`、`SuggestionStoreTests.cs` ローカル提案JSON蓄積: 不透明で安定した ID の割り当てと legacy migration、全編集対象 field の revision conflict、ハッシュ重複排除、submission finalization race、永続化、破損復旧、アトミック書き込み。提案 redaction のケースは table-driven に保ち、構造化された PascalCase、snake_case、先頭 underscore 付き、recipe 形式の識別子を negative fixture、不透明な混合文字列と既知の token 形式を positive fixture として含めてください。永続化ケースでは、同じ context 内の識別子を保持しつつ secret を伏字化することを確認します。 + 証拠リンクの検証（#5345）では、報告された日付付き Markdown 参照と代表的なソース・文書パスに対し、構成要素内の資格情報、機密名、親ディレクトリへの移動、絶対パス・URL・制御文字、パス・構成要素・入力の上限を net8／net9 で比較します。共有の保存テストでは、リンクと #4403／#4751 の識別子を保持しつつ資格情報を伏字化します。既存の製品ランタイム用の追加・更新テストで、人間向け／JSON の詳細表示と JSON／Markdown／Issue draft のエクスポートから保存先を読み直し、過去の伏字が読み取りだけでは変わらず、明示した原文によってだけ ID を維持した draft を修復できることを確認します。 + レビューの回帰テストでは、引用符付きの機密名、長い引用値や連結値、バッククォート、エスケープ、入れ子や不正なコレクション、YAML のブロック・タグ・アンカー、繰り返し秘匿した結果の安定性、機密値ではない複数の JSON 証拠参照、文章中のアポストロフィ、括弧や引用符内・エスケープされた空白に続く URL・絶対パスの末尾を検証します。値全体の秘匿を、証拠の文脈検査の128文字上限から独立させ、保存とすべての全文 CLI エクスポートでも対照ケースを確認してください。 - `DataDirectorySecurityTests.cs`、`ProgramCliTests.cs` の suggestion-sidecar coverage shared-temp database routing、owner-only の directory / file mode、private directory での隣接配置、structured CLI filesystem failure。 suggestion-history query coverage は、全 documented field の NFKC / case-insensitive 照合、決定的 pagination より先の structured filter、aggregate omission reason、compact redaction、進捗可能な continuation、document 全体の UTF-8 byte budget を同じ production-runtime fixture で維持します。 diff --git a/USER_GUIDE.md b/USER_GUIDE.md index e2eac90e7..857a93872 100644 --- a/USER_GUIDE.md +++ b/USER_GUIDE.md @@ -4039,6 +4039,12 @@ requests therefore observe one complete generation, and an older in-flight Each stored suggestion has an immutable `id` and a mutable `revision_hash`. Editing a draft changes its revision hash without changing the ID, so saved links, abbreviated IDs, `show`, `delete`, and exports keep addressing the same record. When an older store has only `hash`, cdidx adopts that value as the stable ID, computes the current revision, and retains `hash` as a compatibility alias; CLI and MCP JSON expose `id` and `revision_hash` explicitly. +Ordinary repository evidence links such as `artifacts/full-dogfood-20260912/FINDINGS.md#d01` survive `suggestions add`, `show`, and JSON/Markdown/issue-draft export. Use a relative path with forward slashes and ordinary word/date components. The recognizer checks the spelling, not file existence; absolute paths, URLs, traversal, control characters, and secret-shaped components are outside this exception. Other path spellings may still be redacted; see the [exact limits](DEVELOPER_GUIDE.md#suggestion-identity-and-revisions). Existing export text-size limits still apply. + +Credential values do not qualify as evidence links. When a credential uses YAML block, tag, or anchor/alias syntax, the rest of that text field is conservatively redacted, including any following evidence. + +To repair a previously redacted editable local draft, supply the original evidence explicitly: `cdidx suggestions update --context "See artifacts/full-dogfood-20260912/FINDINGS.md#d01" --db `. This replaces the full context, keeps the ID, and changes the revision. Include any other context you want to retain. Reading or exporting an old record never guesses the missing text, and submitted records keep their existing edit restrictions. + On macOS and Linux, an explicit `--db` directly inside a group- or other-writable shared directory such as `/private/tmp` or `/tmp` keeps the database in place but stores its suggestion JSON, archive, and lock files in a deterministic user-scoped private temporary directory with owner-only permissions. Databases inside private directories continue to use colocated sidecars. If the selected parent or private fallback cannot be used, suggestion commands return `E021_SUGGESTION_STORE_UNAVAILABLE`; `--json` also provides a filesystem `category` such as `permission_denied`, `invalid_path`, or `io_error` and a recovery hint instead of falling through to exit 99. Built-in `suggestions-*` store files under `.cdidx` are excluded from indexing and workspace-freshness scans, while indexable configuration such as `.cdidx/patterns/*.yaml` remains visible. Creating the default `.cdidx/suggestions-codeindex.json` store therefore does not make an otherwise fresh index stale. @@ -8025,6 +8031,12 @@ handshake を上書きすることもありません（#4540）。 保存済みの各提案は、不変の `id` と可変の `revision_hash` を持ちます。draft を編集すると revision hash だけが変わり、ID は変わらないため、保存済み link、短縮 ID、`show`、`delete`、export は同じ record を参照し続けます。旧 store に `hash` しかない場合、cdidx はその値を stable ID として採用し、現在の revision を計算したうえで、`hash` を互換 alias として維持します。CLI と MCP の JSON は `id` と `revision_hash` を明示します。 +`artifacts/full-dogfood-20260912/FINDINGS.md#d01` のような通常のリポジトリ内の証拠リンクは、`suggestions add`・`show`・JSON／Markdown／Issue draft のエクスポートで保持されます。区切りに `/` を使い、通常の単語や日付で構成した相対パスを指定してください。検証するのは文字列の構成であり、ファイルの実在ではありません。絶対パス、URL、親ディレクトリへの移動、制御文字、秘密情報の形式を持つ構成要素は例外対象外です。ほかの表記は引き続き伏字になる場合があります。[具体的な条件](DEVELOPER_GUIDE.md#提案-id-と-revision)を参照してください。エクスポート本文の既存の長さ制限も適用されます。 + +資格情報の値は証拠リンクとして扱いません。資格情報が YAML のブロック・タグ・アンカー／エイリアス形式を使う場合は、後続の証拠を含む、そのテキストフィールドの残りを安全側に秘匿します。 + +過去に伏字になった編集可能なローカル draft を修復するには、原文を明示します。例：`cdidx suggestions update --context "参照: artifacts/full-dogfood-20260912/FINDINGS.md#d01" --db `。この操作は context 全体を置き換え、ID を維持したまま revision を変更します。残したい補足情報も一緒に指定してください。既存レコードの表示やエクスポートでは失われた文字列を推測せず、送信済みレコードの編集制限も維持します。 + macOS と Linux では、明示的な `--db` が `/private/tmp` や `/tmp` のような group / other-writable の共有 directory 直下にある場合、database はその場所に維持しつつ、suggestion JSON・archive・lock file を deterministic な user-scoped private temporary directory に owner-only permission で保存します。private directory 内の database では sidecar を従来どおり隣接配置します。選択した親または private fallback を利用できない場合、suggestion command は exit 99 に fall through せず `E021_SUGGESTION_STORE_UNAVAILABLE` を返し、`--json` では `permission_denied`、`invalid_path`、`io_error` などの filesystem `category` と復旧 hint も返します。 `.cdidx` 配下の組み込み `suggestions-*` ストアファイルは index と workspace freshness の scan 対象外ですが、`.cdidx/patterns/*.yaml` のような index 対象の設定は引き続き読み取られます。そのため、既定の `.cdidx/suggestions-codeindex.json` ストアを作成しても、それ以外が fresh な index は stale になりません。 diff --git a/changelog.d/unreleased/5345.fixed.md b/changelog.d/unreleased/5345.fixed.md new file mode 100644 index 000000000..d80d47bf0 --- /dev/null +++ b/changelog.d/unreleased/5345.fixed.md @@ -0,0 +1,17 @@ +--- +category: fixed +issues: + - 5345 +affected: + - src/CodeIndex/Diagnostics/DiagnosticRedactor.cs + - USER_GUIDE.md + - DEVELOPER_GUIDE.md +--- + +## English + +- **Preserve ordinary repository evidence links in suggestions (#5345)** — bounded, validated relative file references such as `artifacts/full-dogfood-20260912/FINDINGS.md#d01` now survive saving, display, and JSON/Markdown/issue-draft export. Credential detection remains active, including quoted/concatenated values, YAML blocks, and path-like text; URL/rooted-path suffixes remain excluded, and the identifier controls from #4403 and #4751 remain covered. Previously redacted editable drafts can be repaired with `suggestions update --context` only when the original text is explicitly supplied. + +## 日本語 + +- **提案内の通常のリポジトリ証拠リンクを保持します（#5345）** — `artifacts/full-dogfood-20260912/FINDINGS.md#d01` のような、長さと構成を検証した相対ファイル参照を、保存・表示・JSON／Markdown／Issue draft のエクスポートで保持するようになりました。引用符付きの値・連結値・YAML ブロック・パス形式の文字列内でも資格情報の検出を維持し、URL・絶対パスの末尾は引き続き例外対象から除外します。#4403 と #4751 の識別子の回帰テストも維持します。過去に伏字になった編集可能な draft は、`suggestions update --context` で原文を明示した場合だけ修復できます。 diff --git a/src/CodeIndex/Diagnostics/DiagnosticRedactor.cs b/src/CodeIndex/Diagnostics/DiagnosticRedactor.cs index 29b472f53..aeeecd8ed 100644 --- a/src/CodeIndex/Diagnostics/DiagnosticRedactor.cs +++ b/src/CodeIndex/Diagnostics/DiagnosticRedactor.cs @@ -43,7 +43,14 @@ internal static class DiagnosticRedactor RegexTimeout); private static readonly Regex SuggestionNamedSecretPattern = new( - $@"(^|[^\p{{L}}\p{{N}}_-])(?[\p{{L}}\p{{N}}_-]*(?:{SensitiveNameClassifier.RegexFragmentPattern})[\p{{L}}\p{{N}}_-]*)=(?[^&\s]+)", + $@"(^|[^\p{{L}}\p{{N}}_-])(?[\p{{L}}\p{{N}}_-]*(?:{SensitiveNameClassifier.RegexFragmentPattern})[\p{{L}}\p{{N}}_-]*)=(?(?![""'`])[^&\s]+)", + RegexOptions.IgnoreCase | RegexOptions.Compiled | RegexOptions.CultureInvariant, + RegexTimeout); + + private static readonly Regex SuggestionStructuredSecretPattern = new( + @"(^|[^\p{L}\p{N}_-])(?[""'`]?)(?[\p{L}\p{N}_-]*(?:" + + SensitiveNameClassifier.RegexFragmentPattern + + @")[\p{L}\p{N}_-]*)\k\s*[:=]\s*(?=[""'`\[{|>!&*])", RegexOptions.IgnoreCase | RegexOptions.Compiled | RegexOptions.CultureInvariant, RegexTimeout); @@ -332,6 +339,7 @@ internal static string RedactSuggestionText(string text, out IReadOnlyCollection types.Add("bearer_token"); return SuggestionRedactedBearerToken; }); + redacted = RedactStructuredSuggestionSecrets(redacted, types); redacted = SuggestionNamedSecretPattern.Replace(redacted, match => RedactSuggestionNamedSecretMatch(match, types)); redacted = KnownStructuredSecretPattern.Replace(redacted, _ => @@ -344,10 +352,12 @@ internal static string RedactSuggestionText(string text, out IReadOnlyCollection types.Add("high_entropy_token"); return SuggestionRedactedHighEntropyToken; }); + var tokenContext = new EvidencePathTokenContext(redacted); redacted = HighEntropyTokenPattern.Replace(redacted, match => { if (match.Value.StartsWith("[REDACTED:", StringComparison.Ordinal) - || LooksLikeStructuredIdentifier(match.Value)) + || LooksLikeStructuredIdentifier(match.Value) + || (tokenContext.HasRelativePrefix(match.Index) && LooksLikeRepositoryEvidencePath(redacted, match))) return match.Value; types.Add("high_entropy_token"); return SuggestionRedactedHighEntropyToken; @@ -374,6 +384,264 @@ private static string RedactSuggestionNamedSecretMatch(Match match, ISet return $"{match.Groups[1].Value}{name}={SuggestionRedactedCredential}"; } + private static string RedactStructuredSuggestionSecrets(string text, ISet types) + { + StringBuilder? builder = null; + var position = 0; + foreach (Match match in SuggestionStructuredSecretPattern.Matches(text)) + { + if (match.Index < position || !IsSensitiveName(match.Groups["name"].Value)) + continue; + var valueStart = match.Index + match.Length; + var valueEnd = FindStructuredSuggestionSecretEnd(text, valueStart); + // Keep an existing unquoted marker stable across repeated redaction. + if (text.AsSpan(valueStart, valueEnd - valueStart).SequenceEqual(SuggestionRedactedCredential)) + continue; + builder ??= new StringBuilder(text.Length); + builder.Append(text.AsSpan(position, valueStart - position)); + var quote = text[valueStart] is '\'' or '`' ? text[valueStart] : '"'; + builder.Append(quote).Append(SuggestionRedactedCredential).Append(quote); + types.Add("credential"); + position = valueEnd; + } + return builder == null ? text : builder.Append(text.AsSpan(position)).ToString(); + } + + private static int FindStructuredSuggestionSecretEnd(string text, int start) + { + // The field is already capped. Consume each value once, including quoted + // prefixes and nested collections, so no unredacted tail reaches the exception. + // YAML blocks, tags and anchors need a full YAML parser to locate their end; + // conservatively redact the remaining bounded field instead. + if (text[start] is '|' or '>' or '!' or '&' or '*') + return text.Length; + Span closers = stackalloc char[16]; + var depth = 0; + var quote = '\0'; + for (var index = start; index < text.Length; index++) + { + var ch = text[index]; + if (quote != '\0') + { + if (ch == '\\') + index++; + else if (ch == quote) + quote = '\0'; + continue; + } + if (depth == 0 && (char.IsWhiteSpace(ch) || ch is ',' or ';' or '&' or ']' or '}' or ')' or '>')) + return index; + if (ch == '\\') + index++; + else if (ch is '"' or '\'' or '`') + quote = ch; + else if (ch is '[' or '{') + { + if (depth == closers.Length) + return text.Length; + closers[depth++] = ch == '[' ? ']' : '}'; + } + else if (ch is ']' or '}') + { + if (depth == 0 || closers[--depth] != ch) + return text.Length; + } + } + return text.Length; + } + + private sealed class EvidencePathTokenContext(string text) + { + private int _scanned; + private int _tokenStart; + private char _quote; + private bool _quoteHasRelativePrefix; + private bool _closedRelativeValue; + + internal bool HasRelativePrefix(int end) + { + // Regex callbacks arrive in order. Scan each character at most once, + // retaining whitespace inside quotes when identifying the enclosing token. + while (_scanned < end) + { + var index = _scanned++; + var ch = text[index]; + if (_quote != '\0') + { + if (ch == '\\' && _scanned < end) + _scanned++; + else if (ch == _quote) + { + _quote = '\0'; + _closedRelativeValue = _quoteHasRelativePrefix; + } + continue; + } + + if (ch == ',' && _closedRelativeValue) + _tokenStart = _scanned; + if (ch is not (']' or '}')) + _closedRelativeValue = false; + if (ch == '\\' && _scanned < end) + _scanned++; + else if (char.IsWhiteSpace(ch)) + _tokenStart = _scanned; + else if (ch is '"' or '`' + || (ch == '\'' && (index == _tokenStart || text[index - 1] is ':' or '=' or '(' or '[' or '{' or '<' or '/' or '\\'))) + { + _quoteHasRelativePrefix = IsRelativePrefix(text.AsSpan(_tokenStart, index - _tokenStart)); + _quote = ch; + } + } + + return IsRelativePrefix(text.AsSpan(_tokenStart, end - _tokenStart)); + } + + private static bool IsRelativePrefix(ReadOnlySpan prefix) + { + if (prefix.Length > 128 || prefix.IndexOfAny("/\\%~") >= 0) + return false; + prefix = prefix.TrimStart("\"'`(<["); + if (prefix.IsEmpty || !char.IsAsciiLetter(prefix[0])) + return true; + foreach (var ch in prefix) + { + if (ch == ':') + return false; + if (!char.IsAsciiLetterOrDigit(ch) && ch is not ('+' or '-' or '.')) + break; + } + return true; + } + } + + private static bool LooksLikeRepositoryEvidencePath(string text, Match match) + { + // This is a lexical exception for complete, bounded relative file references, + // not a filesystem check. Never approve a suffix of a URL, rooted path or token. + var value = match.ValueSpan; + if (value.Length > 260 + || (match.Index > 0 && !IsEvidencePathBoundary(text[match.Index - 1], before: true)) + || (match.Index + match.Length < text.Length + && !IsEvidencePathBoundary(text[match.Index + match.Length], before: false)) + || HasSensitiveEvidencePathContext(text, match.Index)) + return false; + + var lastSlash = value.LastIndexOf('/'); + var extensionStart = value.LastIndexOf('.'); + if (lastSlash <= 0 || extensionStart <= lastSlash + 1) + return false; + + var extension = value[(extensionStart + 1)..]; + if (extension.Length is < 1 or > 8) + return false; + foreach (var ch in extension) + { + if (!char.IsAsciiLetter(ch)) + return false; + } + + var componentStart = 0; + var componentCount = 0; + for (var index = 0; index <= value.Length; index++) + { + if (index < value.Length && value[index] != '/') + continue; + if (++componentCount > 16 + || !IsOrdinaryEvidencePathComponent(value[componentStart..index])) + return false; + componentStart = index + 1; + } + return true; + } + + private static bool IsEvidencePathBoundary(char ch, bool before) => + ch is ' ' or '\r' or '\n' or '"' or '\'' or '`' + || (before ? ch is '(' or '[' or '<' : ch is ')' or ']' or '>' or ',' or ';' or '.' or '#'); + + private static bool HasSensitiveEvidencePathContext(string text, int end) + { + // A token can use ordinary words and slashes too. Do not exempt values + // after e.g. "bearer", "password: " or a quoted credential assignment. + var lowerBound = Math.Max(0, end - 128); + while (end > lowerBound && (char.IsWhiteSpace(text[end - 1]) || text[end - 1] is '"' or '\'' or '`' or '(' or '[' or '<')) + end--; + if (end > lowerBound && text[end - 1] is ':' or '=') + { + end--; + while (end > lowerBound && (char.IsWhiteSpace(text[end - 1]) || text[end - 1] is '"' or '\'' or '`')) + end--; + } + var start = end; + while (start > lowerBound && (char.IsAsciiLetterOrDigit(text[start - 1]) || text[start - 1] is '_' or '-' or '.')) + start--; + return (start == lowerBound && lowerBound > 0) + || (start < end && IsSensitiveName(text[start..end])); + } + + private static bool IsOrdinaryEvidencePathComponent(ReadOnlySpan component) + { + if (component.IsEmpty || component.Length > 100 || IsSensitiveName(component.ToString())) + return false; + + var wordStart = 0; + for (var index = 0; index <= component.Length; index++) + { + if (index < component.Length && component[index] is not ('.' or '-' or '_')) + continue; + if (!IsOrdinaryEvidencePathWord(component[wordStart..index])) + return false; + wordStart = index + 1; + } + return true; + } + + private static bool IsOrdinaryEvidencePathWord(ReadOnlySpan word) + { + if (word.IsEmpty) + return false; + + var letterEnd = word.Length; + while (letterEnd > 0 && char.IsAsciiDigit(word[letterEnd - 1])) + letterEnd--; + // Standalone dates and short version/issue suffixes are ordinary evidence names. + if (letterEnd == 0) + return word.Length <= 8; + if (word.Length - letterEnd > 4) + return false; + + var letters = word[..letterEnd]; + var allLower = true; + var allUpper = true; + foreach (var ch in letters) + { + if (!char.IsAsciiLetter(ch)) + return false; + allLower &= char.IsAsciiLetterLower(ch); + allUpper &= char.IsAsciiLetterUpper(ch); + } + if (allLower || allUpper) + return letters.Length <= (allUpper ? 12 : 24); + + // Check every camel/Pascal word independently: a harmless directory must + // not lend its natural-word signal to an opaque credential in another part. + var wordStart = 0; + for (var index = 1; index <= letters.Length; index++) + { + if (index < letters.Length + && !(char.IsAsciiLetterUpper(letters[index]) + && (char.IsAsciiLetterLower(letters[index - 1]) + || (index + 1 < letters.Length && char.IsAsciiLetterLower(letters[index + 1]))))) + continue; + var part = letters[wordStart..index]; + if (part.Length is < 3 or > 24) + return false; + wordStart = index; + } + return letters.Length < 16 + || (TryAnalyzeIdentifierSegment(letters, out _, out var hasNaturalWord) && hasNaturalWord); + } + private static bool LooksLikeStructuredIdentifier(string value) { var start = 0; diff --git a/tests/CodeIndex.Tests/DiagnosticRedactorTests.cs b/tests/CodeIndex.Tests/DiagnosticRedactorTests.cs index b427d4bf4..98a00fdee 100644 --- a/tests/CodeIndex.Tests/DiagnosticRedactorTests.cs +++ b/tests/CodeIndex.Tests/DiagnosticRedactorTests.cs @@ -24,6 +24,7 @@ public void RedactSuggestionText_UsesSharedTypedPolicy_Issue3933() Assert.DoesNotContain(awsKey, redacted); Assert.DoesNotContain("secret", redacted); Assert.DoesNotContain(privateKeyValue, redacted); + Assert.Equal(redacted, DiagnosticRedactor.RedactSuggestionText(redacted, out _)); Assert.Equal( ["aws_access_key", "bearer_token", "credential", "high_entropy_token"], redactedTypes.Order(StringComparer.Ordinal)); @@ -73,6 +74,180 @@ public void RedactSuggestionText_RedactsOpaqueSecrets_Issue4751(string secret) Assert.Equal(["high_entropy_token"], redactedTypes); } + [Theory] + [InlineData("See artifacts/full-dogfood-20260912/FINDINGS.md#d01")] + [InlineData("[D01](artifacts/full-dogfood-20260912/FINDINGS.md#d01)")] + [InlineData("`artifacts/full-dogfood-20260912/FINDINGS.md#d01`.")] + [InlineData("Evidence:\nartifacts/full-dogfood-20260912/FINDINGS.md#d01\nEnd")] + [InlineData("See artifacts/run-20260912/results_5345.json.")] + [InlineData("docs/audit-20260912/README.en.md")] + [InlineData("src/CodeIndex/Diagnostics/DiagnosticRedactor.Issue5345.cs")] + [InlineData("tests/CodeIndex.Tests/DiagnosticRedactorTests.Issue5345.cs")] + [InlineData("Here's (`artifacts/full-dogfood-20260912/FINDINGS.md#d01`).")] + [InlineData("{\"evidence\":\"artifacts/full-dogfood-20260912/FINDINGS.md#d01\"}")] + [InlineData("{\"evidence\":[\"docs/report-20260912/FINDINGS.md\",\"docs/report-20260913/FINDINGS.md\"]}")] + [InlineData("{\"evidence\":[[\"docs/report-20260912/FINDINGS.md\"],[\"docs/report-20260913/FINDINGS.md\"]]}")] + public void RedactSuggestionText_PreservesOrdinaryEvidenceReferences_Issue5345(string text) + { + var redacted = DiagnosticRedactor.RedactSuggestionText(text, out var types); + + Assert.Equal(text, redacted); + Assert.Empty(types); + } + + [Theory] + [InlineData("aaBB11ccDD22eeFF33ggHH44iiJJ55kk")] + [InlineData("AbCdEfGhIjKlMnOpQrStUvWxYz123456")] + [InlineData("AbcDefGhiJklMnoPqrStuVwxYz123456")] + [InlineData("docs-aaBB11ccDD22eeFF33ggHH44iiJJ55kk")] + [InlineData("AbcDefGhiJklMnoPqrStuVwx1234")] + [InlineData("ghp_" + "abcdefghijklmnopqrstuvwxyz123456")] + [InlineData("github_" + "pat_11AA22bb33CC44dd55EE66ff77GG88hh")] + [InlineData("glpat" + "-AbcdefGhijklMnopqrstUvwx123456")] + [InlineData("sk_" + "live_AbcdefGhijklMnopqr123456")] + [InlineData("AKIA" + "1234567890ABCDEF")] + [InlineData("Bearer AbCdEfGhIjKlMnOpQrStUvWxYz123456")] + [InlineData("api_key=explicit-secret-value")] + [InlineData("password=explicit-secret-value")] + public void RedactSuggestionText_PathComponentsCannotHideCredentials_Issue5345(string secret) + { + const string evidence = "artifacts/full-dogfood-20260912/FINDINGS.md#d01"; + var text = $"See {evidence}; credential artifacts/{secret}/FINDINGS.md#d02"; + + var redacted = DiagnosticRedactor.RedactSuggestionText(text, out var types); + + Assert.Contains(evidence, redacted); + Assert.DoesNotContain(secret, redacted); + Assert.NotEmpty(types); + Assert.Contains("[REDACTED:", redacted); + } + + [Theory] + [InlineData("../artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("./artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("artifacts/../full-dogfood-20260912/FINDINGS.md")] + [InlineData("artifacts//full-dogfood-20260912/FINDINGS.md")] + [InlineData("/artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("C:/artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("C:\\artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("//host/artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("~/artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("https://example.test/artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("https://example.test/(artifacts/full-dogfood-20260912/FINDINGS.md)")] + [InlineData("/tmp/(artifacts/full-dogfood-20260912/FINDINGS.md)")] + [InlineData("/tmp/[artifacts/full-dogfood-20260912/FINDINGS.md]")] + [InlineData("\"/tmp/private directory/(artifacts/full-dogfood-20260912/FINDINGS.md)\"")] + [InlineData("/tmp/\"private directory (artifacts/full-dogfood-20260912/FINDINGS.md)\"")] + [InlineData("/tmp/private\\ (artifacts/full-dogfood-20260912/FINDINGS.md)")] + [InlineData("/tmp/(\"first\",\"artifacts/full-dogfood-20260912/FINDINGS.md\")")] + [InlineData("file:(\"first\",\"artifacts/full-dogfood-20260912/FINDINGS.md\")")] + [InlineData("C:\\private\\(artifacts/full-dogfood-20260912/FINDINGS.md)")] + [InlineData("file:(artifacts/full-dogfood-20260912/FINDINGS.md)")] + [InlineData("artifacts/full-dogfood-20260912/FINDINGS.md?query=value")] + [InlineData("artifacts/full-dogfood-20260912/FINDINGS.md\u001b")] + [InlineData("artifacts/full-dogfood-20260912/FINDINGS.md\t")] + [InlineData("artifacts/invalid\u0001full-dogfood-20260912/FINDINGS.md")] + [InlineData("artifacts/token-20260912/FINDINGS.md")] + [InlineData("artifacts/password/FINDINGS-20260912.md")] + [InlineData("artifacts/abcdefghijklmnopqrstuvwx123456/FINDINGS.md")] + [InlineData("bearer artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("bEaReR artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("password: artifacts/full-dogfood-20260912/FINDINGS.md")] + [InlineData("api_key : \"artifacts/full-dogfood-20260912/FINDINGS.md\"", "credential")] + [InlineData("{\"password\":\"Maple/Copper/Harbor/Calendar2026.json\"}", "credential")] + [InlineData("{\"api_key\": \"artifacts/full-dogfood-20260912/FINDINGS.md\"}", "credential")] + [InlineData("'password': 'artifacts/full-dogfood-20260912/FINDINGS.md'", "credential")] + [InlineData("\"password\" = \"artifacts/full-dogfood-20260912/FINDINGS.md\"", "credential")] + [InlineData("password: [artifacts/full-dogfood-20260912/FINDINGS.md]", "credential")] + [InlineData("{\"password\": [\"artifacts/full-dogfood-20260912/FINDINGS.md\"]}", "credential")] + [InlineData("\"password\": artifacts/full-dogfood-20260912/FINDINGS.md")] + public void RedactSuggestionText_DoesNotExemptUnvalidatedPaths_Issue5345(string text, string expectedType = "high_entropy_token") + { + var redacted = DiagnosticRedactor.RedactSuggestionText(text, out var types); + + Assert.NotEqual(text, redacted); + Assert.Contains($"[REDACTED:{expectedType}]", redacted); + Assert.Equal([expectedType], types); + } + + [Fact] + public void RedactSuggestionText_StructuredCredentialsDoNotExposeLaterEvidenceLikeValues_Issue5345() + { + const string evidence = "artifacts/full-dogfood-20260912/FINDINGS.md#d01"; + var value = new string('x', 260) + " " + evidence + " suffix"; + foreach (var assignment in new[] + { + $"password=\" {value}\"", + $"{{\"password\":\"prefix \\\"quoted\\\" {value}\"}}", + $"'api_key': '{value}'", + $"{{\"password\": [\"first value\", {{\"nested\": \"{value}\"}}]}}", + $"{{\"password\": {{\"nested\": [\"{value}\"]}}}}", + $"password: `prefix {value}`", + $"password=\"{value}\"private-tail", + "password=\"\"private-tail", + "password='first'private-tail", + "password=[first]private-tail", + "password=[REDACTED:credential]private-tail", + "password=\"first\"\\ private-tail", + "password='first'\" private-tail\"", + }) + { + var redacted = DiagnosticRedactor.RedactSuggestionText(assignment + $"; See {evidence}", out var types); + + Assert.DoesNotContain(value, redacted); + Assert.DoesNotContain("private-tail", redacted); + Assert.EndsWith($"; See {evidence}", redacted); + Assert.Equal(redacted.IndexOf(evidence, StringComparison.Ordinal), redacted.LastIndexOf(evidence, StringComparison.Ordinal)); + Assert.Contains(DiagnosticRedactor.SuggestionRedactedCredential, redacted); + Assert.Equal(["credential"], types); + Assert.Equal(redacted, DiagnosticRedactor.RedactSuggestionText(redacted, out _)); + } + } + + [Fact] + public void RedactSuggestionText_EvidenceRecognitionRetainsInputBounds_Issue5345() + { + var tooManyComponents = "artifacts/" + string.Concat(Enumerable.Repeat("audit/", 16)) + "FINDINGS-20260912.md"; + var tooLongPath = "artifacts/" + string.Concat(Enumerable.Repeat(new string('a', 24) + "/", 10)) + "FINDINGS-20260912.md"; + var tooLongComponent = "artifacts/" + string.Join("-", Enumerable.Repeat("audit", 21)) + "/FINDINGS-20260912.md"; + foreach (var path in new[] { tooManyComponents, tooLongPath, tooLongComponent }) + { + Assert.Equal(DiagnosticRedactor.SuggestionRedactedHighEntropyToken, + DiagnosticRedactor.RedactSuggestionText(path, out var pathTypes)); + Assert.Equal(["high_entropy_token"], pathTypes); + } + + const string evidence = "artifacts/full-dogfood-20260912/FINDINGS.md#d01"; + var text = evidence + new string(' ', DiagnosticRedactor.SuggestionRedactionFieldLengthLimit) + "private-tail"; + var redacted = DiagnosticRedactor.RedactSuggestionText(text, out var types); + + Assert.StartsWith(evidence, redacted); + Assert.DoesNotContain("private-tail", redacted); + Assert.EndsWith(DiagnosticRedactor.SuggestionRedactionTruncationMarker, redacted); + Assert.Equal(["truncated"], types); + Assert.Equal(DiagnosticRedactor.SuggestionRedactionFieldLengthLimit + + DiagnosticRedactor.SuggestionRedactionTruncationMarker.Length, redacted.Length); + + var distantCredential = "bearer" + new string(' ', 128) + evidence; + Assert.DoesNotContain(evidence, DiagnosticRedactor.RedactSuggestionText(distantCredential, out var contextTypes)); + Assert.Equal(["high_entropy_token"], contextTypes); + + foreach (var malformedValue in new[] { "[\"unterminated ", "[{]", new string('[', 17) }) + { + var malformed = "password: " + malformedValue + evidence; + Assert.DoesNotContain(evidence, DiagnosticRedactor.RedactSuggestionText(malformed, out var malformedTypes)); + Assert.Equal(["credential"], malformedTypes); + } + + // Unsupported YAML value syntax is deliberately fail-closed across the field. + foreach (var yamlHeader in new[] { "|", "|-", ">", ">-", "|2-", "!!str", "&anchor |", "*alias" }) + { + var yaml = "password: " + yamlHeader + "\n prefix\n " + new string('x', 260) + "\n " + evidence; + Assert.DoesNotContain(evidence, DiagnosticRedactor.RedactSuggestionText(yaml, out var yamlTypes)); + Assert.Equal(["credential"], yamlTypes); + } + } + [Theory] [InlineData("--github-token")] [InlineData("github_token")] diff --git a/tests/CodeIndex.Tests/ProgramCliTests.cs b/tests/CodeIndex.Tests/ProgramCliTests.cs index e88e43a24..dc9904e76 100644 --- a/tests/CodeIndex.Tests/ProgramCliTests.cs +++ b/tests/CodeIndex.Tests/ProgramCliTests.cs @@ -1530,6 +1530,10 @@ public void Suggestions_ShowRejectsPaginationFlags() public void Suggestions_AddJsonCreatesLocalDraftAndDeduplicates_Issue4310() { using var fixture = SuggestionFixture.Create(); + const string evidence = "artifacts/full-dogfood-20260912/FINDINGS.md#d01"; + const string secondEvidence = "docs/report-20260912/FINDINGS.md"; + const string thirdEvidence = "docs/report-20260913/FINDINGS.md"; + var context = $"See {evidence}; {{\"evidence\":[\"{secondEvidence}\",\"{thirdEvidence}\"]}}"; string[] addArgs = [ "suggestions", "add", @@ -1539,7 +1543,7 @@ public void Suggestions_AddJsonCreatesLocalDraftAndDeduplicates_Issue4310() "--category", "output_format", "--language", "csharp", "--agent", "codex", - "--context", "Observed while triaging local audit output.", + "--context", context, "--title", "Local dogfood finding store", "--evidence-path", "src/CodeIndex/Cli/SuggestionsCommandRunner.cs", ]; @@ -1575,10 +1579,30 @@ public void Suggestions_AddJsonCreatesLocalDraftAndDeduplicates_Issue4310() Assert.Equal("output_format", suggestion.GetProperty("category").GetString()); Assert.Equal("csharp", suggestion.GetProperty("language").GetString()); Assert.Equal("codex", suggestion.GetProperty("agent").GetString()); - Assert.Equal("Observed while triaging local audit output.", suggestion.GetProperty("context").GetString()); + Assert.Equal(context, suggestion.GetProperty("context").GetString()); Assert.Equal("Local dogfood finding store", suggestion.GetProperty("sampled_title").GetString()); Assert.Equal("src/CodeIndex/Cli/SuggestionsCommandRunner.cs", suggestion.GetProperty("evidence_paths")[0].GetString()); Assert.Equal(suggestion.GetProperty("id").GetString(), listDoc.RootElement.GetProperty("results")[0].GetProperty("id").GetString()); + + // Reopen the persisted store through every full-text display/export surface (#5345). + var id = suggestion.GetProperty("id").GetString()!; + foreach (var readArgs in new string[][] + { + ["suggestions", "show", id, "--db", fixture.DbPath], + ["suggestions", "show", id, "--db", fixture.DbPath, "--json"], + ["suggestions", "export", "--db", fixture.DbPath, "--format", "json"], + ["suggestions", "export", "--db", fixture.DbPath, "--format", "markdown"], + ["suggestions", "export", "--db", fixture.DbPath, "--format", "issue-drafts"], + }) + { + var (readExitCode, readStdout, readStderr) = RunCliInSubprocess(readArgs); + Assert.Equal(CommandExitCodes.Success, readExitCode); + Assert.Equal(string.Empty, readStderr); + Assert.Contains(evidence, readStdout); + Assert.Contains(secondEvidence, readStdout); + Assert.Contains(thirdEvidence, readStdout); + Assert.DoesNotContain("[REDACTED:", readStdout); + } } [ProductionRuntimeFact] @@ -1679,11 +1703,21 @@ public void Suggestions_AddHelpIncludesLocalDraftFlags_Issue4310() public void Suggestions_UpdatePreservesStableIdAcrossShowExportAndDelete_Issue4588() { using var fixture = SuggestionFixture.Create(); - var record = fixture.Add("bug", "csharp", "Malformed draft description", submitted: false, context: "bad context", sampledTitle: "Stale title"); + const string historicalContext = "See [REDACTED:high_entropy_token]#d01"; + const string originalContext = "See artifacts/full-dogfood-20260912/FINDINGS.md#d01"; + var record = fixture.Add("bug", "csharp", "Malformed draft description", submitted: false, context: historicalContext, sampledTitle: "Stale title"); + + var (beforeExitCode, beforeStdout, beforeStderr) = RunCliInSubprocess([ + "suggestions", "show", record.Hash[..12], "--db", fixture.DbPath, "--json" + ]); + Assert.Equal(CommandExitCodes.Success, beforeExitCode); + Assert.Equal(string.Empty, beforeStderr); + using var beforeDoc = JsonDocument.Parse(beforeStdout); + Assert.Equal(historicalContext, beforeDoc.RootElement.GetProperty("context").GetString()); var (updateExitCode, updateStdout, updateStderr) = RunCliInSubprocess([ "suggestions", "update", record.Hash[..12], "--db", fixture.DbPath, "--json", - "--description", "Corrected draft description", "--context", "correct context", "--title", "Corrected title", + "--description", "Corrected draft description", "--context", originalContext, "--title", "Corrected title", "--evidence-path", "src/CodeIndex/Cli/SuggestionsCommandRunner.cs" ]); var (showExitCode, showStdout, showStderr) = RunCliInSubprocess([ @@ -1712,7 +1746,7 @@ public void Suggestions_UpdatePreservesStableIdAcrossShowExportAndDelete_Issue45 var revisionHash = suggestion.GetProperty("revision_hash").GetString(); Assert.Equal("updated", updateDoc.RootElement.GetProperty("action").GetString()); Assert.Equal("Corrected draft description", suggestion.GetProperty("description").GetString()); - Assert.Equal("correct context", suggestion.GetProperty("context").GetString()); + Assert.Equal(originalContext, suggestion.GetProperty("context").GetString()); Assert.Equal("Corrected title", suggestion.GetProperty("sampled_title").GetString()); Assert.Equal("src/CodeIndex/Cli/SuggestionsCommandRunner.cs", suggestion.GetProperty("evidence_paths")[0].GetString()); Assert.Equal(record.Hash, suggestion.GetProperty("id").GetString()); @@ -1720,7 +1754,9 @@ public void Suggestions_UpdatePreservesStableIdAcrossShowExportAndDelete_Issue45 Assert.Equal(record.CreatedAt, suggestion.GetProperty("created_at").GetDateTime()); Assert.Equal(record.Hash, showDoc.RootElement.GetProperty("id").GetString()); Assert.Equal(revisionHash, showDoc.RootElement.GetProperty("revision_hash").GetString()); + Assert.Equal(originalContext, showDoc.RootElement.GetProperty("context").GetString()); var exported = Assert.Single(exportDoc.RootElement.GetProperty("suggestions").EnumerateArray()); + Assert.Equal(originalContext, exported.GetProperty("context").GetString()); Assert.Equal(record.Hash, exported.GetProperty("id").GetString()); Assert.Equal(revisionHash, exported.GetProperty("revision_hash").GetString()); Assert.Equal(record.Hash, deleteDoc.RootElement.GetProperty("suggestion").GetProperty("id").GetString()); diff --git a/tests/CodeIndex.Tests/SuggestionStoreTests.cs b/tests/CodeIndex.Tests/SuggestionStoreTests.cs index 46a317d8b..5c5eb0efe 100644 --- a/tests/CodeIndex.Tests/SuggestionStoreTests.cs +++ b/tests/CodeIndex.Tests/SuggestionStoreTests.cs @@ -582,12 +582,13 @@ public void TryAdd_RedactsSensitiveTextBeforePersistence() { var testIdentifier = "RedactSuggestionText_DoesNotBorrowEntropySignalsFromLaterText_Issue4403"; var opaqueSecret = "aaBB11ccDD22eeFF33ggHH44iiJJ55kk"; + const string evidence = "artifacts/full-dogfood-20260912/FINDINGS.md#d01"; var record = MakeRecord( "other", null, "AWS AKIA1234567890ABCDEF and password=swordfish and token=tok123 and github_token=git123 and api_key=abc123 and openai_api_key=oa123 and access-key=def456 and CDIDX_GITHUB_TOKEN=cdidx123 and Bearer AbCdEfGhIjKlMnOpQrStUvWxYz123456 should not persist"); - record.Context = $"test {testIdentifier}; token {opaqueSecret}"; - record.ToolInvocationContext = "secret=hunter2 access_key=ghi789"; + record.Context = $"See {evidence}; test {testIdentifier}; token artifacts/{opaqueSecret}/FINDINGS.md"; + record.ToolInvocationContext = "secret=hunter2 access_key=ghi789 {\"password\":\"Maple/Copper/Harbor/Calendar2026.json\"} password=\"\"private-tail; password: |\n Maple/Copper/Harbor/Calendar2027.json"; record.SampledTitle = "Sensitive text redaction"; record.SampledTags = ["security", "suggestions"]; record.EvidencePaths = ["src/CodeIndex/Cli/SuggestionStore.cs"]; @@ -605,6 +606,7 @@ public void TryAdd_RedactsSensitiveTextBeforePersistence() Assert.Contains("CDIDX_GITHUB_TOKEN=[REDACTED:credential]", stored.Description); Assert.Contains("[REDACTED:bearer_token]", stored.Description); Assert.Contains(testIdentifier, stored.Context); + Assert.Contains(evidence, stored.Context); Assert.Contains("[REDACTED:high_entropy_token]", stored.Context); Assert.DoesNotContain(opaqueSecret, stored.Context); Assert.Contains("secret=[REDACTED:credential]", stored.ToolInvocationContext); @@ -622,6 +624,9 @@ public void TryAdd_RedactsSensitiveTextBeforePersistence() Assert.DoesNotContain("cdidx123", stored.Description); Assert.DoesNotContain("hunter2", stored.ToolInvocationContext); Assert.DoesNotContain("ghi789", stored.ToolInvocationContext); + Assert.DoesNotContain("Maple/Copper/Harbor/Calendar2026.json", stored.ToolInvocationContext); + Assert.DoesNotContain("private-tail", stored.ToolInvocationContext); + Assert.DoesNotContain("Maple/Copper/Harbor/Calendar2027.json", stored.ToolInvocationContext); } [Fact]