Repository navigation
fix(contextmgr): 存储脱敏不再二次改写已编码的 handoff JSON - #497
Open
Qiyuanqiii wants to merge 1 commit into
Open
Qiyuanqiii wants to merge 1 commit into
Qiyuanqiii wants to merge 1 commit into
Conversation
…ff JSON PrepareSummaryForStorage applied the secret regexes to content that was already a handoff_memory envelope, so the pass ran over serialized JSON whose record text is bound by per-record digests. The assigned-value class consumes the JSON escape of an excerpt marker, and redact.String is not idempotent, so the rewrite left content_sha256 binding the old bytes and validateHandoffEnvelope rejected the record the compactor had just produced. saveContextSummaryTx therefore refused a handoff for its own session, and compactAfterTurn returns that error before it can mark messages compacted, so every later turn of that session failed too. Only plain pre-envelope content still needs the pass; a stored envelope is trimmed and re-validated instead of rewritten. Generated content keeps its existing strict receipt check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PrepareSummaryForStorage对已经是handoff_memory.v1信封的正文再跑一次脱敏(internal/contextmgr/handoff.go:524,修复前的 else 分支),而此时summary.Content是序列化好的 JSON,记录文本已经被 per-record 的content_sha256钉住。脱敏里assigned-secret这条规则(internal/redact/redact.go:60-62)值是[^\s"']{8,},它可以跨过 JSON 转义把 excerpt 标记的开头吃掉:Compact产出的这条记录原文是编成 JSON 后
\n变成\+n两个字符,正则把\n[连同其后内容当作"待脱敏的值"改写,解码回来的记录文本就成了正文变了,
content_sha256还是按旧字节算的,于是validateHandoffEnvelope在internal/contextmgr/handoff.go:489(record.ContentSHA256 != handoffContentSHA256(record.Content))判定handoff memory record is invalid。也就是说:压缩器自己产出的 handoff 被它自己的写入校验拒绝了。这条路径每次都走到。
Compact在handoff.go:552-556就把ProtocolVersion设成HandoffMemoryProtocolVersion并把 Content 换成编码后的信封,紧接着context.go:146调m.store.SaveContextSummary,落到SQLiteStore.saveContextSummaryTx(internal/store/sqlite.go:683-688)——那里第一行就是contextmgr.PrepareSummaryForStorage(summary)。所以每次落盘都会对信封再脱敏一次。读取侧同样严格:internal/store/sqlite.go:753每个读都跑ValidateStoredSummary。用户可见后果不止一次写失败。
session.compactAfterTurn(internal/session/session.go:681-692)把MaybeCompact的错误原样返回,因此MarkSessionMessagesCompacted根本执行不到,消息永远不会被标记为已压缩;Send在internal/session/session.go:336-338直接把这个错误抛给调用方。一个恰好把边界压在已脱敏标记之后的载荷,会让这个 session 之后每一轮都失败,而不是只丢一次压缩。本包其实已经写下并回避了这个危险:generated 分支的注释明确写着 "applying a text regex to serialized JSON would consume escaped newlines and invalidate an honest digest"(
internal/contextmgr/handoff.go:516-518),所以带 receipt 的信封走的是"校验每个解码后的字符串"而不是重写。缺的正是没有 receipt 的信封这一支——它们同样带 per-record 摘要,却仍在else里挨一次 JSON 级正则。修法按同一判据收口:正文已经是 handoff 信封时不再跑正则,只
TrimSpace,之后交给本来就会执行的信封校验;仍然需要脱敏的是"还没编成 JSON 的普通文本",也就是ProtocolVersion == ""那一支(它在:534之后才 excerpt、才绑定摘要,顺序是对的)。generated 分支的validateGeneratedHandoffRedaction原样保留,strict decode 和它的错误文本都没动。关于"少跑一次脱敏会不会降低掩蔽强度",我按两条线核过:
session.ProjectContextMessage投影时就用redact.String(message.Content)并以脱敏后文本算ContentSHA256(internal/session/provenance.go:169/173/177);PrepareSummaryForStorage的普通文本分支照旧脱敏,随后才 excerpt 并绑定记录摘要。信封里的记录文本按构造就是已掩蔽的。summary.Content既不含原始凭据supersecretvalue123、又必须留着[REDACTED:标记。需要说明的取舍(这条我不回避):修复前若真有未掩蔽秘密混进信封,正则确实会改写它,但改写必然破坏
content_sha256一致,写入照样被拒——所以旧行为对这种情况不是成功掩蔽,而是失败。修复后这类内容会被判定为"文本与摘要自洽但含秘密",而自洽要求伪造方同时伪造摘要;能自由构造 summary 的调用方本来就已经能写入任意掩蔽后的文本。我选择保留这一层:如果维护者希望在信封层也拒绝未掩蔽文本,正确的做法是让redact提供"是否仍含活秘密"的判定,而不是把正则打到 JSON 上。顺带说明我试过的另一条路:先要求"每个解码字符串都是脱敏不动点",结果在 filler 386–390 产生 5 处假拒绝——因为redact.String本身不幂等(已生成的标记会被再掩蔽一次),所以不动点判据在这里不成立,这也是最终选择"不再跑第二次正则"的原因。红→绿(都是把
handoff.go的条件临时还原成修复前、跑完再恢复得到的,未提交任何临时文件):internal/contextmgr新用例扫 61 个填充位置(filler 360–420):修复前第一处失败即filler 385: the storage write pass rejected the handoff it was given: handoff memory record is invalid,TestPrepareSummaryForStorageIsIdempotentForStoredHandoff同样失败;修复后 61 处全部通过。internal/store端到端用例走NewManager(st, DefaultConfig()).Compact→ 真实saveContextSummaryTx:修复前the store rejected the handoff its own compactor produced: handoff memory record is invalid(FAIL),修复后ok cyberagent-workbench/internal/store 1.164s。TestPrepareSummaryForStorageMasksPlainContentBeforeDigestBinding(普通文本仍被脱敏、记录不留原始凭据、摘要与文本一致)在修复前后都通过,用来证明我没有把还需要的那一层一起删掉。验证:
go test -count=1 ./internal/contextmgr/ ./internal/session/通过:ok contextmgr 0.710s、ok session 0.899s(全包)。go test -count=1 -run "Context|Handoff|Compact|Summary" ./internal/store/ ./internal/application/通过:ok store 89.033s、ok application 111.398s(这两个包的全量在本机单包会撞 10 分钟默认超时,CI 拆 shard,所以我只跑了与本改动相关的用例,全量以 PR CI 为准)。go vet ./internal/contextmgr/ ./internal/store/ ./internal/session/通过;gofmt -l对改动的三个文件无输出;git diff --check干净。ocr delegate preview --from 274fefb6 --to HEAD→ 1 个可复核文件 / 共 3 个改动文件(+185/-6),两个_test.go被default_path规则排除,已人工复核(新增用例只做断言与夹具,未放宽任何既有期望)。按**/*.go规则组逐条核对:Security-Sensitive Boundaries——被移除的是对已编码 JSON 的重复脱敏,掩蔽仍在文本层且由 store 用例端到端断言,见上文;Errors/API 契约——未新增错误路径,storedHandoffEnvelope只在 decode 成功且 version 匹配时才跳过正则,decode 失败仍走原正则支并保持既有decode handoff summary: %w错误;Nil/接口——无新接口、无指针解引用;并发/共享状态——纯函数,无跨调用状态;资源生命周期——无新增句柄或Rows;数值边界——无算术。额外一点成本说明:该分支现在对正文多一次json.Unmarshal(信封大小已由MaxHandoffMemoryChars限界,且:566本来就要再解一次)。无发现。无 Surface 变化:DTO、请求/响应形状、
docs/openapi.json与protocols/registry.json都不受影响(本改动只让写入不再拒绝自家产物);不改变权限、Scope、沙箱、预算或执行判定;不含凭据与本地运行数据(测试用合成token=supersecretvalue123)。