Repository navigation
Rules-aware skip memo: re-decode only changed records, invalidate on any rule change #154
Description
Prompt
Add to crawlkit a small helper that lets a crawler skip re-decoding source records whose bytes have not changed since the last sync, and that invalidates every skip automatically when the crawler's decoding rules change. API sketch:
state.RulesSignature(versions map[string]int, rules any) []bytehashes the declared decoder and derivation versions plus a canonical, path-tagged rendering of the rule tables (strings quoted, maps and sets in key order, compiled regular expressions by pattern; panic on an unknown kind so a new table cannot be skipped silently), and a per-record memo compares(source key, byte digest, signature)before the crawler decodes. Keep record storage in the app; crawlkit owns only the signature and the comparison. Reference implementation: https://github.com/ourostack/teamscrawl/blob/main/internal/teamsdesktop/memo.go (MemoSignature,hashValue).
Why
A crawler that re-reads a large local cache on every sync pays to decode records that have not changed. teamscrawl skips them with a per-record byte digest, which made a re-sync of a real Teams cache (about 62,000 messages) do work in proportion to what changed. The subtle part is invalidation: a new credential-scrub key, deny term or regular expression must re-read everything, and hashing the rule tables themselves (rather than relying on someone remembering to bump a version) is what makes that safe.
Why an issue, not a PR
Only teamscrawl uses this today, and crawlkit's boundary asks for at least two apps. If another crawler (slacrawl, discrawl, imsgcrawl) would adopt it, we can send a PR from the proven code; say the word.
Activity
clawsweeper commentedon Oct 6, 2026
Codex review: this still needs some work.
Summary
This issue remains unimplemented; repository policy permits automatic closure only for implemented requests, and shared extraction still needs a second demonstrated consumer and a maintainer scope decision.
Reproducibility: This is a feature proposal with concrete downstream source; the reported Teams cache performance and memo invalidation were not independently exercised.
Maintainer decision needed
Next step
Decide whether to wait for a second demonstrated consumer or sponsor one before defining the shared memo API.
Review details
Best possible solution:
Keep the helper downstream until two crawlers establish a compatible signature and comparison contract with application-owned storage.
Do we have a high-confidence way to reproduce the issue?
This is a feature proposal with concrete downstream source; the reported Teams cache performance and memo invalidation were not independently exercised.
Is this the best way to solve the issue?
The reference is a useful extraction candidate after shared adoption is established, while decoder and rule-application code changes must retain explicit version invalidation.
AGENTS.md: found and applied where relevant.
Remaining risk / open question:
Codex review notes: model internal, reasoning medium; reviewed against 4fac9f572848.
Label changes
Label changes:
No label changes.
Label justifications:
P3: This optional optimization establishes no broken crawlkit contract and awaits demonstrated shared adoption.Evidence reviewed
What I checked:
docs/boundary.md:3, 4fac9f572848)state/state.go:95, 4fac9f572848)worker/worker.go:25, 4fac9f572848)How this review workflow works
@clawsweeper re-reviewcomments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.@clawsweeper re-reviewor@clawsweeper re-runon an open PR or issue to request a fresh review only.@clawsweeper reviewto request a fresh review only.@clawsweeper explainto ask for more context, or@clawsweeper stopto stop active automation.Reviewed November 3, 2026, 11:59 PM ET / November 4, 2026, 04:59 UTC.
281 remaining items