Skip to content

fix(forester): retain and reuse proofs across eligibility windows - #2391

Draft
sergeytimoshin wants to merge 7 commits into
agent/cache-late-proof-resultsfrom
fix/late-proof-cache-timeout
Draft

sergeytimoshin wants to merge 7 commits into
agent/cache-late-proof-resultsfrom
fix/late-proof-cache-timeout

Conversation

@sergeytimoshin

@sergeytimoshin sergeytimoshin commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • retain V2 proofs that finish after the 30-second cache-warming deadline instead of dropping the receiver
  • release cache warming at the deadline so proof scheduling can continue, while a bounded collector keeps results for up to 10 minutes
  • build reusable proof chains by root linkage, including out-of-order completions, and validate them against the current on-chain root before submission
  • keep cache warmups generation-scoped and cancellation-safe
  • flush partial proof batches before eligibility ends, refinalize stale eligibility, and bound speculative prewarming

Depends on #2387.

@coderabbitai

coderabbitai Bot commented Sep 23, 2026

Copy link
Copy Markdown
Contributor

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sergeytimoshin sergeytimoshin changed the title fix(forester): bound late proof cache warming fix(forester): make proof cache warmups cancellation-safe Sep 24, 2026
@sergeytimoshin sergeytimoshin changed the title fix(forester): make proof cache warmups cancellation-safe fix(forester): retain and reuse proofs across eligibility windows Sep 24, 2026
@sergeytimoshin

Copy link
Copy Markdown
Contributor Author

Follow-up fix: 1ecdb0a.

Ready proof prefixes can now be sent while collection continues; cache entries are acknowledged only after confirmation. Cached-send errors remain typed so ForesterNotEligible triggers schedule recovery. Pending collections suppress duplicate speculative chains, out-of-order handoff retains proofs across gaps, and prover payloads no longer include the changing local sequence number. Cached-work metrics now count queue items rather than proof instructions.

Validation: 70 forester library tests passed; cargo clippy -p forester --lib -- -D warnings passed; forester formatting and release build passed.

Devnet canary, 2026-09-24 17:22:41–17:25:37 UTC: updated devnet-1 confirmed 5 cached-proof transactions / 2,500 queue items with zero cached-send failures and zero warming deferrals. Unchanged devnet-2 confirmed no processing transactions in the same interval and logged 84 warming deferrals. This is a short operational canary, not a controlled sustained-throughput benchmark.

Promoted devnet-2 after the canary; both APIs healthy with zero container restarts on the final image. Devnet-2 subsequently confirmed a 500-item batch. Mainnet and the shared prover were not restarted. An initial host-binary/runtime glibc mismatch was rolled back and corrected before the successful canary; final image: light-forester:ready-proof-cache-1ecdb0a24-noble.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant