fix: expire every build key at write time - #1
Merged
prsimp merged 2 commits intoSep 22, 2026
Merged
Conversation
The EXPIRE for `processed` and `worker:<id>:queue` lived at the tail of Worker#poll, and `running`, `owners`, `requeues-count`, `warnings`, `test_failed_count` and `created-at` never got one at all. Any worker that is SIGKILLed, cancelled, OOMs or loses its Redis connection never reaches the end of poll, so those keys leaked permanently. On Kajabi's ci-queue node that reached 464k permanent keys holding 9.5 GiB: `processed` 4.6 GiB, `worker:<n>:queue` 3.8 GiB, `requeues-count` 1.0 GiB. Because the node runs `volatile-lru`, the leaked keys were the only ones ineligible for eviction, so memory pressure fell entirely on the live queue keys of in-flight builds -- producing red spec lanes with no failing spec. Every key is now expired at the point of write, so no key's lifetime depends on a clean shutdown. The TTL is threaded through the Lua scripts as an argument and through the spawned heartbeat monitor process. Also: `rspec-queue --report` exited 0 against a queue whose keys were gone. An evicted queue reads as exhausted and its error-report hash reads as empty, so the report certified a run whose results it no longer had. It now refuses to report a result when `total` is 0 or no progress was recorded, and explains that this is eviction rather than a test failure.
This fork carries Kajabi-only patches on top of upstream 0.66.0 without bumping the version (see b304463), and the monolith's supply-chain cooldown check fails closed on a version it cannot find on rubygems. The Gemfile.lock revision is the real identifier for a git-sourced gem.
rwc9u
approved these changes
Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
tools-redis-001, the ElastiCache node behindCI_QUEUE_URL, was pinned at 100% memory with 469,496 keys of which only 1,484 had a TTL. Under itsvolatile-lrupolicy the only evictable keys were ci-queue's live queue bookkeeping, so builds losttotal/master-statusmid-run and ruby spec lanes went red with no failing spec (Total tests in queue: 0, orCI::Queue::Redis::LostMaster).The permanent keys were ci-queue's own. Measured on the node before cleanup:
processedworker:<n>:queuerequeues-countowners/running/warningsThat accounts for the whole 9.7 GiB.
What changed
TTLs are now set at write time. The
EXPIREforprocessedandworker:<id>:queueused to live at the tail ofWorker#poll, inside a method whose bottom isrescue *CONNECTION_ERRORS. A worker that is SIGKILLed, cancelled, OOMs, or loses its Redis connection never reaches that line, so the key leaked forever.running,owners,requeues-count,warnings,test_failed_countandcreated-atnever got an expiry at all.The TTL is threaded through the Lua scripts as an argument, and through the spawned heartbeat monitor process. No key's lifetime now depends on a clean shutdown.
rspec-queue --reportno longer certifies a vanished queue. An evicted queue reads as exhausted and its error-report hash reads as empty, so--reportprinted "No errors found" and exited 0 for a run whose results were gone. It now refuses to report a result whentotalis 0 or no progress was recorded, and says explicitly that this indicates eviction rather than a test failure.Verification
Driving reserve/requeue/acknowledge by hand so the end-of-poll
EXPIREnever runs — the leak path:ttl=-1(created-at,owners,processed,requeues-count,running,test_failed_count,warnings,worker:1:queue,worker:2:queue).Covered by a new regression test,
test_every_build_key_has_a_ttl, which fails against the previous Lua scripts naming exactly the leaked keys. Fullredis_test.rbsuite: 27 tests, 75 assertions, 0 failures.For the report guard, against a real
rspec-queue --reportprocess with a completed queue:totalevictedmaster-statusevictedThe middle row is the false green this closes.
Notes
Upstream
Shopify/ci-queuehas the same missing expires inreserve.lua, so this is an inherited bug rather than a fork regression. An upstream PR should follow separately.Running the test suite locally on Ruby 3.3 also needs
rexmlin the gem'sGemfile; that is not included here to keep the diff scoped.PE-3857