Repository navigation
fix: recover runs from a dead worker, fence every run write, and keep a cost row for every paid call - #165
Merged
Conversation
…brief_review The flag was set only by scopedPipeline.test.ts and was the one path where research returned researched. The test now approves the brief between two runs, and its fake drops a cost-log query shape no stage sends. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A killed worker left its pipeline-runs row running (refusing later admin runs and its articles forever) and its payload-jobs row processing under the content-pipeline concurrency key (stalling every later run). Runs now hold a lease: heartbeatAt is written when the worker claims the row and every 60 s after. Every write the handler makes to its row is fenced on the status it expects, under a row lock, and a heartbeat that finds the row recovered aborts runPipeline before its next article write. The claim moved inside the try. A new recover-stale-runs task (scheduled queue, every 5 minutes) fails running runs whose heartbeat (or startedAt) is over 15 minutes old and marks the stuck content-run job failed so the concurrency key is free. A job taken but never started is released after the same 15 minutes. The migration also makes pipeline-runs.template nullable for CLI runs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked run `pipeline run` took no lock, so it could overlap another CLI run or the CMS worker on the same articles, and it skipped the setup gate that every CMS run obeys. The CLI now asks runReadinessRefusal (the readiness rule the admin uses) and exits 1 with the same sentence when setup is unfinished. It then registers a `cli` pipeline-runs row under the creation advisory lock, refusing while any run is queued or running, lists the articles it will advance so admin and selected runs refuse them, heartbeats like the worker, and settles the row succeeded/failed on every exit path, including SIGINT/SIGTERM. fetch and report stay unlocked and ungated. ig-e2e.sh seeds --with-brand-voice so the walkthrough passes the gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Slugs came straight from the model and were indexed but not unique, so two published articles could share one: the public route resolves a slug with limit 1, leaving the other unreachable with its canonical pointing at its twin. A beforeValidate hook normalises a slug being set or changed (lower case, a-z0-9 and single hyphens, 80 characters) and appends -2, -3, ... when another article holds it; a purely numeric slug is always suffixed so it cannot shadow the /articles/<id> fallback. An unchanged slug is left alone. The migration renames existing duplicates (later rows get -<id>), turns empty slugs into NULL, then makes the index unique. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows published before the stampPublication hook kept their used-up schedule, so moving one back to approved made the next publish-due tick republish it at once. A data migration nulls publish_at where status is published; down is a no-op because nothing records the old dates. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The suite creates an article with a fixed slug, and articles are never deleted, so with unique slugs a second run gets event-test-article-2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The message also covers a run recovery failed before the worker claimed it, so it names the cause rather than saying "while working". Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The heartbeat timer left up to a minute in which a recovered worker could still advance articles. The stage loop now commits each article write through `commitArticle`, which the CMS backs with the run row lock: the write happens in the same transaction, and only while the row is still this run's and `running`. A refused write writes nothing and stops the run. The run signal now also reaches discovery and every model call, so long stages stop between steps once the heartbeat notices. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… won Recovery decided to release a content job before its queued-to-failed transition ran, so a worker that claimed the run in between kept a running run while recovery failed its job and freed the concurrency key. The run decision is now made under the run row lock, and the job is released in that same transaction only when the run is not running: recovered stale, failed from queued, or already settled. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An error during the initial claim left the row `queued`; the job has no retries, so recovery found neither a running run nor a processing job and every later launch stayed blocked. The handler now records whether it claimed the run, and on a failure before that it fails the row only if it is still `queued`, never another handler's running claim. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Appending `-<id>` to a duplicate could land on a slug another row already held (ids 1, 2, 3 with `guide`, `guide`, `guide-2` gave two `guide-2`), and the unique index then failed. Each replacement is now checked against every slug in the table, including ones the migration has just assigned, and suffixed further until it is free. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MOCK_MODE true/1/yes is mock, false/0/no is live, unset or empty is live only when a provider key is present, and an unparseable value is mock. The rule is in cms/src/lib/runMode.ts and is shared by the pipeline config, readiness, CMS-side model calls and topic discovery. Readiness reports an unrecognised value as a runtime problem. CMS-side calls follow the workspace mode and, live without that model's key, fail naming the key instead of returning a fixture. Brand-voice extraction and setup assist default to the small model of the generate model's provider. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…on and refusal A reply that is truncated, refused, paused, empty or not JSON now throws a typed LlmReplyError carrying provider, model and usage, and completeJSONLogged writes the cost row before rethrowing (a row that cannot be written still stops the stage as CostLogWriteError). The Anthropic path checks stop_reason (max_tokens, refusal, pause_turn), joins the text blocks after the last tool result, and picks the web search tool version by model. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ction When no ranking page was readable the snapshot is empty whatever else is added, so write the empty row and throw before internal-corpus extraction and clustering. The settlement test now gives its internal-corpus case one readable SERP page, since an unreadable one no longer reaches internal extraction. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The small-model default moved Anthropic workspaces from Opus to Haiku for two quality-sensitive drafting calls; the defect was only that an OpenAI-only workspace got a Claude default. Follow the resolved generate model instead (the platform default when no provider serves it), name any abnormal Anthropic stop reason in CMS errors, and correct the docs. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery can fail a run before its heartbeat aborts the signal; discovery then still created topics, outside the run-row lock the stage loop uses. Each topic is now created through the same commit as article writes, and discovery stops when it is refused. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OpenAI returns a refusal as a `refusal` part on the message with empty output text, so it was reported as an empty reply. It now throws a `refusal` reply error that keeps the billed usage. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars.
|
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If a content worker died mid-run, its run stayed
runningand its job stayedprocessing, so every later content run waited forever. The CLI could also run the same articles at the same time as the admin worker, and a misconfiguredMOCK_MODEcould take the whole admin down at import. Some paid model calls that failed left no cost row, so a retry bought the same call again without any record. This PR addresses the open audit findings for runs, run mode and cost.Runs
running. Every article write takes a lock on the run row and happens in the same transaction, including each topic that discovery creates. A stalled worker that comes back after recovery therefore writes nothing more. An abort signal stops model calls and discovery between steps.queued.pipeline runregisters a lockedclirun and refuses to start until setup is complete, using the same rule as the admin.publishAtdates on articles that are already published.pauseForBriefis removed, so research always stops atbrief_review.Mode and cost
MOCK_MODEvalue runs in mock mode and shows up as a readiness problem.claude-sonnet-5is priced at $2/$10.mainsendsweb_search_20260209to every model, which older models such as Haiku reject.Proof: typecheck and lint clean; pipeline 938/938; CMS integration 992/992 on a freshly migrated database after rebasing on
main. A fullmigrate:downthenmigrateround trip is clean, andgenerate:typesproduces no diff. The slug migration was also checked on seeded duplicates (guide,guide,guide-2), which gave unique slugs. Each fix has a test that failed before it.Reviewed with sol-loop: 2 rounds with GPT-6.1 Sol. Sol verdict: approved
Created with Claude Opus 5.5 (implementer subagents Claude Opus 5.5 and Claude Sonnet 5.5) in Claude Code.
🤖 Generated with Claude Code