Skip to content

fix: recover runs from a dead worker, fence every run write, and keep a cost row for every paid call - #165

Merged
snowopsdev merged 20 commits into
mainfrom
fix/runs-mode-cost
Nov 10, 2026
Merged

snowopsdev merged 20 commits into
mainfrom
fix/runs-mode-cost

Conversation

@snowopsdev

Copy link
Copy Markdown
Owner

If a content worker died mid-run, its run stayed running and its job stayed processing, so every later content run waited forever. The CLI could also run the same articles at the same time as the admin worker, and a misconfigured MOCK_MODE could take the whole admin down at import. Some paid model calls that failed left no cost row, so a retry bought the same call again without any record. This PR addresses the open audit findings for runs, run mode and cost.

Runs

  • A running content run refreshes a heartbeat every 60 s. A scheduled task every 5 minutes fails runs whose heartbeat is more than 15 minutes old, and frees the stuck job only if its own update to the run went through.
  • Fencing: after a run claims its row, it writes to that row only while the row still says running. Every article write takes a lock on the run row and happens in the same transaction, including each topic that discovery creates. A stalled worker that comes back after recovery therefore writes nothing more. An abort signal stops model calls and discovery between steps.
  • A run that fails before its worker claims it is marked failed only if it is still queued.
  • pipeline run registers a locked cli run and refuses to start until setup is complete, using the same rule as the admin.
  • Article slugs are normalised and unique. A migration de-duplicates existing rows first, then adds the unique index.
  • A data migration clears stale publishAt dates on articles that are already published.
  • pauseForBrief is removed, so research always stops at brief_review.

Mode and cost

  • One tolerant mock/live rule is shared by the pipeline and the CMS. Nothing throws at import. An unrecognised MOCK_MODE value runs in mock mode and shows up as a readiness problem.
  • A paid reply that cannot be used (not JSON, truncated, refused, or with no text) still writes its cost row, and the error names the cause. This covers OpenAI refusals too.
  • An unreadable SERP stops before the paid corpus extraction.
  • claude-sonnet-5 is priced at $2/$10.
  • Extraction and setup assist default to the generate model.
  • The Anthropic web-search tool version now follows the model. main sends web_search_20260209 to every model, which older models such as Haiku reject.

Proof: typecheck and lint clean; pipeline 938/938; CMS integration 992/992 on a freshly migrated database after rebasing on main. A full migrate:down then migrate round trip is clean, and generate:types produces no diff. The slug migration was also checked on seeded duplicates (guide, guide, guide-2), which gave unique slugs. Each fix has a test that failed before it.

Reviewed with sol-loop: 2 rounds with GPT-6.1 Sol. Sol verdict: approved

Created with Claude Opus 5.5 (implementer subagents Claude Opus 5.5 and Claude Sonnet 5.5) in Claude Code.

🤖 Generated with Claude Code

snowopsdev and others added 20 commits November 10, 2026 01:53
…brief_review

The flag was set only by scopedPipeline.test.ts and was the one path where
research returned researched. The test now approves the brief between two
runs, and its fake drops a cost-log query shape no stage sends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A killed worker left its pipeline-runs row running (refusing later admin
runs and its articles forever) and its payload-jobs row processing under
the content-pipeline concurrency key (stalling every later run).

Runs now hold a lease: heartbeatAt is written when the worker claims the
row and every 60 s after. Every write the handler makes to its row is
fenced on the status it expects, under a row lock, and a heartbeat that
finds the row recovered aborts runPipeline before its next article write.
The claim moved inside the try.

A new recover-stale-runs task (scheduled queue, every 5 minutes) fails
running runs whose heartbeat (or startedAt) is over 15 minutes old and
marks the stuck content-run job failed so the concurrency key is free.
A job taken but never started is released after the same 15 minutes.

The migration also makes pipeline-runs.template nullable for CLI runs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ked run

`pipeline run` took no lock, so it could overlap another CLI run or the
CMS worker on the same articles, and it skipped the setup gate that every
CMS run obeys.

The CLI now asks runReadinessRefusal (the readiness rule the admin uses)
and exits 1 with the same sentence when setup is unfinished. It then
registers a `cli` pipeline-runs row under the creation advisory lock,
refusing while any run is queued or running, lists the articles it will
advance so admin and selected runs refuse them, heartbeats like the
worker, and settles the row succeeded/failed on every exit path,
including SIGINT/SIGTERM. fetch and report stay unlocked and ungated.

ig-e2e.sh seeds --with-brand-voice so the walkthrough passes the gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Slugs came straight from the model and were indexed but not unique, so
two published articles could share one: the public route resolves a
slug with limit 1, leaving the other unreachable with its canonical
pointing at its twin.

A beforeValidate hook normalises a slug being set or changed (lower
case, a-z0-9 and single hyphens, 80 characters) and appends -2, -3, ...
when another article holds it; a purely numeric slug is always suffixed
so it cannot shadow the /articles/<id> fallback. An unchanged slug is
left alone. The migration renames existing duplicates (later rows get
-<id>), turns empty slugs into NULL, then makes the index unique.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rows published before the stampPublication hook kept their used-up
schedule, so moving one back to approved made the next publish-due tick
republish it at once. A data migration nulls publish_at where status is
published; down is a no-op because nothing records the old dates.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The suite creates an article with a fixed slug, and articles are never
deleted, so with unique slugs a second run gets event-test-article-2.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The message also covers a run recovery failed before the worker claimed
it, so it names the cause rather than saying "while working".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The heartbeat timer left up to a minute in which a recovered worker could
still advance articles. The stage loop now commits each article write
through `commitArticle`, which the CMS backs with the run row lock: the
write happens in the same transaction, and only while the row is still
this run's and `running`. A refused write writes nothing and stops the
run. The run signal now also reaches discovery and every model call, so
long stages stop between steps once the heartbeat notices.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… won

Recovery decided to release a content job before its queued-to-failed
transition ran, so a worker that claimed the run in between kept a
running run while recovery failed its job and freed the concurrency key.
The run decision is now made under the run row lock, and the job is
released in that same transaction only when the run is not running:
recovered stale, failed from queued, or already settled.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An error during the initial claim left the row `queued`; the job has no
retries, so recovery found neither a running run nor a processing job
and every later launch stayed blocked. The handler now records whether
it claimed the run, and on a failure before that it fails the row only
if it is still `queued`, never another handler's running claim.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Appending `-<id>` to a duplicate could land on a slug another row
already held (ids 1, 2, 3 with `guide`, `guide`, `guide-2` gave two
`guide-2`), and the unique index then failed. Each replacement is now
checked against every slug in the table, including ones the migration
has just assigned, and suffixed further until it is free.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
MOCK_MODE true/1/yes is mock, false/0/no is live, unset or empty is live only
when a provider key is present, and an unparseable value is mock. The rule is
in cms/src/lib/runMode.ts and is shared by the pipeline config, readiness,
CMS-side model calls and topic discovery. Readiness reports an unrecognised
value as a runtime problem. CMS-side calls follow the workspace mode and, live
without that model's key, fail naming the key instead of returning a fixture.
Brand-voice extraction and setup assist default to the small model of the
generate model's provider.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…on and refusal

A reply that is truncated, refused, paused, empty or not JSON now throws a
typed LlmReplyError carrying provider, model and usage, and completeJSONLogged
writes the cost row before rethrowing (a row that cannot be written still
stops the stage as CostLogWriteError). The Anthropic path checks stop_reason
(max_tokens, refusal, pause_turn), joins the text blocks after the last tool
result, and picks the web search tool version by model.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…ction

When no ranking page was readable the snapshot is empty whatever else is added,
so write the empty row and throw before internal-corpus extraction and
clustering. The settlement test now gives its internal-corpus case one readable
SERP page, since an unreadable one no longer reaches internal extraction.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The small-model default moved Anthropic workspaces from Opus to Haiku for
two quality-sensitive drafting calls; the defect was only that an
OpenAI-only workspace got a Claude default. Follow the resolved generate
model instead (the platform default when no provider serves it), name any
abnormal Anthropic stop reason in CMS errors, and correct the docs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Recovery can fail a run before its heartbeat aborts the signal; discovery
then still created topics, outside the run-row lock the stage loop uses.
Each topic is now created through the same commit as article writes, and
discovery stops when it is refused.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
OpenAI returns a refusal as a `refusal` part on the message with empty
output text, so it was reported as an empty reply. It now throws a
`refusal` reply error that keeps the billed usage.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Nov 10, 2026

Copy link
Copy Markdown

Important

  • Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6c338d03-1da1-4056-b127-a1fca7374370
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-10T06:57:32.748930Z cde431f PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@snowopsdev
snowopsdev merged commit 6fa3330 into main Nov 10, 2026
13 checks passed
@snowopsdev
snowopsdev deleted the fix/runs-mode-cost branch November 10, 2026 07:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant