From ffa8fd4e9556e2aa50f655aa47fb83ce5dcc169b Mon Sep 17 00:00:00 2001 From: GitHub CI Date: Fri, 21 Aug 2026 22:58:54 -0700 Subject: [PATCH 1/7] deploy(vera): Sonnet 5 on the Claude subscription, + an alarm for the silent fallback MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vera's primary lane moves from the gateway to a native Claude subscription: `model.provider: anthropic-oauth` + `model.name: claude-sonnet-5` (adaptive thinking, effort high), with `routing.fallback_models: ["protolabs/smart"]` behind it. Native OAuth bypasses the gateway entirely (ADR 0097); the fallback alias still routes because #2571 lets a namespaced slot name opt out of the native provider, which is why `model.api_base` and the gateway key stay set even though the primary never touches them. The problem with that shape is what happens when the subscription dies. protoAgent wires langchain's `ModelFallbackMiddleware` raw, and that middleware swallows the primary's exception with no log, no counter and no event — verified against core 0.144.0. So the failure mode is not a broken review, it is a verdict written by a different model with nothing anywhere saying so. Filed upstream as protoAgent#2956 (emit a `model.fallback` lifecycle event on the ADR 0074 seam, plus a counter and a warning); until that lands: - `check_model_fallback.py` infers it from gateway metrics. Her primary never reaches the gateway, so a protoAgent-UA chat completion from her container IP IS a fallback. clawpatch shares the key and the container but lands under `user_agent="node"`, which is the whole discriminator. Alarms on growth (the counter is lifetime — a ceiling would latch red forever, the trap check_review_health.py already documents) with a 60m cooldown, so a rate-limit burst is one post rather than twelve. - `check_oauth_health.py` watches the cause instead: signed-in, refreshable, and name/provider coherence (#2623). It deliberately does NOT probe `/api/config/test-model` — that endpoint returned 429 on four consecutive attempts while a real turn on the same credential completed and telemetry recorded `model=claude-sonnet-5`. An alert wired to it would have paged immediately, permanently, for a healthy lane. `vera-watchdog.sh` gains both modes and enters the repo at last — the alerting half of the guard existed only in one box's `~/.local/bin`, which is the same "watchdog living nowhere legible" risk it was written to avoid. Pins move to their latest upstream: core 0.137.1 → 0.144.0, github-plugin v0.3.0 → v0.4.0 (PR lifecycle events on the bus). pr-reviewer v0.35.0 and protoPatch 0.6.1 are already current. Co-Authored-By: Claude Opus 5 (1M context) Claude-Session: https://claude.ai/code/session_01FnbQVnHNvTsvbnJ1pdUDdD --- Dockerfile | 4 +- README.md | 47 ++++++- protoagent.bundle.yaml | 4 +- scripts/check_model_fallback.py | 214 ++++++++++++++++++++++++++++++++ scripts/check_oauth_health.py | 136 ++++++++++++++++++++ scripts/vera-watchdog.sh | 98 +++++++++++++++ 6 files changed, 496 insertions(+), 7 deletions(-) create mode 100755 scripts/check_model_fallback.py create mode 100755 scripts/check_oauth_health.py create mode 100755 scripts/vera-watchdog.sh diff --git a/Dockerfile b/Dockerfile index b0eefe8..9cdea88 100644 --- a/Dockerfile +++ b/Dockerfile @@ -9,7 +9,7 @@ # protoAgent core forward on the same image roll — core and member bumps are # decoupled. Bump this deliberately (and re-verify), keeping it in step with the # manifest's `verified_against`. Tag format is bare semver (no `v` prefix). -FROM ghcr.io/protolabsai/protoagent:0.137.1 +FROM ghcr.io/protolabsai/protoagent:0.144.0 USER root @@ -27,7 +27,7 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \ # Bake the bundle members at their RELEASE TAGS (both public — no build secrets). # The tags mirror protoagent.bundle.yaml's pins; bump both together (the manifest # is the source of truth, this bake is its image form). -ARG GITHUB_PLUGIN_REF=v0.3.0 +ARG GITHUB_PLUGIN_REF=v0.4.0 RUN git clone --depth 1 --branch "${GITHUB_PLUGIN_REF}" \ https://github.com/protoLabsAI/github-plugin.git /opt/protoagent/plugins/github \ && rm -rf /opt/protoagent/plugins/github/.git diff --git a/README.md b/README.md index 99ea2e8..dea4ef1 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ python -m server plugin install https://github.com/protoLabsAI/qaEngineer | Member | Pin | Role | |---|---|---| | `workflows` (builtin) | core | the recipe engine the review panels run on | -| [github-plugin](https://github.com/protoLabsAI/github-plugin) | v0.3.0 | the verdict surface — formal Review API tools with CI-terminal + self-review guards inside the tools | +| [github-plugin](https://github.com/protoLabsAI/github-plugin) | v0.4.0 | the verdict surface — formal Review API tools with CI-terminal + self-review guards inside the tools | | [pr-reviewer-plugin](https://github.com/protoLabsAI/pr-reviewer-plugin) | v0.35.0 | the machinery — webhook chokepoint, structural trigger, panel dispatch, evidence grounding, convergence, approve-on-green sweep, on-demand summon, telemetry + eval | Persona: [`SOUL.md`](./SOUL.md) (Vera — verdict system, three-layer verification, 80% bar, @@ -84,7 +84,27 @@ plus prior-round context. In practice that's **~37k tokens per finder**. - **No aggressive TPM cap.** A free tier at 12,000 tokens/minute rejects a single finder outright. - Routed through a gateway alias, so swapping models is a gateway edit rather than a code - change. Model settings are host-scoped (ADR 0047). + change. Model settings are host-scoped (ADR 0047) — **or** a native subscription lane + (see below), which is not a gateway edit at all. + +**The reference host's lane (since 2026-08-21):** `model.provider: anthropic-oauth` + +`model.name: claude-sonnet-5`, with `routing.fallback_models: ["protolabs/smart"]` behind +it. Native OAuth **bypasses the gateway entirely** (ADR 0097) — the fallback alias is +reachable only because protoAgent#2571 lets a namespaced slot name opt out of the native +provider, so `model.api_base` + a gateway key must stay set even though the primary never +uses them. Set `model.name` and `model.provider` in the SAME config POST: they are one +decision (protoAgent#2623), and a native provider paired with a `protolabs/*` name is +rejected on every call. + +**A fallback is SILENT.** protoAgent wires langchain's `ModelFallbackMiddleware` raw, and +that middleware swallows the primary's exception with no log, no counter and no event — +verified against core 0.144.0, filed as protoAgent#2956. So a dead subscription doesn't +break the review; it quietly changes which model writes the verdict. That is what +`scripts/check_model_fallback.py` exists to catch, by inference from gateway metrics: +protoAgent-UA traffic arriving at the gateway from Vera's container *is* a fallback, +because her primary never goes there. If #2956 lands a `model.fallback` lifecycle event, +delete that script and subscribe to the event — it would be ground truth where this is +an inference. **Reviews are not cheap.** One structural review is nine LLM steps and 5–9 minutes of wall clock. On a hosted frontier model that's roughly $0.12–0.15 each; on local inference @@ -164,7 +184,7 @@ from the payload): ## Deploying Vera (the reference host) This repo doubles as Vera's image source: `Dockerfile` = stock protoAgent (**pinned -base** — `protoagent:0.137.1`, in step with the manifest's `verified_against`; bump +base** — `protoagent:0.144.0`, in step with the manifest's `verified_against`; bump deliberately so a member-pin bump can't drag the core forward on the same roll) + node/`clawpatch` + the bundle members baked at their manifest pins + `deploy/vera.langgraph-config.yaml` (seed, not force) + `SOUL.md`. @@ -191,6 +211,27 @@ static check can't see a *running* instance whose live config drifted; `scripts/check_card_drift.py` is the runtime half — point it at the (tailnet-only) card from the ava fleet cron: `python3 scripts/check_card_drift.py` (exit 1 on drift). +### The watchdogs + +Four checks, all run from the ava fleet cron through `scripts/vera-watchdog.sh`, which is +the piece that makes a failure LOUD (a Discord `#alerts` post, with exit 1 = verdict and +exit 2 = unreachable kept as distinct alarms). Every one of them exists because Vera fails +*quietly* — a starved panel, a drifted card, a swapped model underneath a verdict. + +| Mode | Check | Asks | +|---|---|---| +| `health` | `check_review_health.py` | is the gate still producing verdicts? (growth in unreviewed/exhausted, completion rate) | +| `drift` | `check_card_drift.py` | does the live card still match the seed? | +| `fallback` | `check_model_fallback.py` | did she silently answer from her fallback model? (gateway-metrics inference — protoAgent#2956) | +| `oauth` | `check_oauth_health.py` | is the subscription credential still signed in, refreshable, and coherent with `model.name`? | + +The wrapper runs **installed copies** in `~/.local/bin`, not `scripts/*.py` — this repo is +also the deploy source, so a branch switch would silently disarm a guard that lived inside +it. Re-`install` them after changing a script (there is no auto-update) — the install lines +are in the wrapper's header. It reads `DISCORD_WEBHOOK_ALERTS` from `infisical run`, so an +expired Infisical session downgrades every alarm to a log line nobody reads; that is worth +checking whenever the alerts channel goes quiet for a suspiciously long time. + ## Other orgs Nothing here is protoLabs-specific except the pins and the seed: the bundle installs into diff --git a/protoagent.bundle.yaml b/protoagent.bundle.yaml index 50a4674..2a23c28 100644 --- a/protoagent.bundle.yaml +++ b/protoagent.bundle.yaml @@ -25,11 +25,11 @@ description: >- own work; never posts a blocking verdict against pending CI. # The core version this pin set was last verified against (ADR 0049 rule 2). -verified_against: 0.137.1 +verified_against: 0.144.0 plugins: - { id: workflows, builtin: true } # the recipe engine (code-review panels run through it) - - { id: github, url: https://github.com/protoLabsAI/github-plugin, ref: v0.3.0 } + - { id: github, url: https://github.com/protoLabsAI/github-plugin, ref: v0.4.0 } - { id: pr-reviewer, url: https://github.com/protoLabsAI/pr-reviewer-plugin, ref: v0.35.0 } enabled: [workflows, github, pr-reviewer] # suggested turn-on list (applied to plugins.enabled) diff --git a/scripts/check_model_fallback.py b/scripts/check_model_fallback.py new file mode 100755 index 0000000..74b777c --- /dev/null +++ b/scripts/check_model_fallback.py @@ -0,0 +1,214 @@ +#!/usr/bin/env python3 +"""Health check: is Vera silently answering from her FALLBACK model? + +`routing.fallback_models` is Vera's only degrade path, and when it fires it fires +SILENTLY. protoAgent wires langchain's `ModelFallbackMiddleware` raw (graph/agent.py), +and that middleware swallows the primary's exception with no log, no counter, and no +event — a successful fallback is byte-for-byte indistinguishable from a normal turn. +Verified against core 0.144.0; filed upstream as protoAgent#2956. If that issue lands a +`model.fallback` lifecycle event, DELETE this script and subscribe to the event instead: +this is an inference, and the event would be ground truth. + +THE INFERENCE. Since 2026-08-21 Vera's primary is a NATIVE OAUTH provider +(`model.provider: anthropic-oauth`, claude-sonnet-5), which per ADR 0097 bypasses the +gateway entirely — her subscription traffic never touches LiteLLM. Her fallback is a +gateway alias (`protolabs/smart`), reachable only because #2571 lets a namespaced slot +name opt out of the native provider. So the two lanes are cleanly separable at the +gateway, and the rule is simply: + + a protoAgent-UA chat completion from Vera's container IP == a fallback + +Two things share that gateway key and must NOT be counted: + + * clawpatch (the protoPatch structural engine) — the pr-reviewer plugin exports + OPENAI_API_KEY to the subprocess, and it calls protolabs/smart too. It is a node + process, so it lands under `user_agent="node"` while protoAgent's own calls carry + `user_agent="protoAgent/0.1 (+...)"`. That label is the whole discriminator. + * embeddings (`qwen3-embedding`) — a different route and a different model; knowledge + embeddings are off today but that can be flipped from the console, so filter by + route rather than trusting the config. + +ALARM ON GROWTH, NOT ON DEPTH. `litellm_proxy_total_requests_metric_total` is a +lifetime counter, so a fixed ceiling would latch red forever after the first bad hour — +the same trap check_review_health.py documents. State carries the previous run's total; +a run alarms only on NEW fallback traffic since the last one. + +COOLDOWN, because the loud failure here is not one 429. A subscription that is +rate-limited stays rate-limited for a window, and every review in that window falls +back — alerting per run would post to #alerts every few minutes for an hour and train +everyone to mute the channel. After an alert, stay quiet for --cooldown-min while +still tracking the counter, then re-alarm if it is STILL growing. Sustained degradation +gets through; a burst gets one post. + +Run it from the ava fleet cron (the gateway's metrics port is container-local): + + python3 scripts/check_model_fallback.py --container vera + +Exit 0 = primary lane healthy (or inside a cooldown); exit 1 = fallback traffic since +the last run; exit 2 = could not reach the gateway or the container (an operational +error, NOT a verdict — a dead scraper must not read as a clean lane). +""" + +from __future__ import annotations + +import argparse +import json +import re +import subprocess +import sys +import time +import urllib.error +import urllib.request +from pathlib import Path + +DEFAULT_STATE = Path.home() / ".cache" / "vera-model-fallback.json" +# The gateway publishes on the shared ai_default net; from ava's host namespace it is +# reachable on the published port. Overridable for a different host/stack. +DEFAULT_METRICS = "http://localhost:4000/metrics/" +# protoAgent stamps its own User-Agent on every model call it makes; clawpatch (node) +# and ad-hoc curl do not. This prefix IS the "was it the agent itself" test. +AGENT_UA_PREFIX = "protoAgent" +# Embeddings ride the same key and container but are not a chat lane. +EMBEDDING_ROUTE = "/v1/embeddings" +DEFAULT_COOLDOWN_MIN = 60 + +_SAMPLE = re.compile(r"^litellm_proxy_total_requests_metric_total\{(?P.*)\}\s+(?P[0-9.eE+-]+)$") +_LABEL = re.compile(r'(\w+)="((?:[^"\\]|\\.)*)"') + + +def _container_ip(container: str) -> str: + """Vera's current address on the gateway's network. + + Resolved fresh every run on purpose: a watchtower roll gives her a new IP, and a + hardcoded one would silently stop matching — the counter would flatline and the + check would report a healthy lane forever. That is the exact failure class this + script exists to catch, so it must not reproduce it. + """ + out = subprocess.run( + ["docker", "inspect", container, "--format", "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"], + capture_output=True, + text=True, + timeout=30, + ) + if out.returncode != 0: + raise RuntimeError(f"docker inspect {container} failed: {out.stderr.strip()[:200]}") + ips = out.stdout.split() + if not ips: + raise RuntimeError(f"{container} has no container IP (is it running?)") + return ips[0] + + +def _scrape(url: str) -> str: + try: + with urllib.request.urlopen(url, timeout=20) as resp: + return resp.read().decode("utf-8", "replace") + except (urllib.error.URLError, OSError) as exc: + raise RuntimeError(f"cannot scrape {url}: {exc}") from exc + + +def fallback_requests(metrics_text: str, agent_ip: str) -> tuple[float, dict[str, float]]: + """Total protoAgent-UA gateway chat requests from ``agent_ip``, and the per-model split. + + Pure over the scrape text so the attribution rule is unit-testable without a live + gateway — the rule is the load-bearing part, not the HTTP. + """ + total = 0.0 + by_model: dict[str, float] = {} + for line in metrics_text.splitlines(): + match = _SAMPLE.match(line.strip()) + if not match: + continue + labels = {k: v for k, v in _LABEL.findall(match.group("labels"))} + if labels.get("client_ip") != agent_ip: + continue + if not labels.get("user_agent", "").startswith(AGENT_UA_PREFIX): + continue + if labels.get("route") == EMBEDDING_ROUTE: + continue + value = float(match.group("value")) + total += value + model = labels.get("requested_model", "?") + by_model[model] = by_model.get(model, 0.0) + value + return total, by_model + + +def _load_state(path: Path) -> dict: + try: + return json.loads(path.read_text()) + except (OSError, ValueError): + return {} + + +def _save_state(path: Path, state: dict) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(json.dumps(state, indent=2, sort_keys=True)) + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--container", default="vera") + ap.add_argument("--metrics-url", default=DEFAULT_METRICS) + ap.add_argument("--state", type=Path, default=DEFAULT_STATE) + ap.add_argument("--cooldown-min", type=float, default=DEFAULT_COOLDOWN_MIN) + ap.add_argument("--no-save", action="store_true", help="do not update the stored baseline (test runs)") + args = ap.parse_args() + + try: + agent_ip = _container_ip(args.container) + total, by_model = fallback_requests(_scrape(args.metrics_url), agent_ip) + except Exception as exc: # noqa: BLE001 — every failure here is operational, exit 2 + print(f"UNREACHABLE: {exc}") + return 2 + + state = _load_state(args.state) + previous = state.get("total") + now = time.time() + last_alert = state.get("last_alert_ts", 0.0) + split = ", ".join(f"{m}={int(v)}" for m, v in sorted(by_model.items())) or "(none)" + + new_state = dict(state) + new_state["total"] = total + new_state["checked_at"] = now + new_state["agent_ip"] = agent_ip + new_state["by_model"] = by_model + + verdict = 0 + if previous is None: + print(f"BASELINE: {args.container} @ {agent_ip} — {int(total)} fallback requests so far [{split}]") + print("First run: recorded. Growth is what alarms, so this run cannot.") + elif total < previous: + # The gateway restarted and its counters reset. Re-baseline rather than reading + # the negative delta as "healthy" — and say so, because a silent re-baseline + # across a restart could swallow a real burst. + print(f"COUNTER RESET: gateway counters went {int(previous)} → {int(total)} (restart). Re-baselined.") + else: + delta = total - previous + if delta <= 0: + print(f"OK: no fallback traffic since the last run ({int(total)} lifetime) [{split}]") + else: + cooling = (now - last_alert) < args.cooldown_min * 60 + head = ( + f"{int(delta)} FALLBACK requests since the last run " + f"({int(total)} lifetime) [{split}]" + ) + if cooling: + quiet_for = int((args.cooldown_min * 60 - (now - last_alert)) / 60) + print(f"DEGRADED (cooldown, quiet ~{quiet_for}m more): {head}") + else: + print(f"FALLBACK: {head}") + print( + f"Vera answered {int(delta)} model calls from her fallback lane, not " + "claude-sonnet-5. Check the subscription: " + "`curl -X POST localhost:7870/api/config/test-model` in the container " + "(401/403 = re-auth needed, 429 = rate-limited, it will pass)." + ) + new_state["last_alert_ts"] = now + verdict = 1 + + if not args.no_save: + _save_state(args.state, new_state) + return verdict + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/check_oauth_health.py b/scripts/check_oauth_health.py new file mode 100755 index 0000000..c6a0949 --- /dev/null +++ b/scripts/check_oauth_health.py @@ -0,0 +1,136 @@ +#!/usr/bin/env python3 +"""Health check: is Vera's Claude subscription credential still one she can use? + +The companion to check_model_fallback.py, and deliberately a different question. That +script asks "did she already degrade?" (observed, after the fact). This one asks "is the +thing that would make her degrade still sound?" (the cause, before the fact). Both are +needed: the credential can be perfect and the lane still fall back on a 529, and the +credential can be dead for hours on an idle agent with no traffic to reveal it. + +WHY NOT JUST CALL /api/config/test-model. Because it lies in the direction that matters. +It streams a real 1-token turn through the subscription, which sounds like the perfect +liveness probe — but measured on 2026-08-21 it returned `429 rate_limit_error` on four +consecutive attempts WHILE a real A2A turn on the same credential completed fine and +telemetry recorded `model=claude-sonnet-5`. Wiring the alert to that probe would have +paged #alerts immediately and permanently, for a lane that was working. A monitor whose +false-positive rate is 100% on day one is worse than no monitor. So this check reads +STATE, not liveness, and leaves "did it actually degrade" to the fallback detector, +which is grounded in traffic that really happened. + +What it checks, all from `/api/config` + `/api/config/oauth-status` (core ≥0.137.1, +where protoAgent#2564 started publishing expiry and refreshability): + + * signed_in — the credential is gone or was disconnected. Every call raises; the + fallback lane carries 100% of her traffic. This is the "OAuth expired" case. + * refreshable — a credential that cannot refresh is a deadline, not a credential. + This is also the CLAUDE_CODE_OAUTH_TOKEN trap: that env path is never refreshed and + never inspectable, and reads `signed_in: true` right up until it 401s. + * provider/name coherence — protoAgent#2623 made model.name and model.provider ONE + decision; a native-OAuth provider with a namespaced name ("protolabs/smart") is + rejected by the native builder on every call. That config is silently fatal, and it + is exactly what a careless half-edit of the model config produces. + * expires_at — reported always, alarmed on only when the credential is NOT + refreshable. Refresh is ON USE, so a healthy busy agent legitimately sits near its + expiry all day; alarming on proximity alone would cry wolf every few hours. + +Run from the ava fleet cron: + + python3 scripts/check_oauth_health.py --container vera + +Exit 0 = credential sound; exit 1 = a real problem (prints which); exit 2 = could not +reach the agent (operational, NOT a verdict). +""" + +from __future__ import annotations + +import argparse +import json +import subprocess +import sys +import time + +# Native-OAuth providers (ADR 0097). A gateway-backed agent has no credential of its own +# to check — this whole script is a no-op for one, and says so rather than passing mutely. +NATIVE_PROVIDERS = {"anthropic-oauth", "openai-codex"} + + +def _api(container: str, path: str) -> dict: + cmd = f'curl -s -m 25 -H "Authorization: Bearer $A2A_AUTH_TOKEN" localhost:7870{path}' + out = subprocess.run( + ["docker", "exec", container, "sh", "-c", cmd], capture_output=True, text=True, timeout=60 + ) + if out.returncode != 0: + raise RuntimeError(f"docker exec failed for {path}: {out.stderr.strip()[:200]}") + return json.loads(out.stdout) + + +def evaluate(model_cfg: dict, providers: list[dict]) -> tuple[int, list[str]]: + """(exit_code, report lines). Pure over the two API bodies so the rules are testable.""" + lines: list[str] = [] + provider = (model_cfg.get("provider") or "").strip().lower() + name = (model_cfg.get("name") or "").strip() + + if provider not in NATIVE_PROVIDERS: + return 0, [f"OK: model.provider={provider!r} is not a native OAuth lane — no credential to check."] + + lines.append(f"lane: {provider} · {name}") + problems: list[str] = [] + + # #2623 — the two halves are one decision, and a mismatched pair fails every call. + if "/" in name: + problems.append( + f"model.name={name!r} is a gateway alias but model.provider={provider!r} is native — " + "the native builder rejects any name containing '/', so EVERY call raises and the " + "fallback lane is carrying all traffic. Set model.name and model.provider together." + ) + + status = next((p for p in providers if (p.get("provider") or "").lower() == provider), None) + if status is None: + problems.append(f"/api/config/oauth-status reports nothing for {provider!r}") + return 1, lines + problems + + if not status.get("signed_in"): + problems.append( + f"NOT SIGNED IN ({status.get('detail') or 'no detail'}) — re-auth with " + f"POST /api/config/oauth/start {{\"provider\":\"{provider}\"}} and approve on any device." + ) + else: + source = status.get("source") or "?" + durability = status.get("durability") or "?" + lines.append(f"signed in · source={source} · durability={durability}") + if not status.get("refreshable"): + problems.append( + f"credential is NOT refreshable (source={source}) — it is a deadline, not a " + "credential. If this is CLAUDE_CODE_OAUTH_TOKEN, drop the env var and sign in " + "through /api/config/oauth/start so protoAgent owns a refreshing copy." + ) + expires_at = status.get("expires_at") + if expires_at: + remaining_h = (float(expires_at) - time.time()) / 3600.0 + when = time.strftime("%Y-%m-%d %H:%M", time.localtime(float(expires_at))) + lines.append(f"access token expires {when} ({remaining_h:+.1f}h) — refreshed on use") + + return (1 if problems else 0), lines + problems + + +def main() -> int: + ap = argparse.ArgumentParser() + ap.add_argument("--container", default="vera") + args = ap.parse_args() + + try: + model_cfg = _api(args.container, "/api/config").get("config", {}).get("model", {}) + providers = _api(args.container, "/api/config/oauth-status").get("providers", []) + except Exception as exc: # noqa: BLE001 — operational, exit 2 + print(f"UNREACHABLE: {exc}") + return 2 + + code, lines = evaluate(model_cfg, providers) + print(("FAIL: " if code else "OK: ") + lines[0]) + for line in lines[1:]: + print(f" {line}") + return code + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/vera-watchdog.sh b/scripts/vera-watchdog.sh new file mode 100755 index 0000000..749e08f --- /dev/null +++ b/scripts/vera-watchdog.sh @@ -0,0 +1,98 @@ +#!/usr/bin/env bash +# Runner + alerter for Vera's two watchdogs (qaEngineer#37). +# +# WHY THIS FILE EXISTS AT ALL, AND WHY IT LIVES HERE: +# +# Both checks shipped in qaEngineer/scripts/ with docstrings saying "run from the ava +# fleet cron" — and then nothing ever ran them. The review-health alarm was written +# precisely because 24 PRs merged with no review and the only record was an inbox +# nobody read; leaving that alarm unscheduled reproduced the same failure one level up. +# +# It runs the INSTALLED copies in ~/.local/bin, NOT qaEngineer/scripts/*.py: the repo +# working tree is also the deploy source, so a branch switch deletes/reverts the +# in-repo script and the guard stops silently. That is not hypothetical — the same +# trap cost config-drift.sh ~5h of silent failure on 2026-08-10, and this repo sat on +# a feature branch for part of 2026-08-17 during the 0.137.1 bump. A watchdog must not +# live inside the thing it watches. Refresh these copies when the repo version changes: +# +# install -m 755 ~/dev/qaEngineer/scripts/check_review_health.py ~/.local/bin/vera-review-health.py +# install -m 755 ~/dev/qaEngineer/scripts/check_card_drift.py ~/.local/bin/vera-card-drift.py +# install -m 755 ~/dev/qaEngineer/scripts/check_model_fallback.py ~/.local/bin/vera-model-fallback.py +# install -m 755 ~/dev/qaEngineer/scripts/check_oauth_health.py ~/.local/bin/vera-oauth-health.py +# install -m 755 ~/dev/qaEngineer/scripts/vera-watchdog.sh ~/.local/bin/vera-watchdog.sh +# +# THIS FILE is the canonical copy (it was unversioned until 2026-08-21 — the alerting +# half of the guard living nowhere but one box's ~/.local/bin is its own quiet risk). +# The installed copy is still what cron runs, for the branch-trap reason above. +# +# A failing check must be LOUD. Exit 1 (a real verdict) and exit 2 (couldn't reach the +# agent) are deliberately different alerts — the scripts draw that line on purpose, and +# collapsing it would let an outage read as a clean gate. +# +# Usage: vera-watchdog.sh health|drift|fallback|oauth [extra args passed to the check] +# Exit: passes the underlying check's exit code through (0 ok, 1 verdict, 2 unreachable) + +set -uo pipefail + +REPO="${VERA_WATCHDOG_REPO:-$HOME/dev/qaEngineer}" +BIN="$HOME/.local/bin" +MODE="${1:-}"; shift || true # remaining args pass through to the underlying check + +alert() { # alert <body> + local hook="${DISCORD_WEBHOOK_ALERTS:-}" + if [ -z "$hook" ]; then + echo "vera-watchdog: DISCORD_WEBHOOK_ALERTS unset — run via infisical; NOT alerted" >&2 + return + fi + local body + body="$(printf '**%s** on %s\n```\n%s\n```' "$1" "$(hostname)" "$2")" + python3 -c 'import json,sys;print(json.dumps({"content":sys.argv[1][:1900]}))' "$body" \ + | curl -sf -X POST -H 'Content-Type: application/json' -d @- "$hook" >/dev/null \ + && echo "vera-watchdog: alerted Discord" >&2 \ + || echo "vera-watchdog: Discord post FAILED" >&2 +} + +case "$MODE" in + health) + out="$("$BIN/vera-review-health.py" --container vera "$@" 2>&1)"; rc=$? + ;; + drift) + # The seed is the reference this check compares the live card against, so it must + # come from the repo — but reading the WORKING TREE would reintroduce the branch + # trap through the back door (a feature branch's seed is not the deployed one). + # Read it out of the committed main ref instead: always the deployed truth, and + # immune to whatever the tree is currently checked out at. + seed="$(mktemp)"; trap 'rm -f "$seed"' EXIT + if ! git -C "$REPO" show main:deploy/vera.langgraph-config.yaml >"$seed" 2>/dev/null; then + out="cannot read deploy/vera.langgraph-config.yaml from main in $REPO"; rc=2 + else + out="$("$BIN/vera-card-drift.py" --seed "$seed" "$@" 2>&1)"; rc=$? + fi + ;; + fallback) + # Did Vera silently answer from her FALLBACK model? Her primary is a native-OAuth + # Claude subscription, which bypasses the gateway entirely (ADR 0097) — so + # protoAgent-UA traffic arriving AT the gateway from her container is, by + # construction, a fallback. protoAgent emits nothing when this happens (langchain's + # ModelFallbackMiddleware swallows the primary's exception without so much as a log + # line; filed as protoAgent#2956), so an inference from gateway metrics is the only + # signal there is until that issue lands a `model.fallback` event. + out="$("$BIN/vera-model-fallback.py" --container vera "$@" 2>&1)"; rc=$? + ;; + oauth) + # Is the subscription credential itself still sound (signed in, refreshable, and + # coherent with model.name)? The CAUSE side of the same failure — an idle agent can + # outlive its refresh token with no traffic to reveal it. + out="$("$BIN/vera-oauth-health.py" --container vera "$@" 2>&1)"; rc=$? + ;; + *) + echo "usage: $(basename "$0") health|drift|fallback|oauth" >&2; exit 64 ;; +esac + +echo "$out" +case "$rc" in + 0) ;; + 1) alert "vera $MODE check FAILED" "$out" ;; + *) alert "vera $MODE check UNREACHABLE (exit $rc)" "$out" ;; +esac +exit "$rc" From fae1873135bfa31361c492c42d3c4ae4f731a032 Mon Sep 17 00:00:00 2001 From: GitHub CI <ci@example.com> Date: Fri, 21 Aug 2026 23:06:39 -0700 Subject: [PATCH 2/7] fix(watchdogs): share the operator-API helper, and test the rules that page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Both findings from Vera's own panel on this PR (WARN, 2 minor, both confirmed against the head diff) — she reviewed it on the new claude-sonnet-5 lane, which is a decent first look at the lane's output. - `_api()` was a byte-for-byte copy of check_review_health.py's helper. Two copies of the auth/timeout/error handling for one endpoint is two places to fix when the port or the bearer moves, and the second copy is the one that gets missed. Now `scripts/vera_api.py`, flat and installed alongside — cron runs the copies in ~/.local/bin, and Python puts a script's own dir on sys.path[0], so a flat module imports cleanly there. It MUST be installed too; the header and README say so. - Both scripts advertised their core functions as "deliberately pure so the rule is testable" and shipped no tests. The rule is the load-bearing part: an attribution bug in fallback_requests() is either a silent degrade nobody hears about or a pager that cries wolf, and neither shows up in a smoke run against a healthy container. 13 stdlib-unittest cases now cover it — clawpatch's `user_agent="node"` (the whole discriminator), other containers' traffic, embeddings, counter summing, and every verdict branch of evaluate() including "expired but refreshable is not an alarm". CI runs them; stdlib because this CI is python3 + PyYAML and a watchdog suite earning a dependency install is the wrong trade. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FnbQVnHNvTsvbnJ1pdUDdD --- .github/workflows/ci.yml | 9 ++ README.md | 6 +- scripts/check_oauth_health.py | 18 +--- scripts/check_review_health.py | 18 +--- scripts/vera-watchdog.sh | 1 + scripts/vera_api.py | 50 +++++++++++ tests/test_watchdog_checks.py | 151 +++++++++++++++++++++++++++++++++ 7 files changed, 224 insertions(+), 29 deletions(-) create mode 100644 scripts/vera_api.py create mode 100644 tests/test_watchdog_checks.py diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a6786a3..4df6ec8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -21,6 +21,15 @@ jobs: assert m["archetype"]["soul"].strip(), "archetype soul present" print("manifest ok:", m["id"], "-", len(m["plugins"]), "members") PY + - name: Watchdog check logic + # The attribution rule in check_model_fallback.py and the verdict rules in + # check_oauth_health.py decide whether #alerts gets woken — a bug there is + # either a silent degrade nobody hears about or a pager that cries wolf, and + # neither shows up in a smoke run against a healthy container. Stdlib + # unittest: CI here is python3 + PyYAML, and a watchdog suite earning a + # dependency install is the wrong trade. + run: python3 -m unittest discover tests -v + - name: README pins match bundle manifest and Dockerfile # Prevents the member-table and base-image prose from drifting out of sync # with the canonical pins in protoagent.bundle.yaml and the Dockerfile FROM line. diff --git a/README.md b/README.md index dea4ef1..b5d4990 100644 --- a/README.md +++ b/README.md @@ -228,7 +228,11 @@ exit 2 = unreachable kept as distinct alarms). Every one of them exists because The wrapper runs **installed copies** in `~/.local/bin`, not `scripts/*.py` — this repo is also the deploy source, so a branch switch would silently disarm a guard that lived inside it. Re-`install` them after changing a script (there is no auto-update) — the install lines -are in the wrapper's header. It reads `DISCORD_WEBHOOK_ALERTS` from `infisical run`, so an +are in the wrapper's header. That includes **`scripts/vera_api.py`**, the shared +operator-API helper: Python puts a script's own directory on `sys.path[0]`, so a flat +module installed alongside imports cleanly — and a check installed *without* it dies on +ImportError. `python3 -m unittest discover tests` covers the attribution and verdict rules +(CI runs it). It reads `DISCORD_WEBHOOK_ALERTS` from `infisical run`, so an expired Infisical session downgrades every alarm to a log line nobody reads; that is worth checking whenever the alerts channel goes quiet for a suspiciously long time. diff --git a/scripts/check_oauth_health.py b/scripts/check_oauth_health.py index c6a0949..74afb9f 100755 --- a/scripts/check_oauth_health.py +++ b/scripts/check_oauth_health.py @@ -44,26 +44,16 @@ from __future__ import annotations import argparse -import json -import subprocess import sys import time +from vera_api import operator_api_get + # Native-OAuth providers (ADR 0097). A gateway-backed agent has no credential of its own # to check — this whole script is a no-op for one, and says so rather than passing mutely. NATIVE_PROVIDERS = {"anthropic-oauth", "openai-codex"} -def _api(container: str, path: str) -> dict: - cmd = f'curl -s -m 25 -H "Authorization: Bearer $A2A_AUTH_TOKEN" localhost:7870{path}' - out = subprocess.run( - ["docker", "exec", container, "sh", "-c", cmd], capture_output=True, text=True, timeout=60 - ) - if out.returncode != 0: - raise RuntimeError(f"docker exec failed for {path}: {out.stderr.strip()[:200]}") - return json.loads(out.stdout) - - def evaluate(model_cfg: dict, providers: list[dict]) -> tuple[int, list[str]]: """(exit_code, report lines). Pure over the two API bodies so the rules are testable.""" lines: list[str] = [] @@ -119,8 +109,8 @@ def main() -> int: args = ap.parse_args() try: - model_cfg = _api(args.container, "/api/config").get("config", {}).get("model", {}) - providers = _api(args.container, "/api/config/oauth-status").get("providers", []) + model_cfg = operator_api_get(args.container, "/api/config").get("config", {}).get("model", {}) + providers = operator_api_get(args.container, "/api/config/oauth-status").get("providers", []) except Exception as exc: # noqa: BLE001 — operational, exit 2 print(f"UNREACHABLE: {exc}") return 2 diff --git a/scripts/check_review_health.py b/scripts/check_review_health.py index 32cecf8..1f35e2f 100755 --- a/scripts/check_review_health.py +++ b/scripts/check_review_health.py @@ -36,10 +36,11 @@ import argparse import json -import subprocess import sys from pathlib import Path +from vera_api import operator_api_get + # New PRs that exhausted and have no verdict SINCE THE LAST RUN. Not zero because the # backfill sweep recovers some on a delay — protoAgent#2546 sat unreviewed for ~35min # before a later pass posted — so a run landing inside that window sees a transient +1. @@ -55,17 +56,6 @@ DEFAULT_STATE = Path.home() / ".cache" / "vera-review-health.json" -def _api(container: str, path: str) -> dict: - """Read a token-gated operator-API endpoint from inside the container.""" - cmd = f'curl -s -m 25 -H "Authorization: Bearer $A2A_AUTH_TOKEN" localhost:7870{path}' - out = subprocess.run( - ["docker", "exec", container, "sh", "-c", cmd], capture_output=True, text=True, timeout=60 - ) - if out.returncode != 0: - raise RuntimeError(f"docker exec failed for {path}: {out.stderr.strip()[:200]}") - return json.loads(out.stdout) - - def _load_state(path: Path) -> dict: """Previous run's counters. A missing or corrupt file is not an error — it means 'no baseline yet', which suppresses the growth checks rather than failing the run.""" @@ -110,8 +100,8 @@ def main() -> int: args = ap.parse_args() try: - report = _api(args.container, "/api/plugins/pr-reviewer/eval") - inbox = _api(args.container, "/api/inbox") + report = operator_api_get(args.container, "/api/plugins/pr-reviewer/eval") + inbox = operator_api_get(args.container, "/api/inbox") except Exception as exc: # noqa: BLE001 — any failure here is operational, not a verdict print(f"UNREACHABLE: {exc}", file=sys.stderr) return 2 diff --git a/scripts/vera-watchdog.sh b/scripts/vera-watchdog.sh index 749e08f..b56ea32 100755 --- a/scripts/vera-watchdog.sh +++ b/scripts/vera-watchdog.sh @@ -15,6 +15,7 @@ # a feature branch for part of 2026-08-17 during the 0.137.1 bump. A watchdog must not # live inside the thing it watches. Refresh these copies when the repo version changes: # +# install -m 644 ~/dev/qaEngineer/scripts/vera_api.py ~/.local/bin/vera_api.py # install -m 755 ~/dev/qaEngineer/scripts/check_review_health.py ~/.local/bin/vera-review-health.py # install -m 755 ~/dev/qaEngineer/scripts/check_card_drift.py ~/.local/bin/vera-card-drift.py # install -m 755 ~/dev/qaEngineer/scripts/check_model_fallback.py ~/.local/bin/vera-model-fallback.py diff --git a/scripts/vera_api.py b/scripts/vera_api.py new file mode 100644 index 0000000..121ce74 --- /dev/null +++ b/scripts/vera_api.py @@ -0,0 +1,50 @@ +"""The one way these checks talk to Vera's operator API. + +Extracted because `check_oauth_health.py` shipped a byte-for-byte copy of +`check_review_health.py`'s helper — caught by Vera's own panel reviewing the PR that +added it (qaEngineer#45, minor/conventions, confirmed). Two copies of the auth, timeout +and error-handling for the same endpoint is two places to fix when the port moves or the +bearer changes, and the second copy is the one that gets missed. + +WHY IT LIVES BESIDE THE SCRIPTS AND NOT IN A PACKAGE: cron runs INSTALLED copies out of +`~/.local/bin` (the repo tree is also the deploy source, so a branch switch would +silently disarm a guard living inside it). Python puts a script's own directory on +`sys.path[0]`, so a flat module installed alongside imports cleanly — but it MUST be +installed alongside, or every check dies on ImportError: + + install -m 644 ~/dev/qaEngineer/scripts/vera_api.py ~/.local/bin/vera_api.py + +The operator API is container-local and token-gated, so the call shape is fixed: exec +into the container and read the bearer from its own environment. There is no host-side +credential to leak or expire — which is the point. +""" + +from __future__ import annotations + +import json +import subprocess + +DEFAULT_PORT = 7870 +CURL_TIMEOUT_S = 25 +EXEC_TIMEOUT_S = 60 + + +def operator_api_get(container: str, path: str, *, port: int = DEFAULT_PORT) -> dict: + """GET a token-gated operator-API endpoint from inside ``container``. + + Raises ``RuntimeError`` when the container cannot be reached — callers turn that + into exit 2 (operational), never into a health verdict. + """ + cmd = ( + f'curl -s -m {CURL_TIMEOUT_S} -H "Authorization: Bearer $A2A_AUTH_TOKEN" ' + f"localhost:{port}{path}" + ) + out = subprocess.run( + ["docker", "exec", container, "sh", "-c", cmd], + capture_output=True, + text=True, + timeout=EXEC_TIMEOUT_S, + ) + if out.returncode != 0: + raise RuntimeError(f"docker exec failed for {path}: {out.stderr.strip()[:200]}") + return json.loads(out.stdout) diff --git a/tests/test_watchdog_checks.py b/tests/test_watchdog_checks.py new file mode 100644 index 0000000..e5025a5 --- /dev/null +++ b/tests/test_watchdog_checks.py @@ -0,0 +1,151 @@ +"""Tests for the two watchdog checks whose logic decides whether #alerts gets woken. + +Vera's panel flagged their absence on qaEngineer#45 (minor/tests, confirmed): both +scripts document their core functions as "deliberately pure so the rule is testable", +and then shipped no tests. The rule IS the load-bearing part — an attribution bug in +`fallback_requests` means either a silent degrade nobody hears about or a pager that +cries wolf, and neither shows up in a smoke run against a healthy container. + +Stdlib unittest on purpose: CI here is python3 with PyYAML and nothing else, and a +watchdog's test suite earning a dependency install is the wrong trade. + + python3 -m unittest discover tests -v +""" + +from __future__ import annotations + +import sys +import time +import unittest +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "scripts")) + +from check_model_fallback import fallback_requests # noqa: E402 +from check_oauth_health import evaluate # noqa: E402 + +VERA_IP = "10.0.14.6" +AGENT_UA = "protoAgent/0.1 (+https://github.com/protoLabsAI/protoAgent)" + + +def sample(**labels) -> str: + """One `litellm_proxy_total_requests_metric_total` line, in the gateway's real shape.""" + value = labels.pop("value", 1.0) + base = { + "api_key_alias": "studio-gw-75bce515", + "client_ip": VERA_IP, + "requested_model": "protolabs/smart", + "route": "/v1/chat/completions", + "status_code": "200", + "user_agent": AGENT_UA, + } + base.update(labels) + rendered = ",".join(f'{k}="{v}"' for k, v in base.items()) + return f"litellm_proxy_total_requests_metric_total{{{rendered}}} {value}" + + +class FallbackAttribution(unittest.TestCase): + """The rule: a protoAgent-UA chat completion from Vera's IP IS a fallback.""" + + def test_counts_agent_traffic_from_vera(self): + total, by_model = fallback_requests(sample(value=7.0), VERA_IP) + self.assertEqual(total, 7.0) + self.assertEqual(by_model, {"protolabs/smart": 7.0}) + + def test_ignores_clawpatch(self): + # clawpatch shares the gateway key, the container AND the model alias — the + # user_agent is the only thing separating it from a real fallback. Miscounting + # it would report a permanent degrade on a perfectly healthy lane. + total, _ = fallback_requests(sample(user_agent="node", value=94.0), VERA_IP) + self.assertEqual(total, 0.0) + + def test_ignores_other_containers(self): + # Fleet peers share the gateway and the key; only the IP tells them apart. + total, _ = fallback_requests(sample(client_ip="10.0.14.14", value=171.0), VERA_IP) + self.assertEqual(total, 0.0) + + def test_ignores_embeddings(self): + # Same UA, same container, not a model-lane fallback. + line = sample(route="/v1/embeddings", requested_model="qwen3-embedding", value=45.0) + total, _ = fallback_requests(line, VERA_IP) + self.assertEqual(total, 0.0) + + def test_sums_across_models_and_skips_noise(self): + text = "\n".join( + [ + "# HELP litellm_proxy_total_requests_metric_total noise", + sample(value=3.0), + sample(requested_model="protolabs/cloud", value=2.0), + sample(user_agent="node", value=99.0), + "litellm_something_else_total{foo=\"bar\"} 5.0", + ] + ) + total, by_model = fallback_requests(text, VERA_IP) + self.assertEqual(total, 5.0) + self.assertEqual(by_model, {"protolabs/smart": 3.0, "protolabs/cloud": 2.0}) + + def test_empty_scrape_is_zero_not_an_error(self): + # A gateway that just restarted serves no samples yet; that is "nothing to + # report", not an alarm. + self.assertEqual(fallback_requests("", VERA_IP), (0.0, {})) + + +def oauth_status(**over) -> list[dict]: + base = { + "provider": "anthropic-oauth", + "signed_in": True, + "refreshable": True, + "source": "instance_store", + "expires_at": time.time() + 3600, + } + base.update(over) + return [base] + + +class OAuthHealth(unittest.TestCase): + NATIVE = {"provider": "anthropic-oauth", "name": "claude-sonnet-5"} + + def test_healthy_lane_passes(self): + code, lines = evaluate(self.NATIVE, oauth_status()) + self.assertEqual(code, 0) + self.assertTrue(any("claude-sonnet-5" in line for line in lines)) + + def test_signed_out_fails(self): + code, lines = evaluate(self.NATIVE, oauth_status(signed_in=False, detail="disconnected")) + self.assertEqual(code, 1) + self.assertIn("NOT SIGNED IN", lines[-1]) + + def test_unrefreshable_credential_fails(self): + # The CLAUDE_CODE_OAUTH_TOKEN trap: reads signed_in right up until it 401s. + code, lines = evaluate(self.NATIVE, oauth_status(refreshable=False, source="env")) + self.assertEqual(code, 1) + self.assertIn("not refreshable", lines[-1].lower()) + + def test_incoherent_provider_and_name_fails(self): + # protoAgent#2623: one decision, two fields. A gateway alias under a native + # provider is rejected on every call — silently fatal, and exactly what a + # half-edited model config produces. + code, lines = evaluate({"provider": "anthropic-oauth", "name": "protolabs/smart"}, oauth_status()) + self.assertEqual(code, 1) + self.assertIn("gateway alias", lines[-1]) + + def test_missing_status_entry_fails(self): + code, lines = evaluate(self.NATIVE, []) + self.assertEqual(code, 1) + self.assertIn("reports nothing", lines[-1]) + + def test_gateway_backed_agent_is_a_noop(self): + # No subscription, no credential to check — must pass, and say why. + code, lines = evaluate({"provider": "openai", "name": "protolabs/cloud"}, []) + self.assertEqual(code, 0) + self.assertIn("not a native OAuth lane", lines[0]) + + def test_expired_but_refreshable_is_not_an_alarm(self): + # Refresh is ON USE, so a busy agent legitimately sits at or past its access + # token's expiry. Alarming on proximity would cry wolf every few hours. + code, _ = evaluate(self.NATIVE, oauth_status(expires_at=time.time() - 60)) + self.assertEqual(code, 0) + + +if __name__ == "__main__": + unittest.main() From af2479f58049c660ede34ac0d8737c26b140cf61 Mon Sep 17 00:00:00 2001 From: GitHub CI <ci@example.com> Date: Sat, 22 Aug 2026 16:34:47 -0700 Subject: [PATCH 3/7] =?UTF-8?q?deploy(vera):=20pin=20core=200.145.0=20+=20?= =?UTF-8?q?github-plugin=20v0.5.0=20=E2=80=94=20the=20fallback=20is=20no?= =?UTF-8?q?=20longer=20silent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Upstream shipped all three gaps this PR filed, within hours, in 0.145.0: - **#2956** — `ObservableModelFallbackMiddleware` replaces the raw langchain middleware at the same call site: a WARNING naming the primary failure and the serving fallback, plus a `model.fallback` bus event (ADR 0039) carrying the primary exception class, fallback model and index. GraphBubbleUp now propagates untouched instead of triggering fallback retries, and when every fallback fails the PRIMARY exception is re-raised rather than the last one. Verified present in the 0.145.0 image before pinning. - **#2957** — the OAuth "Test connection" probe now sends the Claude Code identity prefix, so it stops reporting a working subscription as rate-limited. - **#2958** — `secrets_manager.host` tolerates the CLI's `/api` suffix, which removes the sharp edge #46 documents. 0.145.0 also carries #2950, which is directly relevant to the exhaustion burst Vera hit at 09:18Z today: sweep harvests now space themselves with jittered gaps because back-to-back model calls were "manufacturing rate-limit bursts" on a shared OAuth account. Three protoAgent PRs merged with no verdict in that window. Once this rolls, `scripts/check_model_fallback.py` should be RETIRED in favour of subscribing to `model.fallback` — its own docstring says so, and an event is ground truth where the gateway-metrics rule is an inference. Leaving it in place for now: it costs nothing, and the event needs to be seen working first. github-plugin v0.5.0: the ADR 0095 projects registry is ADDED to an explicit `github.repos` list rather than hidden by it. pr-reviewer stays v0.35.0 — the fix for pr-reviewer-plugin#84 (replay of a merged PR silently returning PASS) is merged on main but not yet tagged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FnbQVnHNvTsvbnJ1pdUDdD --- Dockerfile | 4 ++-- README.md | 6 +++--- protoagent.bundle.yaml | 4 ++-- 3 files changed, 7 insertions(+), 7 deletions(-) diff --git a/Dockerfile b/Dockerfile index 9cdea88..af34e8c 100644 --- a/Dockerfile +++ b/Dockerfile @@ -9,7 +9,7 @@ # protoAgent core forward on the same image roll — core and member bumps are # decoupled. Bump this deliberately (and re-verify), keeping it in step with the # manifest's `verified_against`. Tag format is bare semver (no `v` prefix). -FROM ghcr.io/protolabsai/protoagent:0.144.0 +FROM ghcr.io/protolabsai/protoagent:0.145.0 USER root @@ -27,7 +27,7 @@ RUN curl -fsSL https://deb.nodesource.com/setup_22.x | bash - \ # Bake the bundle members at their RELEASE TAGS (both public — no build secrets). # The tags mirror protoagent.bundle.yaml's pins; bump both together (the manifest # is the source of truth, this bake is its image form). -ARG GITHUB_PLUGIN_REF=v0.4.0 +ARG GITHUB_PLUGIN_REF=v0.5.0 RUN git clone --depth 1 --branch "${GITHUB_PLUGIN_REF}" \ https://github.com/protoLabsAI/github-plugin.git /opt/protoagent/plugins/github \ && rm -rf /opt/protoagent/plugins/github/.git diff --git a/README.md b/README.md index b5d4990..ca20b7c 100644 --- a/README.md +++ b/README.md @@ -14,7 +14,7 @@ python -m server plugin install https://github.com/protoLabsAI/qaEngineer | Member | Pin | Role | |---|---|---| | `workflows` (builtin) | core | the recipe engine the review panels run on | -| [github-plugin](https://github.com/protoLabsAI/github-plugin) | v0.4.0 | the verdict surface — formal Review API tools with CI-terminal + self-review guards inside the tools | +| [github-plugin](https://github.com/protoLabsAI/github-plugin) | v0.5.0 | the verdict surface — formal Review API tools with CI-terminal + self-review guards inside the tools | | [pr-reviewer-plugin](https://github.com/protoLabsAI/pr-reviewer-plugin) | v0.35.0 | the machinery — webhook chokepoint, structural trigger, panel dispatch, evidence grounding, convergence, approve-on-green sweep, on-demand summon, telemetry + eval | Persona: [`SOUL.md`](./SOUL.md) (Vera — verdict system, three-layer verification, 80% bar, @@ -98,7 +98,7 @@ rejected on every call. **A fallback is SILENT.** protoAgent wires langchain's `ModelFallbackMiddleware` raw, and that middleware swallows the primary's exception with no log, no counter and no event — -verified against core 0.144.0, filed as protoAgent#2956. So a dead subscription doesn't +verified against core 0.144.0 — FIXED in 0.145.0 (#2956), filed as protoAgent#2956. So a dead subscription doesn't break the review; it quietly changes which model writes the verdict. That is what `scripts/check_model_fallback.py` exists to catch, by inference from gateway metrics: protoAgent-UA traffic arriving at the gateway from Vera's container *is* a fallback, @@ -184,7 +184,7 @@ from the payload): ## Deploying Vera (the reference host) This repo doubles as Vera's image source: `Dockerfile` = stock protoAgent (**pinned -base** — `protoagent:0.144.0`, in step with the manifest's `verified_against`; bump +base** — `protoagent:0.145.0`, in step with the manifest's `verified_against`; bump deliberately so a member-pin bump can't drag the core forward on the same roll) + node/`clawpatch` + the bundle members baked at their manifest pins + `deploy/vera.langgraph-config.yaml` (seed, not force) + `SOUL.md`. diff --git a/protoagent.bundle.yaml b/protoagent.bundle.yaml index 2a23c28..64a0f37 100644 --- a/protoagent.bundle.yaml +++ b/protoagent.bundle.yaml @@ -25,11 +25,11 @@ description: >- own work; never posts a blocking verdict against pending CI. # The core version this pin set was last verified against (ADR 0049 rule 2). -verified_against: 0.144.0 +verified_against: 0.145.0 plugins: - { id: workflows, builtin: true } # the recipe engine (code-review panels run through it) - - { id: github, url: https://github.com/protoLabsAI/github-plugin, ref: v0.4.0 } + - { id: github, url: https://github.com/protoLabsAI/github-plugin, ref: v0.5.0 } - { id: pr-reviewer, url: https://github.com/protoLabsAI/pr-reviewer-plugin, ref: v0.35.0 } enabled: [workflows, github, pr-reviewer] # suggested turn-on list (applied to plugins.enabled) From 01f702684f51148d1691405da83b849cced854a6 Mon Sep 17 00:00:00 2001 From: GitHub CI <ci@example.com> Date: Sat, 22 Aug 2026 19:50:31 -0700 Subject: [PATCH 4/7] fix(watchdogs): guard the state write, and stop the docs contradicting the pin MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vera's third-round FAIL on this PR, and she was right on all three — the first has been open since round 1 and I hadn't been reading her earlier passes. - **`_save_state()` sat outside the only try/except in `main()`** (major, confirmed by four review angles plus the protopatch structural pass). An OSError from `write_text()` — full disk, bad permissions — escaped as a traceback instead of the documented exit 2. A watchdog that dies with a stack trace instead of its own error contract is precisely the failure this whole PR is about. Guarded now, with the asymmetry made explicit: a save failure downgrades to exit 2 only when there was nothing else to report; a real FALLBACK verdict survives it. Relabelling a silent degrade as an "unreachable" outage would send the operator looking in the wrong place, and the cost of not saving is a duplicate alert next run — strictly better than a missed one. - **The README claimed #2956 was "FIXED in 0.145.0" while the script's docstring still told the reader to delete it once that fix landed** — this PR pins 0.145.0, so it already satisfied its own retirement condition. Reconciled around the distinction that was missing from both: *pinning a version is not running it.* Vera rolls on watchtower after merge, so between pin and roll she is on 0.137.1 with no event at all. The script retires when the RUNNING instance reports 0.145.0 and the event has been seen firing — not before, or the window is covered by neither. - **nit: "OK: OK:"** — the non-native branch baked in a prefix `main()` also prepends. Fixed, and pinned by a test, since her note that the suite only ever calls `evaluate()` and never `main()`'s print path is the reason it survived. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FnbQVnHNvTsvbnJ1pdUDdD --- README.md | 18 +++++++++++++----- scripts/check_model_fallback.py | 29 +++++++++++++++++++++++++---- scripts/check_oauth_health.py | 4 +++- tests/test_watchdog_checks.py | 15 +++++++++++++++ 4 files changed, 56 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index ca20b7c..828174b 100644 --- a/README.md +++ b/README.md @@ -98,13 +98,21 @@ rejected on every call. **A fallback is SILENT.** protoAgent wires langchain's `ModelFallbackMiddleware` raw, and that middleware swallows the primary's exception with no log, no counter and no event — -verified against core 0.144.0 — FIXED in 0.145.0 (#2956), filed as protoAgent#2956. So a dead subscription doesn't -break the review; it quietly changes which model writes the verdict. That is what +found on core 0.144.0 and filed as protoAgent#2956. So a dead subscription doesn't break +the review; it quietly changes which model writes the verdict. That is what `scripts/check_model_fallback.py` exists to catch, by inference from gateway metrics: protoAgent-UA traffic arriving at the gateway from Vera's container *is* a fallback, -because her primary never goes there. If #2956 lands a `model.fallback` lifecycle event, -delete that script and subscribe to the event — it would be ground truth where this is -an inference. +because her primary never goes there. + +**#2956 is FIXED in core 0.145.0**, which the pins above now carry: +`ObservableModelFallbackMiddleware` logs a WARNING and publishes a `model.fallback` bus +event (ADR 0039). The inference script is therefore scheduled for deletion — but not +yet, and the distinction matters: **pinning a version is not running it.** Vera rolls on +watchtower after a merge, so between the pin landing and the roll completing she is on +the old core with no event at all. Retire the script once the running instance reports +0.145.0 *and* the event has been seen firing; deleting the inference before its +replacement is observed working would leave the silent-degrade window covered by +neither. **Reviews are not cheap.** One structural review is nine LLM steps and 5–9 minutes of wall clock. On a hosted frontier model that's roughly $0.12–0.15 each; on local inference diff --git a/scripts/check_model_fallback.py b/scripts/check_model_fallback.py index 74b777c..410847d 100755 --- a/scripts/check_model_fallback.py +++ b/scripts/check_model_fallback.py @@ -5,9 +5,16 @@ SILENTLY. protoAgent wires langchain's `ModelFallbackMiddleware` raw (graph/agent.py), and that middleware swallows the primary's exception with no log, no counter, and no event — a successful fallback is byte-for-byte indistinguishable from a normal turn. -Verified against core 0.144.0; filed upstream as protoAgent#2956. If that issue lands a -`model.fallback` lifecycle event, DELETE this script and subscribe to the event instead: -this is an inference, and the event would be ground truth. +Filed upstream as protoAgent#2956 and **FIXED in core 0.145.0** — which this repo now +pins: `ObservableModelFallbackMiddleware` logs a WARNING and publishes a `model.fallback` +bus event (ADR 0039). So this script is already living on borrowed time, deliberately: + +RETIRE IT once the RUNNING instance is on 0.145.0 *and* the event has been seen firing. +Pinning a version is not the same as running it — Vera rolls on watchtower after this +merges, and until then she is on 0.137.1 with no event at all. Deleting the inference +before its replacement is observed working would leave the silent-degrade window +uncovered by both, which is the one outcome worth avoiding. Once the event is confirmed, +delete this file: the event is ground truth where this is an inference from traffic. THE INFERENCE. Since 2026-08-21 Vera's primary is a NATIVE OAUTH provider (`model.provider: anthropic-oauth`, claude-sonnet-5), which per ADR 0097 bypasses the @@ -206,7 +213,21 @@ def main() -> int: verdict = 1 if not args.no_save: - _save_state(args.state, new_state) + try: + _save_state(args.state, new_state) + except OSError as exc: + # Persisting the baseline is bookkeeping; the verdict is the product. A + # full disk must never turn a REAL fallback into an "unreachable" alarm — + # that would relabel a silent degrade as an outage and send the operator + # looking in the wrong place. So a save failure downgrades to exit 2 only + # when there was nothing else to report; a verdict of 1 survives it and + # says so. (Cost of not saving: the next run re-alarms off a stale + # baseline. A duplicate alert is strictly better than a missed one.) + print(f"WARNING: could not persist the baseline to {args.state}: {exc}") + if verdict == 0: + print("No fallback to report, but the next run cannot alarm on growth — treating as operational.") + return 2 + print("Keeping the FALLBACK verdict; the next run may re-alarm from a stale baseline.") return verdict diff --git a/scripts/check_oauth_health.py b/scripts/check_oauth_health.py index 74afb9f..fe74eb7 100755 --- a/scripts/check_oauth_health.py +++ b/scripts/check_oauth_health.py @@ -61,7 +61,9 @@ def evaluate(model_cfg: dict, providers: list[dict]) -> tuple[int, list[str]]: name = (model_cfg.get("name") or "").strip() if provider not in NATIVE_PROVIDERS: - return 0, [f"OK: model.provider={provider!r} is not a native OAuth lane — no credential to check."] + # No "OK: " here — main() prepends the verdict prefix from the exit code, and + # baking a second one in produced "OK: OK: …" for every gateway-backed agent. + return 0, [f"model.provider={provider!r} is not a native OAuth lane — no credential to check."] lines.append(f"lane: {provider} · {name}") problems: list[str] = [] diff --git a/tests/test_watchdog_checks.py b/tests/test_watchdog_checks.py index e5025a5..1e75001 100644 --- a/tests/test_watchdog_checks.py +++ b/tests/test_watchdog_checks.py @@ -140,6 +140,21 @@ def test_gateway_backed_agent_is_a_noop(self): self.assertEqual(code, 0) self.assertIn("not a native OAuth lane", lines[0]) + def test_messages_carry_no_verdict_prefix(self): + # main() prepends "OK: "/"FAIL: " from the exit code. A branch that bakes its + # own prefix in prints "OK: OK: …" — caught in review, and invisible to the + # other tests here because they all call evaluate() directly and never main(). + for cfg, status in ( + ({"provider": "openai", "name": "protolabs/cloud"}, []), + (self.NATIVE, oauth_status()), + (self.NATIVE, oauth_status(signed_in=False)), + ): + _, lines = evaluate(cfg, status) + self.assertFalse( + lines[0].startswith(("OK:", "FAIL:")), + f"evaluate() must not prefix its own verdict: {lines[0]!r}", + ) + def test_expired_but_refreshable_is_not_an_alarm(self): # Refresh is ON USE, so a busy agent legitimately sits at or past its access # token's expiry. Alarming on proximity would cry wolf every few hours. From 8331b492ab8696280e43dd23b0e5236af2059a00 Mon Sep 17 00:00:00 2001 From: GitHub CI <ci@example.com> Date: Sat, 22 Aug 2026 20:58:30 -0700 Subject: [PATCH 5/7] docs(watchdog): the header still said "two watchdogs" after growing to four MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vera's nit on #45, and correct — I added the fallback and oauth modes to the case statement and left the header describing the original pair. The install lines and usage string were already updated; only the prose lagged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FnbQVnHNvTsvbnJ1pdUDdD --- scripts/vera-watchdog.sh | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/scripts/vera-watchdog.sh b/scripts/vera-watchdog.sh index b56ea32..fc12c45 100755 --- a/scripts/vera-watchdog.sh +++ b/scripts/vera-watchdog.sh @@ -1,9 +1,11 @@ #!/usr/bin/env bash -# Runner + alerter for Vera's two watchdogs (qaEngineer#37). +# Runner + alerter for Vera's four watchdogs — health, drift, fallback, oauth +# (qaEngineer#37 shipped the first two; the model-lane pair arrived with the move to a +# native Claude subscription). # # WHY THIS FILE EXISTS AT ALL, AND WHY IT LIVES HERE: # -# Both checks shipped in qaEngineer/scripts/ with docstrings saying "run from the ava +# The first two shipped in qaEngineer/scripts/ with docstrings saying "run from the ava # fleet cron" — and then nothing ever ran them. The review-health alarm was written # precisely because 24 PRs merged with no review and the only record was an inbox # nobody read; leaving that alarm unscheduled reproduced the same failure one level up. From 299bc0c168175db2577673f0afb655206431e043 Mon Sep 17 00:00:00 2001 From: GitHub CI <ci@example.com> Date: Sun, 23 Aug 2026 00:04:24 -0700 Subject: [PATCH 6/7] =?UTF-8?q?fix(watchdogs):=20make=20the=20fallback=20r?= =?UTF-8?q?ule=20survive=20a=20lane=20change=20=E2=80=94=20it=20just=20inv?= =?UTF-8?q?erted?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vera's Claude subscription exhausted its 7-DAY quota after ~2 days of panel work (`7d-status: rejected`, `7d-utilization: 1.0`, `retry-after: 356343` = 4.1 days), so she is back on the gateway: `protolabs/smart` primary, `protolabs/cloud` fallback. That switch silently inverted the fallback detector. Its rule was "a protoAgent-UA gateway request from her container IS a fallback" — sound only while the primary was a native-OAuth lane that bypassed the gateway entirely. With a gateway primary, every ordinary review matches it, and the 15-minute cron would have paged #alerts continuously starting minutes after the switch. The rule now asks the LIVE CONFIG which models are the fallback and counts only traffic requesting those. Correct for both lane shapes, and it keeps working across the next switch without anyone remembering to edit it — which is the actual fix, since nobody was going to remember. It refuses outright (exit 2) if fallback_models contains the primary, because a no-op lane cannot distinguish anything and crying wolf is worse than staying quiet. Four tests pin the discriminator. Also: `vera-watchdog.sh`'s alerts.env fallback moves here from the Infisical branch, where I had misfiled it. It is watchdog robustness, not secrets work, and keeping it one branch up meant every install from this branch silently regressed the alert path — the branch trap the script's own header warns about, which I have now walked into three times. Verified end-to-end with the env var stripped: still posts to Discord. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FnbQVnHNvTsvbnJ1pdUDdD --- README.md | 15 +++++++-- scripts/check_model_fallback.py | 56 ++++++++++++++++++++++++++------- scripts/vera-watchdog.sh | 21 ++++++++++++- tests/test_watchdog_checks.py | 39 +++++++++++++++++++++++ 4 files changed, 115 insertions(+), 16 deletions(-) diff --git a/README.md b/README.md index 828174b..435f69a 100644 --- a/README.md +++ b/README.md @@ -87,9 +87,18 @@ plus prior-round context. In practice that's **~37k tokens per finder**. change. Model settings are host-scoped (ADR 0047) — **or** a native subscription lane (see below), which is not a gateway edit at all. -**The reference host's lane (since 2026-08-21):** `model.provider: anthropic-oauth` + -`model.name: claude-sonnet-5`, with `routing.fallback_models: ["protolabs/smart"]` behind -it. Native OAuth **bypasses the gateway entirely** (ADR 0097) — the fallback alias is +**The reference host's lane (since 2026-08-23):** `protolabs/smart` on the gateway +(`model.provider: openai`), with `routing.fallback_models: ["protolabs/cloud"]` behind +it. She ran a native Claude subscription (`anthropic-oauth` / `claude-sonnet-5`) from +2026-08-21 to 08-23 and it reviewed well — ~25% faster, same grounding rate — but a +five-finder panel across a fleet's worth of repos **exhausted the subscription's 7-day +quota in under two days** (`anthropic-ratelimit-unified-7d-status: rejected`, +`retry-after: 356343` = 4.1 days). A reviewer that cannot review for four days is not a +reviewer, so the gateway is the sustainable primary and a subscription is the treat. +Set `model.name` and `model.provider` in the SAME config POST either way: they are one +decision (protoAgent#2623). + +**The old native-OAuth shape, for reference:** Native OAuth **bypasses the gateway entirely** (ADR 0097) — the fallback alias is reachable only because protoAgent#2571 lets a namespaced slot name opt out of the native provider, so `model.api_base` + a gateway key must stay set even though the primary never uses them. Set `model.name` and `model.provider` in the SAME config POST: they are one diff --git a/scripts/check_model_fallback.py b/scripts/check_model_fallback.py index 410847d..501449c 100755 --- a/scripts/check_model_fallback.py +++ b/scripts/check_model_fallback.py @@ -16,14 +16,19 @@ uncovered by both, which is the one outcome worth avoiding. Once the event is confirmed, delete this file: the event is ground truth where this is an inference from traffic. -THE INFERENCE. Since 2026-08-21 Vera's primary is a NATIVE OAUTH provider -(`model.provider: anthropic-oauth`, claude-sonnet-5), which per ADR 0097 bypasses the -gateway entirely — her subscription traffic never touches LiteLLM. Her fallback is a -gateway alias (`protolabs/smart`), reachable only because #2571 lets a namespaced slot -name opt out of the native provider. So the two lanes are cleanly separable at the -gateway, and the rule is simply: +THE RULE. Ask the LIVE CONFIG which models are the fallback, then count only gateway +traffic requesting THOSE: - a protoAgent-UA chat completion from Vera's container IP == a fallback + a protoAgent-UA chat completion from Vera's container IP, + whose `requested_model` is in `routing.fallback_models` == a fallback + +This is deliberately NOT "any gateway traffic from her container". That shortcut worked +only while her primary was a native-OAuth subscription bypassing the gateway entirely +(ADR 0097) — and it silently inverted the moment she moved back to a gateway primary on +2026-08-23, when it would have called every ordinary review a fallback and paged #alerts +every 15 minutes. Reading the fallback list from the live config instead means the check +is correct for BOTH lane shapes, and keeps working across the next switch without anyone +remembering to come back and edit it. Two things share that gateway key and must NOT be counted: @@ -68,6 +73,8 @@ import urllib.request from pathlib import Path +from vera_api import operator_api_get + DEFAULT_STATE = Path.home() / ".cache" / "vera-model-fallback.json" # The gateway publishes on the shared ai_default net; from ava's host namespace it is # reachable on the published port. Overridable for a different host/stack. @@ -83,6 +90,22 @@ _LABEL = re.compile(r'(\w+)="((?:[^"\\]|\\.)*)"') +def _fallback_models(container: str) -> tuple[set[str], str]: + """(the configured fallback model names, the primary's name) from the LIVE config. + + Read fresh every run, never hardcoded: the whole point of this check is to notice a + lane change, so it must not carry a stale idea of which lane is which. + """ + cfg = operator_api_get(container, "/api/config").get("config", {}) + fallbacks = {str(m).strip() for m in (cfg.get("routing", {}).get("fallback_models") or []) if str(m).strip()} + primary = str(cfg.get("model", {}).get("name") or "") + if primary in fallbacks: + # A fallback identical to the primary is a no-op lane, and counting it would + # report every ordinary call as a degrade. Refuse rather than cry wolf. + raise RuntimeError(f"fallback_models contains the primary model {primary!r} — nothing to distinguish") + return fallbacks, primary + + def _container_ip(container: str) -> str: """Vera's current address on the gateway's network. @@ -113,8 +136,13 @@ def _scrape(url: str) -> str: raise RuntimeError(f"cannot scrape {url}: {exc}") from exc -def fallback_requests(metrics_text: str, agent_ip: str) -> tuple[float, dict[str, float]]: - """Total protoAgent-UA gateway chat requests from ``agent_ip``, and the per-model split. +def fallback_requests( + metrics_text: str, agent_ip: str, fallback_models: set[str] | None = None +) -> tuple[float, dict[str, float]]: + """Gateway chat requests from ``agent_ip`` that the AGENT made to a FALLBACK model. + + ``fallback_models`` is the live `routing.fallback_models`; None counts every model + (the old native-primary shape, kept for the case where nothing is configured). Pure over the scrape text so the attribution rule is unit-testable without a live gateway — the rule is the load-bearing part, not the HTTP. @@ -132,6 +160,8 @@ def fallback_requests(metrics_text: str, agent_ip: str) -> tuple[float, dict[str continue if labels.get("route") == EMBEDDING_ROUTE: continue + if fallback_models is not None and labels.get("requested_model") not in fallback_models: + continue # the primary lane doing its job value = float(match.group("value")) total += value model = labels.get("requested_model", "?") @@ -162,7 +192,8 @@ def main() -> int: try: agent_ip = _container_ip(args.container) - total, by_model = fallback_requests(_scrape(args.metrics_url), agent_ip) + fallbacks, primary = _fallback_models(args.container) + total, by_model = fallback_requests(_scrape(args.metrics_url), agent_ip, fallbacks) except Exception as exc: # noqa: BLE001 — every failure here is operational, exit 2 print(f"UNREACHABLE: {exc}") return 2 @@ -181,7 +212,8 @@ def main() -> int: verdict = 0 if previous is None: - print(f"BASELINE: {args.container} @ {agent_ip} — {int(total)} fallback requests so far [{split}]") + print(f"BASELINE: {args.container} @ {agent_ip} · primary={primary} fallback={sorted(fallbacks)} — " + f"{int(total)} fallback requests so far [{split}]") print("First run: recorded. Growth is what alarms, so this run cannot.") elif total < previous: # The gateway restarted and its counters reset. Re-baseline rather than reading @@ -205,7 +237,7 @@ def main() -> int: print(f"FALLBACK: {head}") print( f"Vera answered {int(delta)} model calls from her fallback lane, not " - "claude-sonnet-5. Check the subscription: " + f"{primary}. Check the primary lane: " "`curl -X POST localhost:7870/api/config/test-model` in the container " "(401/403 = re-auth needed, 429 = rate-limited, it will pass)." ) diff --git a/scripts/vera-watchdog.sh b/scripts/vera-watchdog.sh index fc12c45..7cbbc54 100755 --- a/scripts/vera-watchdog.sh +++ b/scripts/vera-watchdog.sh @@ -41,10 +41,29 @@ REPO="${VERA_WATCHDOG_REPO:-$HOME/dev/qaEngineer}" BIN="$HOME/.local/bin" MODE="${1:-}"; shift || true # remaining args pass through to the underlying check +# The alert path must not depend on a credential that can silently expire. On +# 2026-08-21 ava's `infisical login` session had lapsed and two health runs fell through +# to the no-secrets branch: the checks ran, passed, and would have alerted to NOTHING if +# they had failed — the guard disarmed with no signal, which is the exact failure class +# these watchdogs exist to catch, one level up. So: env first (infisical run, when a +# session is good), then a local 0600 file that no session can invalidate. +# +# install -d -m 700 ~/.config/vera +# printf 'DISCORD_WEBHOOK_ALERTS=%s\n' "$(infisical secrets get DISCORD_WEBHOOK_ALERTS --plain ...)" \ +# > ~/.config/vera/alerts.env && chmod 600 ~/.config/vera/alerts.env +# +# Regenerate it if the webhook is ever rotated — this is a cached copy, not the source. +ALERT_ENV="${VERA_ALERT_ENV:-$HOME/.config/vera/alerts.env}" + alert() { # alert <title> <body> local hook="${DISCORD_WEBHOOK_ALERTS:-}" + if [ -z "$hook" ] && [ -r "$ALERT_ENV" ]; then + # shellcheck disable=SC1090 + . "$ALERT_ENV" + hook="${DISCORD_WEBHOOK_ALERTS:-}" + fi if [ -z "$hook" ]; then - echo "vera-watchdog: DISCORD_WEBHOOK_ALERTS unset — run via infisical; NOT alerted" >&2 + echo "vera-watchdog: no DISCORD_WEBHOOK_ALERTS (env or $ALERT_ENV) — NOT alerted" >&2 return fi local body diff --git a/tests/test_watchdog_checks.py b/tests/test_watchdog_checks.py index 1e75001..395cded 100644 --- a/tests/test_watchdog_checks.py +++ b/tests/test_watchdog_checks.py @@ -102,6 +102,45 @@ def oauth_status(**over) -> list[dict]: return [base] +class FallbackModelFilter(unittest.TestCase): + """The lane-shape-independent rule: only traffic to a CONFIGURED fallback counts. + + Added when Vera moved from a native-OAuth primary back to a gateway primary + (2026-08-23). Under the old "any gateway traffic is a fallback" rule that switch + silently inverted the check — every ordinary review would have read as a degrade and + paged #alerts every 15 minutes. These pin the rule that makes it survive a lane + change without anyone editing the script. + """ + + def test_primary_traffic_is_not_a_fallback(self): + text = sample(requested_model="protolabs/smart", value=40.0) + total, _ = fallback_requests(text, VERA_IP, {"protolabs/cloud"}) + self.assertEqual(total, 0.0, "the primary lane doing its job is not a degrade") + + def test_fallback_traffic_counts(self): + text = sample(requested_model="protolabs/cloud", value=7.0) + total, by_model = fallback_requests(text, VERA_IP, {"protolabs/cloud"}) + self.assertEqual(total, 7.0) + self.assertEqual(by_model, {"protolabs/cloud": 7.0}) + + def test_mixed_traffic_counts_only_the_fallback(self): + text = "\n".join([ + sample(requested_model="protolabs/smart", value=100.0), # primary + sample(requested_model="protolabs/cloud", value=3.0), # fallback + sample(requested_model="protolabs/cloud", user_agent="node", value=50.0), # clawpatch + ]) + total, by_model = fallback_requests(text, VERA_IP, {"protolabs/cloud"}) + self.assertEqual(total, 3.0) + self.assertEqual(by_model, {"protolabs/cloud": 3.0}) + + def test_none_counts_every_model(self): + # The native-primary shape: nothing configured to filter on, so any agent + # gateway call is a fallback by construction. + text = sample(requested_model="protolabs/smart", value=9.0) + total, _ = fallback_requests(text, VERA_IP, None) + self.assertEqual(total, 9.0) + + class OAuthHealth(unittest.TestCase): NATIVE = {"provider": "anthropic-oauth", "name": "claude-sonnet-5"} From 225afdf96bace521a8de797e10e54368460a5c89 Mon Sep 17 00:00:00 2001 From: GitHub CI <ci@example.com> Date: Sun, 23 Aug 2026 00:22:50 -0700 Subject: [PATCH 7/7] docs(watchdog): the fallback case comment still described the abandoned rule MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Vera's WARN on #45 (minor, confirmed). I rewrote check_model_fallback.py's rule to read routing.fallback_models from the live config, and left the wrapper's case comment asserting the old native-OAuth premise — that any protoAgent-UA gateway traffic is by construction a fallback. That premise is exactly what the switch back to a gateway primary invalidated, so the comment now documents the trap rather than the rule. Her two carried-forward majors on this head are stale, not open: both were fixed in the previous commit and she marked them "⚠️ uncertain — no repo access this pass", which is the panel correctly refusing to re-confirm what it could not read. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FnbQVnHNvTsvbnJ1pdUDdD --- scripts/vera-watchdog.sh | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/scripts/vera-watchdog.sh b/scripts/vera-watchdog.sh index 7cbbc54..06125ad 100755 --- a/scripts/vera-watchdog.sh +++ b/scripts/vera-watchdog.sh @@ -92,13 +92,18 @@ case "$MODE" in fi ;; fallback) - # Did Vera silently answer from her FALLBACK model? Her primary is a native-OAuth - # Claude subscription, which bypasses the gateway entirely (ADR 0097) — so - # protoAgent-UA traffic arriving AT the gateway from her container is, by - # construction, a fallback. protoAgent emits nothing when this happens (langchain's - # ModelFallbackMiddleware swallows the primary's exception without so much as a log - # line; filed as protoAgent#2956), so an inference from gateway metrics is the only - # signal there is until that issue lands a `model.fallback` event. + # Did Vera silently answer from her FALLBACK model? The check reads + # `routing.fallback_models` from her LIVE config and counts only gateway traffic + # requesting those, so it stays correct whichever lane is primary — an earlier + # version hardcoded "any protoAgent-UA gateway traffic is a fallback", which was + # true only while the primary was a native-OAuth subscription bypassing the gateway, + # and inverted the moment she moved back to a gateway primary. + # + # Why infer at all: on cores before 0.145.0 protoAgent emitted NOTHING on failover + # (langchain's ModelFallbackMiddleware swallows the primary's exception without so + # much as a log line; filed as protoAgent#2956, fixed there). Once the RUNNING + # instance is on 0.145.0+ and its `model.fallback` event is seen firing, retire this + # inference and subscribe to the event instead. out="$("$BIN/vera-model-fallback.py" --container vera "$@" 2>&1)"; rc=$? ;; oauth)