Skip to content

Observability - #780

Merged
fretchen merged 6 commits into
mainfrom
observ
Nov 7, 2026
Merged

fretchen merged 6 commits into
mainfrom
observ

Conversation

@fretchen

@fretchen fretchen commented Nov 7, 2026

Copy link
Copy Markdown
Owner

No description provided.

fretchen and others added 6 commits November 6, 2026 06:28
The bunny twins lost all log-based alerting at the scw_js -> bunny_js move;
this restores it on Scaleway Cockpit (EU, same Grafana as the rest).

- logger.ts: buffer + awaited flushLogs per request (no waitUntil on the
  edge; empty buffer = no fetch), {service, level} labels only, monotonic
  ns counter against Loki's duplicate-timestamp batch rejection, X-Token
  auth, failures console.error'd to the bunny dashboard. withLogFlush
  wraps the three entries' handlers (early returns AND throws).
- alerts/bunny.yaml: BunnySellersNeedAttention on level="error", pushed to
  the custom bunny-logs data source's ruler (alerts.ts --url override);
  budget test pins the count at four, raised deliberately.
- env set on all three bunny scripts (SCW_COCKPIT_PUSH_TOKEN secret,
  COCKPIT_LOKI_URL plain); the deploy Action ships code only, unchanged.

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
…, ruler guard)

From the pre-merge code + security review of the Cockpit flush:

- P1: the awaited flush fetch now carries AbortSignal.timeout(1500) — without
  a bound, a slow Cockpit stalls every logged request (unpaid 402s included).
  Regression test asserts an AbortSignal on the fetch init and that an abort
  surfaces via console.error instead of throwing.
- Security: debug lines are no longer buffered — the llm twins log { prompt }
  at debug and scw's pino gate (LOG_LEVEL ?? "info") suppressed them pre-move,
  so the flush would have been a NEW egress of user content, not parity.
  Regression test: debug alone sends no fetch; mixed payloads exclude it.
- Op-safety: alerts.ts refuses --push alerts/bunny.yaml without --url — the
  group would mount on the default-source ruler, bill daily, match nothing.
  Verified live: refusal fires, ruler untouched.
- Plan §2 rewritten to match the shipped logger.ts (Bearer claim corrected to
  the probe result, sketch marked as sketch).

Generated by Mistral Vibe.
Co-Authored-By: Mistral Vibe <vibe@mistral.ai>
@fretchen
fretchen merged commit 11a4785 into main Nov 7, 2026
7 checks passed
@fretchen
fretchen deleted the observ branch November 7, 2026 12:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant