Skip to content

Health/probe gap: /api/v1/health can't distinguish two origins with different data; EU served an empty /api/v1/scores for 7 months #35

Description

@emooreatx

What went unnoticed

api.ethicsengine.org has two A records (US 207.148.13.157, EU 91.99.150.121), unproxied. Both returned identical /api/v1/health (version 1, build 45, "CIRISNode is healthy") for seven months while:

US EU
/api/v1/scores 8 models {"scores": []}
/api/v1/embed/scores 1534 B 119 B
evals per model 5–6 2–4 (below the 5-eval publish threshold)

Root cause is infrastructure (databases never replicated — CIRISAI/CIRISCore#2), but the application gave a probe nothing to catch it with, and from North America happy-eyeballs picked the US origin ~100% of the time so manual checks looked fine.

Asks

  1. Data-aware health: add /api/v1/health/data (or extend /api/v1/health) with {"evaluations": N, "publishable_models": M, "latest_completed_at": ts, "db_host_fingerprint": <hash of current_setting('cluster_name')||inet_server_addr()>} so a per-origin probe can diff two origins in one line.
  2. /api/v1/health should report the real build: git SHA / image tag, not a hard-coded version: 1, build: 45. Every origin has said "build 45" through at least five deploys.
  3. Ship a tiny per-origin probe (script or GitHub Action) that --resolves each A record and alerts when publishable_models differ. It belongs in CIRISCore's ops alerting, but the endpoint has to exist here first.

Found during the 2026-09-24 site audit.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions