Skip to content

fix(gateway): boot-seed challenge backends on restart - #93

Open
echobt wants to merge 2 commits into
mainfrom
fix/gateway-backend-boot-seed
Open

fix(gateway): boot-seed challenge backends on restart#93
echobt wants to merge 2 commits into
mainfrom
fix/gateway-backend-boot-seed

Conversation

@echobt

@echobt echobt commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Gateway challenge registry is in-memory and was only re-seeded by remote-deploy.sh, so a plain gateway restart left [] and /challenge/* returned 503 no healthy backends.
  • Boot-seed from BASE_GATEWAY_BACKENDS (compose default: prism + design Docker DNS URLs) on every gateway process start; admin POST reseed stays idempotent.
  • Docs/runbooks updated; no site-api/frontend retain-page changes.

Test plan

  • cargo test -p gateway-registry --lib
  • cargo clippy -p gateway-registry -p gateway --all-targets -- -D warnings
  • ./deploy/scripts/assert-compose-matrix.sh
  • Staging: deploy gateway from this branch, docker compose restart gateway, confirm GET /v1/admin/backends lists design+prism and site submissions non-empty
  • Prod: same restart verification after image/pin promote

Summary by CodeRabbit

  • New Features

    • Gateway challenge backends can now be seeded automatically at startup from comma- or newline-separated configuration entries.
    • Supports optional backend weights, URL validation, duplicate handling, and safe repeated seeding.
    • File-based configuration takes precedence over inline settings.
    • Compose deployments now provide default Prism and Design challenge endpoints.
  • Documentation

    • Updated deployment guides and API documentation to explain automatic seeding and manual registration fallback.
    • Clarified that registrations are in-memory and deployment scripts can safely reseed backends without restarting.

Survive gateway-only restarts without empty in-memory registry / 503s by seeding prism+design from BASE_GATEWAY_BACKENDS on process start.
@coderabbitai

coderabbitai Bot commented Aug 8, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: bada60d8-ec9f-41aa-87e5-810196c7902e

📥 Commits

Reviewing files that changed from the base of the PR and between 657dec8 and 687ad22.

📒 Files selected for processing (1)
  • deploy/scripts/remote-deploy.sh
🚧 Files skipped from review as they are similar to previous changes (1)
  • deploy/scripts/remote-deploy.sh

📝 Walkthrough

Walkthrough

The gateway now parses backend seed lists, loads them from environment configuration, seeds the registry during startup, and provides Compose defaults. Deployment scripts and runbooks document boot seeding and idempotent reseeding.

Changes

Backend boot seeding

Layer / File(s) Summary
Registry parsing and idempotent seeding
crates/gateway-registry/src/lib.rs
Backend definitions support comma- or newline-separated input, optional weights, URL normalization, validation, duplicate handling, and selection tests.
Environment seed loading
crates/gateway/src/gw_config.rs
The gateway loads seeds from BASE_GATEWAY_BACKENDS_FILE or BASE_GATEWAY_BACKENDS. File input takes precedence.
Startup registry seeding
crates/gateway/src/lib.rs, crates/gateway/src/gw_config.rs
Startup seeds the registry before application construction, logs newly created backends, and re-exports the seed helpers.
Deployment defaults and operational documentation
docker-compose.yml, deploy/AGENTS.md, deploy/scripts/*, docs/SITE_API.md, docs/runbooks/staging-testnet-e2e.md
Compose defines default Prism and Design backends. Deployment scripts synchronize challenge-review and perform idempotent reseeding. Documentation describes boot seeding and fallback registration.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ComposeEnvironment
  participant GatewayStartup
  participant gw_config
  participant Registry
  ComposeEnvironment->>gw_config: BASE_GATEWAY_BACKENDS configuration
  GatewayStartup->>gw_config: seed_registry_from_env()
  gw_config->>gw_config: load and parse backend seeds
  gw_config->>Registry: seed(Vec<CreateBackend>)
  Registry-->>gw_config: created count
  gw_config-->>GatewayStartup: seed result
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: boot-seeding challenge backends when the gateway restarts.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/gateway-backend-boot-seed

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docker-compose.yml`:
- Around line 96-98: Update the BASE_GATEWAY_BACKENDS interpolation in
docker-compose.yml to use the single-dash default form so an explicitly empty
environment variable is preserved. Keep the existing empty-override claims
unchanged in deploy/scripts/remote-deploy.sh (lines 441-443) and
docs/runbooks/staging-testnet-e2e.md (lines 56-58); those sibling sites require
no direct changes.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fd508f06-aa67-4a8e-9755-7e5d6fb0bf24

📥 Commits

Reviewing files that changed from the base of the PR and between 8c0dc59 and 657dec8.

📒 Files selected for processing (9)
  • crates/gateway-registry/src/lib.rs
  • crates/gateway/src/gw_config.rs
  • crates/gateway/src/lib.rs
  • deploy/AGENTS.md
  • deploy/scripts/register-challenge-backends.sh
  • deploy/scripts/remote-deploy.sh
  • docker-compose.yml
  • docs/SITE_API.md
  • docs/runbooks/staging-testnet-e2e.md

Comment thread docker-compose.yml
Comment on lines +96 to +98
# Durable boot seed for the in-memory challenge registry (survives
# gateway-only restarts; remote-deploy POST reseed stays idempotent).
BASE_GATEWAY_BACKENDS: "${BASE_GATEWAY_BACKENDS:-prism=http://prism-challenge:8092,design=http://design-challenge:8093}"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

BASE_GATEWAY_BACKENDS='' docker compose -f docker-compose.yml config |
  sed -n '/BASE_GATEWAY_BACKENDS:/,+1p'

Repository: BaseIntelligence/base

Length of output: 203


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== candidate files =="
git ls-files | rg '(^docker-compose\.yml$|^deploy/scripts/remote-deploy\.sh$|^docs/runbooks/staging-testnet-e2e\.md$)'

echo
echo "== docker-compose relevant lines =="
nl -ba docker-compose.yml | sed -n '85,105p'

echo
echo "== remote-deploy relevant lines =="
nl -ba deploy/scripts/remote-deploy.sh | sed -n '430,450p'

echo
echo "== staging runbook relevant lines =="
nl -ba docs/runbooks/staging-testnet-e2e.md | sed -n '48,62p'

echo
echo "== Bash expansion probe =="
(set -u; unset BASE_GATEWAY_BACKENDS; echo "unset=${BASE_GATEWAY_BACKENDS:-PROVISIONAL}" "${BASE_GATEWAY_BACKENDS-PROVISIONAL}" | tr ' ' '\n')
(BASE_GATEWAY_BACKENDS=''; set -u; echo "empty=${BASE_GATEWAY_BACKENDS:-PROVISIONAL}" "${BASE_GATEWAY_BACKENDS-PROVISIONAL}" | tr ' ' '\n')

Repository: BaseIntelligence/base

Length of output: 346


🏁 Script executed:

#!/bin/bash
set -euo pipefail

python3 - <<'PY'
from pathlib import Path
for path in ["docker-compose.yml", "deploy/scripts/remote-deploy.sh", "docs/runbooks/staging-testnet-e2e.md"]:
    p = Path(path)
    lines = p.read_text().splitlines()
    print(f"== {path} ==")
    if path == "docker-compose.yml":
        start, end = 85, 105
    elif path == "deploy/scripts/remote-deploy.sh":
        start, end = 430, 450
    else:
        start, end = 48, 62
    for i in range(start, min(end, len(lines)) + 1):
        print(f"{i}\t{lines[i-1]}")
PY

echo "== Bash expansion probe =="
(set -u; unset BASE_GATEWAY_BACKENDS; echo "unset=${BASE_GATEWAY_BACKENDS:-PROVISIONAL}" "${BASE_GATEWAY_BACKENDS-PROVISIONAL}" | tr ' ' '\n')
(BASE_GATEWAY_BACKENDS=''; set -u; echo "empty=${BASE_GATEWAY_BACKENDS:-PROVISIONAL}" "${BASE_GATEWAY_BACKENDS-PROVISIONAL}" | tr ' ' '\n')

Repository: BaseIntelligence/base

Length of output: 3135


Allow an explicit empty backend seed.

Compose treats :- the same for unset and empty variables, so running BASE_GATEWAY_BACKENDS='' docker compose ... still applies the Prism and Design defaults. Use - in the Compose file so the empty override takes effect, then keep the empty-override claims in deploy/scripts/remote-deploy.sh and docs/runbooks/staging-testnet-e2e.md.

📍 Affects 3 files
  • docker-compose.yml#L96-L98 (this comment)
  • deploy/scripts/remote-deploy.sh#L441-L443
  • docs/runbooks/staging-testnet-e2e.md#L56-L58
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@docker-compose.yml` around lines 96 - 98, Update the BASE_GATEWAY_BACKENDS
interpolation in docker-compose.yml to use the single-dash default form so an
explicitly empty environment variable is preserved. Keep the existing
empty-override claims unchanged in deploy/scripts/remote-deploy.sh (lines
441-443) and docs/runbooks/staging-testnet-e2e.md (lines 56-58); those sibling
sites require no direct changes.

Prebuilt docker builds COPY challenge-review; without syncing it, remote-deploy --build-from prebuilt fails the image context.
@echobt

echobt commented Aug 8, 2026

Copy link
Copy Markdown
Contributor Author

Deploy / verification

Root cause: in-memory registry; only remote-deploy POSTed backends. Plain docker compose restart gateway cleared []503 no healthy backends → site soft-empty.

Fix shipped: BASE_GATEWAY_BACKENDS boot-seed (compose default prism+design).

Staging (base-staging)

  • Deployed seeded gateway + compose env
  • After restart gateway: registry has design+prism, proxy /health 200/200, design submissions 24, prism 2
  • Logs: gateway_backends_seeded created=2

Prod (base-prod / chain.joinbase.ai)

  • Hotfixed gateway binary + docker-compose.yml env only (challenges untouched)
  • After restart gateway: registry has design+prism, proxy 200/200, design submissions 24, prism 2, public site 200
  • Logs: gateway_backends_seeded created=2

No site-api/frontend retain-page changes in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant