Sidecar preflight: Layer 1's 160s worst case excludes discover_all_models()'s own sequential network time #1455

Background

Devin Review, on PR #1452 (ADR-0005's implementation), verified against current code and the vendored
contextual-orchestrator source: the 180s healthz-readiness watchdog
(scripts/ci/contextual_orchestrator_review_sidecar.sh, the until curl .../healthz ...; do ... i -ge 180 ... loop) starts polling immediately after the launcher process (contextual_orchestrator_review_launcher.py main()) is spawned — and that process's first substantial work is
contextual_orchestrator.model_discovery.discover_all_models(), not preflight probing. Layer 1's own
"160s worst case, under the 180s ceiling" arithmetic (REVIEW_PREFLIGHT_MAX_ESCALATIONS's docstring
comment, scripts/ci/contextual_orchestrator_review_launcher.py around line 63) accounts only for the
probing/escalation phase (REVIEW_PREFLIGHT_MAX_TOTAL_ROUTES * REVIEW_PREFLIGHT_TIMEOUT_SECONDS + REVIEW_PREFLIGHT_MAX_ESCALATIONS * REVIEW_PREFLIGHT_TIMEOUT_SECONDS = 160s) — it does not include
discovery's own time, even though both phases run inside the same 180s watchdog.

Verified against the actual vendored source (not assumed)

contextual_orchestrator/model_discovery.py (checked directly in the vendored checkout):

• DISCOVERY_TIMEOUT_SECONDS = 15.0, applied per HTTP call via urllib.request.urlopen(..., timeout=timeout).
• discover_all_models() makes these calls sequentially, not concurrently:
  1. One _fetch_json(_MODELS_DEV_URL, ...) call, if any source with models_dev_provider_id has a
     registered credential (true for this sidecar's pool: openai, nvidia_nim, nvidia_nim_sub all
     declare it and all have credentials the sidecar registers).
  2. One discover_provider_models(source, ...) call per entry in PROVIDER_MODEL_SOURCES (6
     entries: openai, openrouter, opencode_zen, nvidia_nim, nvidia_nim_sub, bytez) — a source
     with no registered credential returns [] immediately with no network call
     (if not api_key: return []), but this sidecar registers 5 of the 6 (all but opencode_zen), so 5
     of these are real, sequential network calls.
  3. One final _openrouter_zdr_model_ids(...) call (openrouter has a credential here, so this runs too).
• Worst case: 7 sequential calls × up to 15s each = up to ~105s, entirely before Layer 1 probing
  starts, inside the same 180s watchdog that also has to fit the 160s probing/escalation worst case.
  Combined real worst case is therefore up to ~265s, not the ~160s the current comments imply — a
  slow-but-not-fully-hung discovery pass alone could exhaust the watchdog before probing gets any of its
  own 160s budget.

Options considered (not evaluated to a decision)

1. Thread one monotonic remaining-time deadline through discovery (discover_all_models(timeout=...)
   already accepts a timeout kwarg on the launcher's side — no vendored-source change needed) and
   probing, so the two phases share one real budget instead of each assuming the full window.
2. Scale down probe/escalation route count or the escalation budget based on however much of the 180s
   discovery already consumed.
3. Extend the 180s watchdog to a new, evidence-justified figure that honestly covers the full sequence
   (discovery + probing) — needs real observed timing, not an inspection-only guess (this org's ADR-0005
   explicitly rejects "heuristics and rules of thumb" as a justification standard).

Ask

Design and land a fix that makes the actual, combined worst-case bound explicit and correct (or justify
extending the watchdog with real telemetry). Not blocking PR #1452's fix of the 7 verified Devin Review
findings already in that PR — flagged there as architecturally significant enough to need its own pass
rather than a guessed patch, per this org's convergence convention (initial values from precedent,
refinement from telemetry, never from inspection alone).

Cross-ref: ADR-0005 (docs/adr/0005-sidecar-preflight-token-budget.md), PR #1452, PR #1449,
ContextualWisdomLab/contextual-orchestrator's model_discovery.py.