Noema 리뷰 반복 실패 RCA: orchestrator/free 비JSON 응답 + 900초 repair 시한 초과 #2054
Description
현상
- newsdom-api PR 773: Run Noema LLM review and submit verdict 실패. Noema LLM response was not valid JSON (char 2222). 소요 약 31분.
- html4tree PR 595: Prepare Noema model verdict 실패. 초기 실패는 비JSON 응답 (char 1529), 이후 repair가 900초 절대 시한 초과로 2차 실패.
원인 분석
- 초기 실패는 양쪽 모두 orchestrator/free 모델이 유효한 JSON 봉투를 반환하지 않은 것.
- repair는 같은 모델에 같은 프롬프트를 반복하므로 비JSON이 반복되고, 절대 시한 900초를 소진함.
- 원시 출력 미기록은 보안상 의도적이나, 실패 위치(char 오프셋)·길이·sha256 외에 스키마 검증 오류 경로가 없어 디버깅 불가.
처방 제안 (시한 연장이 아님)
- Noema 모델 호출에 JSON 모드 강제 (response_format json_object 또는 json_schema).
- 비JSON 초기 실패 시 repair 진입 전 조기 중단하고 봉투 재생성 1회만 시도.
- 텔레메트리 확충: 파싱 실패 위치, 기대 스키마 대비 누락·여분 필드 경로를 로그에 기록 (원시 출력은 계속 미기록).
- repair 시한은 절대 누적 대신 시도당 예산으로 변경 검토.
근거 로그
- newsdom-api run 33510285421 job 99864028341 step 13
- html4tree run 33560972491 job 100033086428 step 13
Activity
seonghobae commentedon Sep 11, 2026
2026-09-11 follow-up evidence: the current Noema implementation already sends a schema-based response_format, performs one caller request, and leaves repair/failover to contextual-orchestrator. Regression suites pass:
uv run pytest tests/test_noema_repair_attempt_telemetry.py tests/test_noema_review_gate.py tests/test_repository_branch_coverage_review_schedulers.py -q-> 149 passed in 28.74s. The external failure remains reproducible at the gateway boundary: PR #2098 exact head69dded5cd596acd4ba77b7e78e4219b5e0bf27dcreceived HTTP 502 fromorchestrator/freeafter 399.5s (served modeldeepseek-ai/deepseek-v4-pro-0813, caller attempts=1); PR #2096 exact head9b3585f08147c076ce4a530e8e47bb9bd0b6bc29received HTTP 502 after 643.8s (served modeldeepseek-ai/deepseek-v4-flash-0731, caller attempts=1). Raw logs are retained locally with SHA-256: 20981ed56836519caaea82e85c42fa35ef3ac5724efab40ac91051a678f1d555e7d3; 2096 Noemad439c4ae418a373e6417b515c8c394e507641d1d7bbb9e50ba144d350111a135; 2096 Strixb56f18f5fafe1574963970863c69ce37ed99c33017a3b4c4c086e8f6527521e8. This is not closed: the code-side contract is green, but the provider/gateway reliability gap remains operationally unresolved.seonghobae commentedon Sep 13, 2026
Closing on re-verification, 2026-09-13: every central-side prescription here is on protected
main, and the recorded failure shape no longer occurs.scripts/ci/noema_review_gate.pybuilds_noema_verdict_json_schema()and sends it asresponse_format: json_schema(:72-142).NOEMA_REPAIR_DEADLINEno longer exists inscripts/ci/noema_review_gate.py; the caller makes one request and delegates structured-output repair/failover to contextual-orchestrator.What still fails today is a different class — gateway transport (
HTTP 429after ~300 s,HTTP 502after ~1000 s,provider_attempt_failedacross the free pool), e.g. runs34747265609/34736616822on 2026-09-13 — tracked as gateway capacity/budget work in #1948 and #2000, and owned upstream in contextual-orchestrator. Keeping this RCA open would mislabel those as the non-JSON/900 s defect.