Skip to content

knowledge(vllm-omni): 1 change(s) from refine #422

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tzhouam merged 1 commit into from
Oct 26, 2026
Merged

knowledge(vllm-omni): 1 change(s) from refine #422

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions knowledge/repos/vllm-omni/_index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,3 +48,4 @@ owner 仍不明确时才看 [components 职责表](components/_index.md)；模
- [vllm-omni review-lessons rules](rules-review-lessons.md)
- [vllm-omni AMD/ROCm CI 流水线规则](rules-amd-ci.md)
- [vllm-omni /v1/realtime WebSocket 一致性评审规则](rules-realtime-conformance.md)
- [vllm-omni Omni-DuplexEval 判分校验规则](rules-duplexeval-judge.md)
19 changes: 19 additions & 0 deletions knowledge/repos/vllm-omni/rules-duplexeval-judge.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
---
title: "vllm-omni Omni-DuplexEval 判分校验规则"
created: 2026-10-26
updated: 2026-10-26
type: rule
tags: [vllm-omni]
sources: ["PR #9390"]
---

# vllm-omni Omni-DuplexEval 判分校验规则

## VLLM-OMNI-PR9390-DUPLEXEVAL-JUDGE-SCORES — Omni-DuplexEval 判分门槛：finish_reason=stop、按任务分数校验、原始回复轨迹与重试归档，judge 失败先走既有回退

- 触发：改动 `vllm_omni/benchmarks/duplex/` 判分链路——`omni_duplex_eval_metrics.py`（`parse_judge_json`）、`omni_duplex_eval_judge.py`（`DuplexJudge`、`JudgeResponseError`）、`omni_duplex_eval_eval.py`（`evaluate_sample`、`summarize_scores` 及 content 判分失败后的回退）——或其测试 `tests/benchmarks/duplex/test_omni_duplex_eval_judge.py`、`tests/benchmarks/duplex/test_omni_duplex_eval_serving.py`、`tests/e2e/features/fullduplex/test_omni_duplex_eval_cli.py`，或 `docs/cli/bench/serve.md` 中 judge 回复要求与重试工件小节。`vllm bench serve` 接线与运行目录语义见 VLLM-OMNI-PR8645-DUPLEXEVAL-BENCH，夜间守卫调用方式见 VLLM-OMNI-PR8871-DUPLEXEVAL-CI；本规则只覆盖判分校验本身，不改变那两条规则的门槛。

@chatgpt-codex-connector chatgpt-codex-connector Bot Oct 26, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Place the judge rule under the benchmark owner

This rule exclusively governs vllm_omni/benchmarks/duplex/, but its body is added at the repository root and linked only from the root index. The existing benchmark/_index.md already routes Omni-DuplexEval work within the benchmark owner, so readers following that entry do not encounter the new scoring constraints. This violates the nearest-owner requirement in root AGENTS.md. Put the rule in benchmark/rules.md (or an owner-scoped topic page reachable from it), update the benchmark index, and keep any repository-root entry as routing only.

AGENTS.md reference: AGENTS.md:L12-L14

Useful? React with 👍 / 👎.

- 强制：judge 回复必须显式以 `finish_reason="stop"` 结束才可评分：`finish_reason` 缺失或为 null 时即使分数 JSON 可解析也判失败（能解析出分数不是生成完成的证据），`length`、`content_filter`、`tool_calls`、`function_call` 及未知值同样被 `DuplexJudge` 以 `JudgeResponseError` 拒绝。分数校验按任务进行：temporal 判分需 `temporal_score`（0–3 整数）与 `is_relevant`（0 或 1），content 判分需 `content_score`（0–3 有限数值），PR 判分需 `success_score`（0 或 1）；越界、NaN/Infinity（含其字符串形式）、布尔、null、类型不符（如整数字段给小数）或缺失字段一律被 `parse_judge_json` 拒绝，而完成回复中的合法 0 分仍计为已评测样本，受支持的数字字符串（"1"、"2.0"、"2.50"）及 ```json 围栏、噪声前缀文本中的可解析分数继续接受并归一化为数值。拒绝后的走向按判分链路而定，不得一概判死样本：`omni_duplex_eval_eval.py` 在 content 判分的 `judge_result(...)` 失败（如全视频上下文溢出）后捕获 `Exception`，存在采样帧时改用 `mode="frame-sample"` 重试，既有的分块提醒评分路径同样保留——一次 judge 调用被拒或失败不必然使样本失败，样本只有当判分链路（含回退）无法给出有效结果时才判失败；PR 判分被拒时 `evaluate_sample` 抛错且该样本不写分数文件。每次 judge 调用的原始回复写入分数旁的 `<sample>.judge.jsonl`；`finish_reason` 仅在捕获 `JudgeResponseError`（回复不完整或完成状态缺失/非 stop 被拒）时随该轨迹条目落盘，成功的 judge 调用不承诺写入 `finish_reason`，不得把"每次调用都记录 finish_reason"当作验收条件；被拒样本在摘要计 0 但保留 raw_response 诊断轨迹。重试评测须先把既有分数归档为 `<sample>.json.<unique-id>.previous`（后缀保持稳定：摘要与准确率 CI 均按 `*.json` 扫描，`.previous` 因此被排除）并重置回复轨迹；失败的重试不留活动分数，新分数仅在判分所需的 judge 调用（含 `frame-sample` 回退重试）成功后发布；无关样本不受其他样本重试影响仍计入摘要，重试成功后只计新鲜分数。serving 评测经既有 failure/coverage 字段报告无效 judge 输出，judge 失败仍不废弃已完成的性能测量。
- 禁止：把不完整回复（非 stop 或缺失 finish_reason）解析出的分数发布为有效样本；放宽校验以接受越界、非有限、布尔或 "bad" 类字符串分数；取消 `.judge.jsonl` 原始回复落盘或被拒回复的 finish_reason 诊断保留；移除 content 判分失败后的 `mode="frame-sample"` 回退或分块提醒评分路径，令单次 judge 失败直接判死样本；令 `.previous` 归档分数重新进入摘要或准确率统计；在判分链路（含回退）整体失败时仍发布或保留活动分数。
- 验收：`tests/benchmarks/duplex/test_omni_duplex_eval_judge.py` 以 `core_model and cpu` 标记通过，覆盖无效分数参数化拒绝、合法官方格式保留、不完整回复无分数但轨迹含 raw_response 与被拒 finish_reason、缺失或非 stop 完成状态即使分数有效也被拒、失败重试归档旧分数并从摘要排除且无关样本仍计数；`omni_duplex_eval_eval.py` 保留 content 判分失败后的 `frame-sample` 回退；`docs/cli/bench/serve.md` 保留 finish_reason 与按任务分数字段要求及 `.judge.jsonl`、`.previous` 工件说明。^[PR #9390]

<!-- kb:rule status=active since=v0.30.0 -->
Loading