Skip to content

[Feature] provide review result metrics #96

Closed

Description

@zengchuang-hw

our review-bot kanban now does not show the metrics of the review task.
we cann't judge whether the review result is good or bad.

Activity

  1. added a commit that references this issue on Aug 25, 2026
  2. zengchuang-hw commented on Aug 27, 2026

    @zengchuang-hw
    ContributorAuthor

    以版本为周期进行评估,更新到kanban
    指标:质量(GPT 5.6 Sol,已经合入的PR与人工检视意见)、效率(平均一个检视PR耗时)

  3. zengchuang-hw commented on Aug 27, 2026

    @zengchuang-hw
    ContributorAuthor

    1、metrics评估方案:抽样三个人工检视意见最多的PR,进行评估

    2、workflow学习trace更新

  4. tzhouam commented on Aug 28, 2026

    @tzhouam
    Collaborator

    Shipped in three parts on the review-bot side:

    • zuiho-kai/omni-reviewbot#15 — every review job now records what it produced: severity histogram (P0–P3), inline anchor rate, gate failures, posted review ids. The dashboard shows the aggregates; NULL renders as "未知", never as 0, and every rate is displayed with its sample size.
    • zuiho-kai/omni-reviewbot#19 — the "was it good" half: a daily read-only collector gathers 👍/👎 reactions on the bot's inline comments, thread replies, and each PR's final state. Maintainer convention: react 👍/👎 on the bot's inline comments — that is the signal the dashboard's useful-rate is computed from (GitHub doesn't support reactions on top-level review bodies).
    • eval: monthly review-quality probe of omni-reviewbot as a harness arm #109 — a monthly ground-truth probe: bash eval/dataset/run_reviewbot_monthly.sh <YYYY-MM> re-reviews the 15 train+val benchmark PRs and judges them blind against the pinned CC+Opus 5 baseline (paired, item-clustered statistics; results in eval/dataset/results/reviewbot/). This answers "is the review quality holding" with actual recall/precision deltas, monthly.

    Closing as implemented. Live numbers appear on the kanban as reviews flow; the first monthly probe report will land under results/reviewbot/.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions