Generate software patches and evaluate them with official benchmark tests
English · 简体中文
Install · Choose a command · Run Duo
OpenCollab-Eval prepares tasks, runs OpenCollab solvers, and evaluates their patches in isolated official test workspaces. The current version is 0.9.5, paired with OpenCollab >=0.9,<0.10.
Use Python 3.10 through 3.12 on Linux or macOS. The source installation below uses Python 3.12. Official SWE evaluation uses a Linux worker with Docker. OpenHands support requires Python 3.12.
mkdir -p "$HOME/oc-evaluation"
cd "$HOME/oc-evaluation"
git clone https://github.com/RISE-X-Lab/OpenCollab.git
git clone https://github.com/RISE-X-Lab/OpenCollab-Eval.git
python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install -e ./OpenCollab
python -m pip install -e ./OpenCollab-Eval
oc-eval --version
oc-eval --helpKeep task inputs, credentials, and generated results outside the source checkouts. Getting started also covers wheel installation and optional dependencies.
Prepare the input shown in the middle column, then follow the linked guide.
| Goal | Input | Command and guide |
|---|---|---|
| Generate a patch for your own repository | Generic task JSONL with an absolute repository path | oc-eval run · Getting started |
| Check a benchmark dataset | Benchmark JSONL and an evaluator-owned identity key | oc-eval inspect · Task formats |
| Run Duo with Single2 and official scoring | Ordered benchmark JSONL, task images, and a model configuration | oc-eval duo · Duo on a Linux worker |
| Run another workflow or a bounded remote slice | Prepared Linux worker, dataset, and model settings | oc-eval swe-v1-prolite · Operations |
| Score saved candidates again | A queue plan bound to the existing candidates | oc-eval rejudge-queue · Recovery |
| Prepare an installed runtime for a server | Installed compatible OC and OCE packages | oc-eval package-runtime · Evaluation suite |
| Build a final comparison | Two completed fact reports and their evidence | oc-eval final-report · Final reports |
The generic task file and benchmark dataset have different fields. Choose their matching command using the task format reference. Operations covers model configuration, and CLI reference describes other solver entrypoints.
Use a Git repository with at least one commit and a clean worktree. Replace /work/calculator with its absolute path and describe the change you want. Confirm that the first check prints a commit ID and the second prints nothing. The status check includes untracked files. Commit your intended changes or use a separate clean checkout.
git -C /work/calculator rev-parse --verify HEAD
git -C /work/calculator status --short --untracked-files=allCreate the input file outside that repository.
export EVAL_ROOT="$HOME/oc-evaluation/eval-data"
umask 077
mkdir -p "$EVAL_ROOT"
cat > "$EVAL_ROOT/tasks.jsonl" <<'TASKS'
{"task_id":"calculator-1","description":"Fix calculator.add and run its tests","repo_path":"/work/calculator","timeout":600,"max_tokens":100000}
TASKSSave the file as $HOME/oc-evaluation/eval-data/tasks.jsonl. Set your model and provider, then enter the API key at the prompt. For the openai provider, oc-eval run uses Chat Completions. Set OPENCOLLAB_BASE_URL to a base URL that supports this API. OPENCOLLAB_WIRE_PROTOCOL does not switch this command to Responses. The Duo tutorial supplies a separate Responses configuration.
export OPENCOLLAB_MODEL=your-model
export OPENCOLLAB_PROVIDER=openai
export OPENCOLLAB_API_KEY="$(python -c 'import getpass; print(getpass.getpass("API key > "))')"
oc-eval run "$EVAL_ROOT/tasks.jsonl" \
--output "$EVAL_ROOT/candidate-run" \
--concurrency 1 --timeout 600Read results.jsonl in the selected output directory. patch_produced tells you whether a patch was captured, and submission_eligible tells you whether it can proceed to evaluation. The command can exit with code 0 after a task fails, so check these fields and read error for the cause. Official pass or failure requires the benchmark's target tests to execute against that patch. Getting started explains the result fields and dataset inspection.
Duo uses two coders and an adjudicator to select a candidate, then Eval runs its official tests. The default Base agent profile resolves to Single2. Follow Duo on one Linux worker to prepare the ordered dataset, images, model relay, and configuration. After that preparation, run one task.
oc-eval duo --config "$EVAL_ROOT/duo.json" \
--indices 1 --workers 1 --run-id duo-smoke-001Read task_1_report.json under the configuration directory's results/duo-smoke-001 directory. It links the candidate to its official test report. A passing resolved result requires executed official targets and matching evidence. Missing evidence, failed environment preparation, and invalid candidate capture produce technical_failure. Operations explains these outcomes, batch execution, and recovery. oc-eval g22 is a compatibility alias for Duo.
The documentation index groups guides by task. Use CLI reference for options, Troubleshooting for failures, and Evaluation integrity for scoring rules. Evaluation suite covers server operation and provider capacity.
CONTRIBUTING.md covers development setup, repository checks, paired-wheel verification, and review. CHANGELOG.md records releases. The license is MulanPSL-2.0, with dependency notices in THIRD_PARTY_NOTICES.md.
生成代码补丁，并用基准的正式测试评测补丁
English · 简体中文
OpenCollab-Eval 负责准备任务、调用 OpenCollab 求解器，并在独立的正式测试工作区中评测补丁。当前版本为 0.9.5，配套依赖为 OpenCollab >=0.9,<0.10。
Linux 和 macOS 支持 Python 3.10 至 3.12。下面使用 Python 3.12 安装源码。正式 SWE 评测需要一台装有 Docker 的 Linux 工作机，OpenHands 支持需要 Python 3.12。
mkdir -p "$HOME/oc-evaluation"
cd "$HOME/oc-evaluation"
git clone https://github.com/RISE-X-Lab/OpenCollab.git
git clone https://github.com/RISE-X-Lab/OpenCollab-Eval.git
python3.12 -m venv .venv
. .venv/bin/activate
python -m pip install -e ./OpenCollab
python -m pip install -e ./OpenCollab-Eval
oc-eval --version
oc-eval --help任务输入、凭据和生成结果保存在源码目录之外。快速入门还介绍了 wheel 安装和可选依赖。
先准备中间一列的输入，再按对应指南运行。
| 目标 | 输入 | 命令与指南 |
|---|---|---|
| 给自己的仓库生成补丁 | 含仓库绝对路径的通用任务 JSONL | oc-eval run · 快速入门 |
| 检查基准数据集 | 基准 JSONL 和评测器持有的身份密钥 | oc-eval inspect · 任务格式 |
| 用 Single2 运行 Duo 并正式评分 | 有序的基准 JSONL、任务镜像和模型配置 | oc-eval duo · 在 Linux 工作机上运行 Duo |
| 运行其他工作流或远程任务子集 | 已准备好的 Linux 工作机、数据集和模型设置 | oc-eval swe-v1-prolite · 操作指南 |
| 重新评分已保存的候选 | 绑定已有候选的队列计划 | oc-eval rejudge-queue · 恢复运行 |
| 为服务器准备安装后的运行代码 | 已安装的兼容 OC 与 OCE 软件包 | oc-eval package-runtime · 服务器评测指南 |
| 生成最终对比报告 | 两份完成的事实报告及其证据 | oc-eval final-report · 最终报告 |
通用任务文件和基准数据集使用不同字段，按任务格式选择匹配的命令。操作指南介绍模型配置，CLI 参考介绍其他求解器入口。
使用已有提交且工作区干净的 Git 仓库。将 /work/calculator 换成其绝对路径，并写清想完成的修改。确认第一条检查输出提交 ID，第二条没有输出。状态检查包含未跟踪文件。将自己的改动提交，或使用另一份干净的 checkout。
git -C /work/calculator rev-parse --verify HEAD
git -C /work/calculator status --short --untracked-files=all下面在目标仓库之外创建输入文件。
export EVAL_ROOT="$HOME/oc-evaluation/eval-data"
umask 077
mkdir -p "$EVAL_ROOT"
cat > "$EVAL_ROOT/tasks.jsonl" <<'TASKS'
{"task_id":"calculator-1","description":"Fix calculator.add and run its tests","repo_path":"/work/calculator","timeout":600,"max_tokens":100000}
TASKS文件保存为 $HOME/oc-evaluation/eval-data/tasks.jsonl。设置模型与 provider 后，在提示处输入 API key。使用 openai provider 时，oc-eval run 采用 Chat Completions。将 OPENCOLLAB_BASE_URL 设为支持这一 API 的 base URL。OPENCOLLAB_WIRE_PROTOCOL 无法将此命令切换为 Responses。Duo 教程提供另一套 Responses 配置。
export OPENCOLLAB_MODEL=your-model
export OPENCOLLAB_PROVIDER=openai
export OPENCOLLAB_API_KEY="$(python -c 'import getpass; print(getpass.getpass("API key > "))')"
oc-eval run "$EVAL_ROOT/tasks.jsonl" \
--output "$EVAL_ROOT/candidate-run" \
--concurrency 1 --timeout 600打开所选输出目录中的 results.jsonl。patch_produced 表示是否提取到了补丁，submission_eligible 表示候选是否可以继续评测。任务失败后，命令也可能以退出码 0 结束，因此应检查这些字段，并通过 error 查看原因。官方通过或失败判定来自对这份补丁实际执行的基准目标测试。快速入门介绍结果字段和数据集检查。
Duo 由两个编码角色和一个裁决角色选择候选，再交给 Eval 执行正式测试。默认 Base 角色配置会解析为 Single2。按照在一台 Linux 工作机上运行 Duo准备有序数据集、镜像、模型 relay 和配置文件，然后运行一题。
oc-eval duo --config "$EVAL_ROOT/duo.json" \
--indices 1 --workers 1 --run-id duo-smoke-001配置文件所在目录的 results/duo-smoke-001 目录中会生成 task_1_report.json，其中包含候选与正式测试报告的对应关系。resolved 通过判定需要正式目标测试实际执行，并有匹配证据。证据缺失、环境准备失败、候选提取无效会记为 technical_failure。操作指南介绍结果含义、批量执行和恢复步骤。oc-eval g22 是 Duo 的兼容命令别名。
文档索引按用途组织指南。参数见 CLI 参考，运行失败时看故障排查，评分规则见评测完整性。服务器评测指南介绍服务器运行和 provider 请求容量。
CONTRIBUTING.md介绍开发环境、仓库检查、配套 wheel 验证和评审方式。CHANGELOG.md记录版本变化。许可证为 MulanPSL-2.0，依赖声明见 THIRD_PARTY_NOTICES.md。