Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 9 additions & 22 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -235,38 +235,25 @@ tandem-eval rjob run --run-id <name> --benchmark osworld2 --tasks <task-id> \
bash scripts/run_experiment.sh \
--experiment kimi12-retry-20261030 \
--tasks 024,029,037,045,057,059,069,082,088,093,005,031 \
--concurrency 12 --slow-model kimi-k3 --slow-key-ids 1,2,3,4 \
--concurrency 12 --slow-model kimi-k3 \
--max-steps 500 --max-wall 14400
```

目录结构:`manifest.json`(实验参数与 git describe)、`batch.log`、
`drivers/<task>.driver.log` 和 `<experiment>-<task>/`(worker、模型、轨迹与结果)。
worker 的 adpc 挂载只读时仍使用 evobox 暂存,任务结束后立即归档到该实验目录。

集群内使用模板配置时,`api_key` 留空会使用默认号池,也可以手动填写自己的 `api_key`。

需要指定号池中的 key 时,在原启动命令后追加:

```bash
--slow-key-ids 1,2,3,4
```

也可以在所选慢模型条目的 `key_pool` 中配置,命令行参数优先:

```json
{"key_ids": [1, 2, 3, 4]}
```

编号对应凭据列表中的第 1、2、3、4 把 key,不是密钥内容:

- 不填编号或配置 `[]`:使用默认优先级。
- 填一个编号:只使用这一把。
- 填多个编号:这些 key 按同级轮询分配给 worker,正常请求固定使用分配到的 key;故障仅在指定集合内切换。

不同实验可指定不重叠的编号集合。这里均分的是 worker,不保证请求量严格相等;编号选择与手填完整 `api_key` 二选一。
慢模型支持自定义 OpenAI 兼容 API。在 `harness.slow_model.apis` 的条目中填写
服务地址 `base_url`、访问密钥 `api_key` 和模型名称 `model`。
客户端以非流式方式向 `<base_url>/chat/completions` 发送 POST 请求,
使用 `Authorization: Bearer <api_key>` 鉴权;`base_url` 可带尾部斜杠。
真实密钥只填写在被 gitignore 的 `infra/config.json`;模板中的密钥留空。

慢模型的调用纪律(`models/slow.py`):

- HTTP 408、429、5xx,以及 MiniMax 通过 HTTP 200 返回的临时故障,
最多调用 3 次(首次 + 2 次重试),优先遵循 `Retry-After`,
缺失时每次等待 30 秒;等待和请求均受任务剩余预算约束。其他 API 错误直接上报。
- 思考强度取所选 API 的配置;空正文(全留在思考通道)→ 下一次重试自动升档;
- 叙述文字包裹 JSON → 自动剥围栏取首个平衡 `{…}` 抢救;
- `finish_reason=length` → 明确报错,提示调大 `slow_max_tokens`。
Expand Down
2 changes: 1 addition & 1 deletion docs/CLUSTER_RUNTIME.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,7 +324,7 @@ worker 侧各段日志:
每次实验必须指定唯一名称(小写字母、数字、连字符),所有结果归档到
`rjob_logs/<experiment>/`:

- `manifest.json`:tasks、并发、模型、key_ids、预算、watch timeout、git describe;
- `manifest.json`:tasks、并发、模型、预算、watch timeout、git describe;
- `batch.log`:批次输出;
- `drivers/<task>.driver.log`:单个 rjob 生命周期日志;
- `<experiment>-<task>/`:worker、模型调用、轨迹、截图和评分结果。
Expand Down
12 changes: 0 additions & 12 deletions infra/config.example.json
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,6 @@
"base_url": "https://api.minimax.cn/v1",
"api_key": "",
"model": "MiniMax-M3.1-Flash",
"key_pool": {
"provider": "minimax",
"key_ids": [],
"credentials_file": "/mnt/shared-storage-gpfs2/evobox-share-gpfs2/xuyuanxun/secrets/tandem_api_pool/credentials.json",
"state_file": "/mnt/shared-storage-gpfs2/evobox-share-gpfs2/xuyuanxun/secrets/tandem_api_pool/state.json"
},
"max_tokens": 16384,
"timeout_seconds": 180,
"reasoning_effort": "max",
Expand All @@ -22,12 +16,6 @@
"base_url": "https://api.kimi.com/coding/v1",
"api_key": "",
"model": "k3.1-pro",
"key_pool": {
"provider": "kimi",
"key_ids": [],
"credentials_file": "/mnt/shared-storage-gpfs2/evobox-share-gpfs2/xuyuanxun/secrets/tandem_api_pool/credentials.json",
"state_file": "/mnt/shared-storage-gpfs2/evobox-share-gpfs2/xuyuanxun/secrets/tandem_api_pool/state.json"
},
"max_tokens": 16384,
"timeout_seconds": 120,
"reasoning_effort": null,
Expand Down
8 changes: 2 additions & 6 deletions scripts/run_experiment.sh
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,6 @@ TASKS=""
CONCURRENCY=4
SLOW_MODEL="kimi-k3"
FAST_MODEL=""
SLOW_KEY_IDS="1,2,3,4"
MAX_STEPS=500
MAX_WALL=14400
EXECUTION_MODE=stage
Expand All @@ -22,7 +21,7 @@ usage() {
Usage:
bash scripts/run_experiment.sh \
--experiment NAME --tasks ID1,ID2,... [--concurrency N] \
[--slow-model NAME] [--fast-model NAME] [--slow-key-ids 1,2,3,4] \
[--slow-model NAME] [--fast-model NAME] \
[--execution-mode step|stage|goal] [--stage-max-actions N] \
[--max-steps N] [--max-wall SECONDS] [--dry-run]

Expand All @@ -38,7 +37,6 @@ while [[ $# -gt 0 ]]; do
--concurrency) CONCURRENCY="$2"; shift 2 ;;
--slow-model) SLOW_MODEL="$2"; shift 2 ;;
--fast-model) FAST_MODEL="$2"; shift 2 ;;
--slow-key-ids) SLOW_KEY_IDS="$2"; shift 2 ;;
--max-steps) MAX_STEPS="$2"; shift 2 ;;
--max-wall) MAX_WALL="$2"; shift 2 ;;
--execution-mode) EXECUTION_MODE="$2"; shift 2 ;;
Expand Down Expand Up @@ -87,7 +85,7 @@ import sys
print(json.loads(Path(sys.argv[1]).read_text())["log"]["results_root"])
PY
)
export STAGING EXPERIMENT DRIVER_DIR TASKS CONCURRENCY SLOW_MODEL SLOW_KEY_IDS MAX_STEPS MAX_WALL
export STAGING EXPERIMENT DRIVER_DIR TASKS CONCURRENCY SLOW_MODEL MAX_STEPS MAX_WALL
export EXECUTION_MODE STAGE_MAX_ACTIONS FAST_MODEL
python3 - <<'PY'
import json
Expand All @@ -103,7 +101,6 @@ args = {
"concurrency": int(os.environ["CONCURRENCY"]),
"slow_model": os.environ["SLOW_MODEL"],
"fast_model": os.environ["FAST_MODEL"] or None,
"slow_key_ids": [int(item) for item in os.environ["SLOW_KEY_IDS"].split(",")],
"max_steps": int(os.environ["MAX_STEPS"]),
"max_wall_seconds": float(os.environ["MAX_WALL"]),
"execution_mode": os.environ["EXECUTION_MODE"],
Expand Down Expand Up @@ -149,7 +146,6 @@ run_task() {
--stage-max-actions "$STAGE_MAX_ACTIONS" \
--slow-model "$SLOW_MODEL" \
"${fast_args[@]}" \
--slow-key-ids "$SLOW_KEY_IDS" \
--max-steps "$MAX_STEPS" \
--max-wall "$MAX_WALL" \
--poll 60 \
Expand Down
14 changes: 14 additions & 0 deletions src/tandem/models/api_errors.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
"""API error classification, independent of credentials and request scheduling."""

_MINIMAX_TRANSIENT_CODES = {"1000", "1001", "1024", "1033", "2064", "2066"}


def classify_api_error(status: int, raw: dict) -> str:
"""Recognize transient HTTP errors and MiniMax's HTTP 200 error envelope."""
if status >= 500 or status in {408, 429}:
return "transient"
base = raw.get("base_resp")
if (status == 200 and isinstance(base, dict)
and str(base.get("status_code")) in _MINIMAX_TRANSIENT_CODES):
return "transient"
return "fatal"
211 changes: 0 additions & 211 deletions src/tandem/models/key_pool.py

This file was deleted.

Loading
Loading