DeepSeek-V4-Pro benchmarking example (vLLM + SGLang)

.gitignore

@@ -190,6 +190,8 @@ outputs/
 # `data/` rule above would otherwise exclude them.
 !examples/07_DeepSeekR1_Example/data/
 !examples/07_DeepSeekR1_Example/data/deepseek_r1_eval.parquet
+!examples/10_DeepSeekV4Pro_Example/data/
+!examples/10_DeepSeekV4Pro_Example/data/perf_stub.jsonl
 
 # Example vLLM virtualenv
 examples/03_BenchmarkComparison/vllm_venv/

examples/10_DeepSeekV4Pro_Example/README.md

@@ -0,0 +1,167 @@
+# DeepSeek-V4-Pro Benchmark (SGLang)
+
+Pass@1 accuracy suite for `deepseek-ai/DeepSeek-V4-Pro` on SGLang (ROCm / MI35x):
+**AIME25 (I+II) → GPQA → LiveCodeBench**.
+
+Validated client/server recipe used for the measured scores below:
+
+| Setting | Value |
+| ------- | ----- |
+| Client concurrency | `target_concurrency: 64` / `num_workers: 64` |
+| `max_new_tokens` | `256000` |
+| Server `CONC` | `64` (`--max-running-requests` / cuda-graph max bs) |
+| TP | `8` |
+| Image | `rocm/mlperf-inference:v0.5.16-rocm720-mi35x-20260803_prs` |
+| API | `/v1/chat/completions` (`api_type: openai`), streaming on |
+| Chat template kwargs | `thinking: true`, `reasoning_effort: max` |
+
+### Measured pass@1 (this recipe)
+
+| Dataset | pass@1 | Samples |
+| ------- | ------ | ------- |
+| AIME25 (I+II) | 100.00% | 30/30 |
+| GPQA | 90.91% | 198/198 |
+| LiveCodeBench | 94.22% | 994/1055 |
+
+## Layout (files needed for this suite)
+
+```text
+examples/10_DeepSeekV4Pro_Example/
+├── start_sglang_server.sh              # launch SGLang (host or Docker)
+├── docker_common.sh                    # shared docker/log helpers
+├── chat_templates/deepseek_v4_thinking.jinja
+├── data/perf_stub.jsonl                # tiny performance stub (required by online mode)
+├── sglang_deepseek_v4_pro_aime_pass1.yaml
+├── sglang_deepseek_v4_pro_gpqa_pass1.yaml
+├── sglang_deepseek_v4_pro_lcb_pass1.yaml
+├── run_sglang_accuracy_benchmark.sh    # AIME → GPQA → LCB
+├── start_lcb_service.sh                # optional containerized LCB scorer
+└── README.md
+```
+
+Accuracy datasets download from HuggingFace (`aime25::deepseek_v4`, `gpqa::deepseek_v4`,
+`livecodebench::deepseek_v4`). GPQA is gated — set `HF_TOKEN`.
+
+## Environment
+
+```bash
+export HF_HOME=<writable HF cache, e.g. $PWD/results/hf_cache>
+export HF_TOKEN=<your HuggingFace token>   # required for GPQA
+export ALLOW_LCB_LOCAL_EVAL=true           # default for LCB local scoring
+
+# Model path used by the pass@1 YAMLs / tokenizer metrics:
+export MODEL_PATH=/data/workloads-inference/hf_hub_cache/models--deepseek-ai--DeepSeek-V4-Pro/snapshots/b5968e9190ef611bbf34a7229255be88a0e937c1
+export TOKENIZER_MODEL_PATH=${MODEL_PATH}
+```
+
+If your checkout lives under a different HF cache root, update `model_params.name` in the
+three `*_pass1.yaml` files (and `TOKENIZER_MODEL_PATH`) to match `GET /v1/models`.
+
+## Launch server (CONC=64)
+
+```bash
+export MODEL_PATH=/data/workloads-inference/hf_hub_cache/models--deepseek-ai--DeepSeek-V4-Pro/snapshots/b5968e9190ef611bbf34a7229255be88a0e937c1
+export HF_CACHE_ROOT=/data/workloads-inference/hf_hub_cache   # so snapshot→blob symlinks resolve in Docker
+export RUN_MODE=docker
+export SGLANG_IMAGE=rocm/mlperf-inference:v0.5.16-rocm720-mi35x-20260803_prs
+export SGLANG_PORT=30000
+export TP=8
+export CONC=64
+export ISL=8192
+export MAX_MODEL_LEN=327680
+./examples/10_DeepSeekV4Pro_Example/start_sglang_server.sh
+```
+
+On a host with the `_prs` SGLang build installed, omit `RUN_MODE=docker`.
+
+| Variable | Default | Description |
+| -------- | ------- | ----------- |
+| `CONC` | `512` in script; **use `64` for this suite** | `--max-running-requests` / `--cuda-graph-max-bs` |
+| `TP` | `8` | Tensor parallel size |
+| `ISL` | `8192` | Chunked-prefill size |
+| `MAX_MODEL_LEN` | `327680` | `--context-length` (≥ 256k generations) |
+| `HF_CACHE_ROOT` | _(unset)_ | Mount HF cache repo root so snapshot blob symlinks work |
+| `SGLANG_IMAGE` | `…20260803_prs` | Docker image when `RUN_MODE=docker` |
+
+Read-only HF snapshots: the launcher patches `model_type` to `deepseek_v3` when writable,
+otherwise passes `--json-model-override-args '{"model_type":"deepseek_v3"}'`.
+
+Verify:
+
+```bash
+curl -sf http://127.0.0.1:30000/health || curl -sf http://127.0.0.1:30000/v1/models
+```
+
+## Run pass@1 suite
+
+Full suite (AIME → GPQA → LCB):
+
+```bash
+export HF_TOKEN=<token>
+export ALLOW_LCB_LOCAL_EVAL=true
+export WAIT_FOR_SGLANG_S=120
+./examples/10_DeepSeekV4Pro_Example/run_sglang_accuracy_benchmark.sh
+```
+
+Or one dataset at a time:
+
+```bash
+uv run inference-endpoint benchmark from-config \
+  -c examples/10_DeepSeekV4Pro_Example/sglang_deepseek_v4_pro_aime_pass1.yaml \
+  --timeout 1209600 --mode both
+
+uv run inference-endpoint benchmark from-config \
+  -c examples/10_DeepSeekV4Pro_Example/sglang_deepseek_v4_pro_gpqa_pass1.yaml \
+  --timeout 1209600 --mode both
+
+uv run inference-endpoint benchmark from-config \
+  -c examples/10_DeepSeekV4Pro_Example/sglang_deepseek_v4_pro_lcb_pass1.yaml \
+  --timeout 1209600 --mode both
+```
+
+Scores land under:
+
+```text
+results/sglang_deepseek_v4_pro_aime_pass1/accuracy/accuracy_results.json
+results/sglang_deepseek_v4_pro_gpqa_pass1/accuracy/accuracy_results.json
+results/sglang_deepseek_v4_pro_lcb_pass1/accuracy/accuracy_results.json
+```
+
+## LiveCodeBench scoring
+
+**Local (default for this suite):** `ALLOW_LCB_LOCAL_EVAL=true` runs the LCB scorer as a
+subprocess. Ensure test cases exist under `dataset_cache/livecodebench/release_v6/` (the
+accuracy helper regenerates them if missing).
+
+**Optional container:** `./start_lcb_service.sh` (needs `docker login dhi.io`). WebSocket
+scoring is preferred when the service is up; otherwise the client falls back to local eval.
+
+## Troubleshooting
+
+**Cannot connect to SGLang**
+
+- `curl http://127.0.0.1:30000/health` / `/v1/models`
+- Confirm `CONC=64` server is the process bound to port 30000
+
+**`address already in use` on `--port`**
+
+- Do not leave `SGLANG_PORT` set in the SGLang process environment (ZMQ conflict).
+  `start_sglang_server.sh` passes `--port` and runs with `env -u SGLANG_PORT`.
+
+**Model load / unknown architecture**
+
+- Expect `model_type` override to `deepseek_v3` (in-place patch or JSON override)
+- Use the `_prs` image so baked DSv4 patches are present
+
+**GPQA download fails**
+
+- Export a valid `HF_TOKEN` (gated dataset)
+
+**OOM / SIGABRT at high concurrency**
+
+- Keep server `CONC` and client `target_concurrency` at **64** for 256k-token runs
+- Higher concurrency has aborted the dsv4 scheduler when the KV pool saturates
+
+**Snapshot symlinks broken in Docker**
+
+- Set `HF_CACHE_ROOT` to the HF cache root that contains `models--deepseek-ai--DeepSeek-V4-Pro`

examples/10_DeepSeekV4Pro_Example/chat_templates/deepseek_v4_thinking.jinja

@@ -0,0 +1,9 @@
+{%- if add_generation_prompt is not defined -%}{%- set add_generation_prompt = true -%}{%- endif -%}
+{{- '<｜begin▁of▁sentence｜>' -}}
+{%- for m in messages -%}
+{%- if m['role'] == 'system' -%}{{- m['content'] -}}
+{%- elif m['role'] == 'user' -%}<｜User｜>{{- m['content'] -}}
+{%- elif m['role'] == 'assistant' -%}<｜Assistant｜>{{- m['content'] -}}<｜end▁of▁sentence｜>
+{%- endif -%}
+{%- endfor -%}
+{%- if add_generation_prompt -%}<｜Assistant｜><think>{%- endif -%}

examples/10_DeepSeekV4Pro_Example/data/perf_stub.jsonl

@@ -0,0 +1 @@
+{"text_input":"hello"}

examples/10_DeepSeekV4Pro_Example/docker_common.sh

@@ -0,0 +1,50 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+#
+# Shared Docker helpers for DeepSeek-V4-Pro example scripts.
+
+# Writable log directory on the host (mounted into containers at /workspace).
+# Accuracy + server logs can grow large; default leaves headroom on the host.
+DOCKER_LOG_STORAGE_GB="${DOCKER_LOG_STORAGE_GB:-16}"
+
+ensure_docker_log_dir() {
+  local subdir="${1:-misc}"
+  LOG_DIR="${LOG_DIR:-${ENDPOINTS_DIR:-.}/results/docker_logs/${subdir}}"
+  mkdir -p "${LOG_DIR}"
+  export LOG_DIR
+}
+
+# Extra docker run args for a larger container writable layer (opt-in only).
+# Logs are written to the mounted host LOG_DIR; most hosts use overlay2 without xfs
+# pquota and reject --storage-opt.
+docker_storage_args() {
+  if [[ "${DOCKER_USE_LOG_STORAGE_OPT:-false}" == "true" ]]; then
+    # shellcheck disable=SC2207
+    echo --storage-opt "size=${DOCKER_LOG_STORAGE_GB}G"
+  fi
+}
+
+# Wait for an OpenAI-compatible or SGLang HTTP server (example script preflight).
+# Tries GET /health (SGLang native), then GET /v1/models (OpenAI compatibility).
+# Args: base_url [max_wait_seconds]
+wait_openai_compatible_server() {
+  local base="${1%/}"
+  local max_wait="${2:-0}"
+  local start
+  start=$(date +%s)
+  while true; do
+    for path in /health /v1/models; do
+      if curl --output /dev/null --silent --fail --max-time 5 "${base}${path}"; then
+        echo "Inference server ready (${base}${path})"
+        return 0
+      fi
+    done
+    if (( max_wait <= 0 )); then
+      return 1
+    fi
+    if (( $(date +%s) - start >= max_wait )); then
+      return 1
+    fi
+    sleep 2
+  done
+}

examples/10_DeepSeekV4Pro_Example/run_sglang_accuracy_benchmark.sh

@@ -0,0 +1,126 @@
+#!/usr/bin/env bash
+# Pass@1 accuracy suite for DeepSeek-V4-Pro (AIME25 → GPQA → LiveCodeBench).
+# Matches the validated recipe: target_concurrency=64, max_new_tokens=256000.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
+ENDPOINTS_DIR="${ENDPOINTS_DIR:-$(cd "${SCRIPT_DIR}/../.." && pwd)}"
+# shellcheck source=docker_common.sh
+source "${SCRIPT_DIR}/docker_common.sh"
+
+SGLANG_PORT="${SGLANG_PORT:-30000}"
+LCB_PORT="${LCB_PORT:-13835}"
+TIMEOUT="${TIMEOUT:-1209600}"
+export DOCKER_LOG_STORAGE_GB="${DOCKER_LOG_STORAGE_GB:-64}"
+export ALLOW_LCB_LOCAL_EVAL="${ALLOW_LCB_LOCAL_EVAL:-true}"
+
+AIME_CFG="${AIME_CFG:-${SCRIPT_DIR}/sglang_deepseek_v4_pro_aime_pass1.yaml}"
+GPQA_CFG="${GPQA_CFG:-${SCRIPT_DIR}/sglang_deepseek_v4_pro_gpqa_pass1.yaml}"
+LCB_CFG="${LCB_CFG:-${SCRIPT_DIR}/sglang_deepseek_v4_pro_lcb_pass1.yaml}"
+
+cd "${ENDPOINTS_DIR}"
+
+if [[ -z "${HF_TOKEN:-}" && -f "${HF_HOME:-${HOME}/.cache/huggingface}/token" ]]; then
+  HF_TOKEN="$(cat "${HF_HOME:-${HOME}/.cache/huggingface}/token")"
+  export HF_TOKEN
+fi
+if [[ -z "${HF_TOKEN:-}" ]]; then
+  echo "ERROR: HF_TOKEN is required (GPQA is a gated HuggingFace dataset)."
+  echo "  export HF_TOKEN=<your HuggingFace token>"
+  exit 1
+fi
+export HUGGING_FACE_HUB_TOKEN="${HF_TOKEN}"
+export HF_HOME="${HF_HOME:-${HOME}/.cache/huggingface}"
+
+if [[ -z "${TOKENIZER_MODEL_PATH:-}" ]]; then
+  export TOKENIZER_MODEL_PATH="${MODEL_PATH:-/data/workloads-inference/hf_hub_cache/models--deepseek-ai--DeepSeek-V4-Pro/snapshots/b5968e9190ef611bbf34a7229255be88a0e937c1}"
+fi
+
+ensure_docker_log_dir "accuracy_pass1"
+export LCB_DATASETS_DIR="${LCB_DATASETS_DIR:-${ENDPOINTS_DIR}/dataset_cache/livecodebench/release_v6}"
+
+echo "=== Pre-flight checks ==="
+
+SGLANG_BASE_URL="${SGLANG_BASE_URL:-http://127.0.0.1:${SGLANG_PORT}}"
+WAIT_FOR_SGLANG_S="${WAIT_FOR_SGLANG_S:-0}"
+
+if ! wait_openai_compatible_server "${SGLANG_BASE_URL}" "${WAIT_FOR_SGLANG_S}"; then
+  echo "ERROR: Inference server not reachable at ${SGLANG_BASE_URL}." >&2
+  echo "  Start SGLang: ${SCRIPT_DIR}/start_sglang_server.sh" >&2
+  echo "  If the server is slow to bind: export WAIT_FOR_SGLANG_S=120" >&2
+  exit 1
+fi
+
+_allow_lcb_local=false
+case "${ALLOW_LCB_LOCAL_EVAL:-}" in
+  true | 1 | yes | TRUE | YES) _allow_lcb_local=true ;;
+esac
+
+if [[ "${_allow_lcb_local}" == "true" ]]; then
+  export ALLOW_LCB_LOCAL_EVAL=true
+  echo "ALLOW_LCB_LOCAL_EVAL=true — LiveCodeBench uses local subprocess scoring"
+  if [[ ! -d "${LCB_DATASETS_DIR}/test_cases" ]]; then
+    echo "LiveCodeBench test_cases missing — regenerating dataset cache..."
+    uv run python -c "
+from pathlib import Path
+from inference_endpoint.dataset_manager.predefined.livecodebench import LiveCodeBench
+
+LiveCodeBench.generate(
+    Path('${ENDPOINTS_DIR}/dataset_cache'),
+    variant='release_v6',
+    force=True,
+    save_test_cases=True,
+)
+"
+  fi
+elif ! curl --output /dev/null --silent --fail "http://127.0.0.1:${LCB_PORT}/info"; then
+  echo "ERROR: lcb-service is not running on port ${LCB_PORT}."
+  echo "Either start it: ${SCRIPT_DIR}/start_lcb_service.sh (requires 'docker login dhi.io')"
+  echo "Or run without the container: export ALLOW_LCB_LOCAL_EVAL=true"
+  exit 1
+else
+  echo "lcb-service OK on port ${LCB_PORT}"
+fi
+
+echo "Log directory (host): ${LOG_DIR}"
+echo ""
+
+run_one() {
+  local label=$1
+  local config=$2
+  local logf="${LOG_DIR}/${label}_pass1.log"
+  echo "=== Running ${label} pass@1 ==="
+  echo "Config: ${config}"
+  local cmd=(
+    uv run inference-endpoint benchmark from-config
+    -c "${config}"
+    --timeout "${TIMEOUT}"
+    --mode both
+  )
+  echo "${cmd[*]}"
+  set +e
+  "${cmd[@]}" 2>&1 | tee "${logf}"
+  local rc=${PIPESTATUS[0]}
+  set -e
+  echo "${label}_EXIT=${rc} (log: ${logf})"
+  return "${rc}"
+}
+
+set +e
+run_one AIME "${AIME_CFG}"
+aime_rc=$?
+run_one GPQA "${GPQA_CFG}"
+gpqa_rc=$?
+run_one LCB "${LCB_CFG}"
+lcb_rc=$?
+set -e
+
+echo ""
+echo "=== Suite summary ==="
+echo "AIME_EXIT=${aime_rc} GPQA_EXIT=${gpqa_rc} LCB_EXIT=${lcb_rc}"
+echo "Reports:"
+echo "  results/sglang_deepseek_v4_pro_aime_pass1/accuracy/accuracy_results.json"
+echo "  results/sglang_deepseek_v4_pro_gpqa_pass1/accuracy/accuracy_results.json"
+echo "  results/sglang_deepseek_v4_pro_lcb_pass1/accuracy/accuracy_results.json"
+
+exit $(( aime_rc != 0 || gpqa_rc != 0 || lcb_rc != 0 ? 1 : 0 ))