💬 InstructTTSEval

InstructTTSEval is a comprehensive benchmark designed to evaluate Text-to-Speech (TTS) systems' ability to follow complex natural-language style instructions. The dataset provides a hierarchical evaluation framework with three progressively challenging tasks that test both low-level acoustic control and high-level style generalization capabilities.

• Data available at Huggingface
• Paper: InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems
• Evaluation script: InstructTTSEval/eval

Hugging Face dataset + evaluation package

The 30-sample English eval slice — with Qwen3-TTS-VoiceDesign generations and Qwen Omni judge scores — lives on the Hub at kosw/instructtts-qwen-eval, so nothing depends on this local folder.

The benchmark runs in two phases, and ships no TTS model of its own (reference/test models live under tests/):

1. Generate — pass the eval set through the TTS model you provide;
2. Judge — score the generated wavs with a Qwen3-Omni-30B-A3B-Thinking judge (talker-free, FP8 on vLLM).

pip install git+https://github.com/Kosw/InstructTTSEval.git@qwen-omni-eval

Phase 1 — generation (your model, plugged in as a callable; wavs are written as {id}_{APS|DSD|RP}.wav):

import instructtts_eval as itts

def my_tts(text, instruction, instruction_type, sample_id):
    ...  # run your TTS system
    return wav, sr

itts.generate(my_tts, out_dir="my_gen_wavs")   # resumable; instructions auto-fetched from the Hub

Phase 2 — judging. Serve the judge once (FP8 fits a single 48 GB GPU), then judge the folder:

pip install "vllm[audio]"
vllm serve tturing/Qwen3-Omni-30B-A3B-Thinking-FP8 \
    --served-model-name Qwen/Qwen3-Omni-30B-A3B-Thinking \
    --port 8901 --max-model-len 32768 --reasoning-parser qwen3

instructtts-eval --wav_dir my_gen_wavs --base_url http://localhost:8901/v1 --num_workers 8 --resume

out = itts.judge(wav_dir="my_gen_wavs", base_url="http://localhost:8901/v1", resume=True)
print(out["stats"]["macro_avg"])

The package itself is dependency-light (no torch): judging goes through the vLLM OpenAI-compatible API. Two optional extras:

• instructtts-eval[local] — in-process transformers judge, no server needed: instructtts-eval --wav_dir my_gen_wavs --model_name Qwen/Qwen3-Omni-30B-A3B-Instruct --quantization bnb4 (NF4 4-bit, ~20 GB VRAM).
• instructtts-eval[lighteval] — the CLI auto-detects lighteval and routes phase 2 through its standard pipeline; without lighteval (or with a version that predates the instructtts tasks) it falls back to the plain runner. Force either with --runner plain|lighteval.

The benchmark itself lives inside lighteval (src/lighteval/tasks/tasks/instructtts/ — huggingface/lighteval#1311): lighteval alone can run it by name, transformers-style, with no extra install:

INSTRUCTTTS_WAV_DIR=my_gen_wavs INSTRUCTTTS_BASE_URL=http://localhost:8901/v1 \
lighteval ... "instructtts:aps|0,instructtts:dsd|0,instructtts:rp|0"   # model = dummy; the judge scores

No-install alternative: hf download kosw/instructtts-qwen-eval judge.py --repo-type dataset --local-dir . then python judge.py --wav_dir my_gen_wavs --base_url http://localhost:8901/v1 (same code, published from instructtts_eval/judge.py).

Other entry points:

• Load the dataset: load_dataset("kosw/instructtts-qwen-eval", split="test")
• Rebuild the local layout (testdata/ + gen_wav/): python eval/fetch_from_hub.py --repo_id kosw/instructtts-qwen-eval
• Re-upload after changes (data or scripts): python hub/build_and_upload.py (--files_only to skip the parquet push)
• End-to-end test with a real TTS model (Qwen3-TTS VoiceDesign): python tests/target_qwen3_tts.py --out_dir gen_wav — explicitly test code, see tests/README.md

Citation

Please cite our paper if you find this work useful:

@misc{huang2025instructttsevalbenchmarkingcomplexnaturallanguage,
      title={InstructTTSEval: Benchmarking Complex Natural-Language Instruction Following in Text-to-Speech Systems}, 
      author={Kexin Huang and Qian Tu and Liwei Fan and Chenchen Yang and Dong Zhang and Shimin Li and Zhaoye Fei and Qinyuan Cheng and Xipeng Qiu},
      year={2025},
      eprint={2506.16381},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2506.16381}, 
}