A competition project exploring tool selection from Chinese user requests with a local Qwen3 model. The repository includes a direct tool prompt, a two-stage category-then-tool prompt, synthetic examples, and evaluation scripts.
core/demo_agent.pyimplements the direct and hierarchical prompting baselines. Both return a proposed function-call string; they do not execute device actions.config/tools_v0.jsondefines the available tools and arguments.evaluation/evaluate_agent.pycompares generated calls with labelled examples. It reports exact-string and function-name matches; these are different measures.core/agent.pyis an unfinished competition template. It is not the runnable baseline.
The checked-in logs/eval_results.json records one 50-case run: 17 exact-string matches and 47 function-name matches. These historical results depend on the model, hardware, prompts and evaluation data used at the time; they are not a general benchmark. A correct function name does not establish correct arguments or safe execution.
The baseline needs a locally available Qwen3-compatible model and PyTorch/Transformers. Install those packages for your hardware, then run from the repository root:
python main.py --agent direct --model-path /path/to/Qwen3-4B-Instruct-2507
python main.py --agent hierarchical --model-path /path/to/Qwen3-4B-Instruct-2507The --tools option can point to another tool-schema JSON file. Model weights are not included. The committed environment.yaml is a historical Linux AArch64 environment export with many unrelated packages; it is not a portable installation specification.
Run offline routing and parser checks without model weights:
python -m unittest discover -s tests -vTo evaluate a local model against the small checked-in set:
python -m evaluation.evaluate_agent --model-path /path/to/Qwen3-4B-Instruct-2507 --data data/eval_test_small10.jsonl --out results.json| Path | Purpose |
|---|---|
core/demo_agent.py |
Direct and two-stage prompt baselines |
config/tools_v0.json |
Tool descriptions and parameter metadata |
data/ |
Synthetic training and evaluation examples |
evaluation/ |
Evaluation and error-analysis scripts |
generators/ |
Historical data-generation experiments |
logs/ |
Historical evaluation artifacts |
The baseline extracts text between <tool> tags with a regular expression. It does not validate the call against the schema, check argument types, or execute the selected tool. The evaluation script retains preceding turns when a labelled case contains them, but the historical metrics predate this fix and should not be read as a validated multi-turn benchmark. Accelerator availability is detected in the code, but this repository does not contain verified cross-hardware deployment tests.