🇨🇳 中文版

🦅 Falcon: Enterprise-Grade Text-to-SQL Benchmark

A Comprehensive Chinese Text-to-SQL Benchmark for Complex, Cross-Domain Analytical Scenarios

Introduction | Dataset Structure | Getting Started | Citation

---

📊 数据集 dbId=14（玩具零售）使用指南

本数据集（dbId=14）为玩具零售行业场景，包含 toy_stores、toy_products、toy_sales、toy_inventory 等数据表。以下步骤说明如何将 Falcon 的 SQLite 格式数据导入 Doris 数据库，并基于 Doris 进行 ChatBI 测试。

步骤一：生成 Doris 建表 DDL 脚本

使用 tis_tool/falcon_tables_to_doris_ddl.py 脚本，从 Falcon 的 tables.json 中提取 dbId=14 的表结构定义，自动生成 Doris 兼容的建库/建表 DDL 语句。

# 生成 dbId=14 的 Doris DDL
python3 tis_tool/falcon_tables_to_doris_ddl.py --db-id 14

# 输出：doris_init.sql 文件
# 然后执行该脚本在 Doris 中创建数据库和表
mysql -h <fe_host> -P 9030 -u root < doris_init.sql

该脚本会将 Falcon 的原生数据类型（如 integer、text、real）映射为 Doris 对应的类型（如 BIGINT、STRING、DOUBLE），并自动创建以 falcon_14 命名的数据库。

步骤二：导出数据表为 CSV 文件

使用 tis_tool/sqlite_to_csv.py 脚本，将 dbId=14 的 SQLite 数据库文件导出为 CSV 格式，便于后续导入 Doris。

# 导出 dbId=14 的所有数据表为 CSV
python3 tis_tool/sqlite_to_csv.py 14

# 输出目录：./csv/14/，每个表对应一个 CSV 文件

步骤三：将 CSV 数据导入 Doris

使用 tis_tool/load_to_doris.py 脚本，通过 Doris Stream Load 接口将上一步导出的 CSV 文件批量导入 Doris 数据库。

# 将 dbId=14 的数据导入 Doris
python3 tis_tool/load_to_doris.py --db-id 14 --doris-fe <fe_host> --user root --pwd <password>

# 前置条件：已完成步骤一和步骤二

该脚本会自动读取 ./csv/14/ 目录下的所有 CSV 文件，逐表通过 Stream Load 方式写入 Doris 的 falcon_14 数据库中。

步骤四：进行 ChatBI 测试

数据导入完成后，参照 test_exam/14/test_exam.md 文档进行 ChatBI 测试。该文档包含了 32 个面向玩具零售场景的自然语言查询问题及其对应的 SQL 语句和预期执行结果，涵盖多表关联、窗口函数、CTE 子查询、聚合分析等多种复杂 SQL 模式。

通过以上四步，即可完成从 Falcon 原始数据到 Doris 的完整数据链路搭建，并基于 Doris 对 ChatBI 模型进行 Text-to-SQL 效果评估。

---

📖 Introduction

Falcon is a continuously evolving, high-quality benchmark designed to bridge the gap between academic Text-to-SQL datasets and real-world enterprise requirements. Unlike traditional benchmarks, Falcon focuses on MaxCompute/Hive dialects and stresses models with complex SQL patterns and linguistic ambiguities common in production environments.

Key Features

• SQL Complexity: Heavy focus on multi-table joins (77% of samples), nested CTEs, window functions, ranking, and type casting.
• Linguistic Challenges: Includes Chinese fuzzy time expressions, colloquial business jargon, ellipsis, and multi-intent questions.
• Enterprise Scale: Schemas involve denormalized fields, implicit foreign keys, and domain-specific synonyms.

The current release is built on curated public datasets covering Finance, Internet, and Retail domains.

---

📂 Dataset Structure

To facilitate robust evaluation, the Falcon benchmark is split into a Development Set (with ground truth) and a Test Set (blind).

Repository Layout

FALCON/
├── dev_data/                   # Development Set
│   ├── dev.json                # Questions, SQL, and Execution Results
│   ├── tables.json             # Schema definitions (PK/FK/Columns)
│   └── dev_databases/          # SQLite/CSV source files for execution
│
├── test_data/                  # Test Set
│   ├── test.json               # Questions ONLY (Ground truth hidden)
│   ├── tables.json             # Schema definitions
│   └── test_databases/         # SQLite/CSV source files
│
├── simple_agent/               # [NEW] Lightweight Evaluation Scripts
│   ├── comparator.py           # SQL execution result comparator
│   ├── utils.py                # Utilities for SQL extraction from LLM response
│   └── simple_benchmark.py     # Main script to run dev/test evaluation
│
├── submission/                 # [NEW] Submission Helpers & Examples
│   ├── example_submission_csv/ # Example CSV files for leaderboard submission
│   ├── example_submission_sql/ # Example SQL files for leaderboard submission
│   └── format_submission.py    # Helper to convert DB-GPT Excel output to Zip
│
└── README.md

Data Format Details

1. Development Data (dev_data/dev.json)

Used for few-shot prompting, fine-tuning, or debugging. Contains the natural language question, the ground truth SQL, and the expected execution result.

[
  {
    "question_id": "1",
    "dataset_id": "finance_01",
    "question": "每个性别的平均年龄是多少，按年龄排序？",
    "sql": "SELECT Gender, AVG(Age) FROM customers GROUP BY Gender ORDER BY AVG(Age)",
    "answer": {
      "Gender": ["Female", "Male"],
      "AvgAge": [27.73, 27.84]
    },
    "is_order": "0"
  }
]

2. Test Data (test_data/test.json)

Used for the official leaderboard. Only the question and schema reference are provided.

---

🚀 Getting Started

We currently provide two methods for evaluating your models on the Falcon benchmark: a lightweight script-based approach and a GUI-based approach via DB-GPT.

Method 1: Simple Agent (Script-based)

The simple_agent directory contains a lightweight evaluation pipeline. You can use simple_benchmark.py to run evaluations on either the development or test sets.

1. Clone the Repository

   git clone https://github.com/eosphoros-ai/Falcon.git
   cd Falcon

2. Setup Environment Ensure you have the necessary Python dependencies installed.

   pip install openai pandas tqdm

3. Run Evaluation

   • Development Set: Run the benchmark on the dev set to check performance against ground truth.

     cd simple_agent
     python simple_benchmark.py dev

   • Test Set: Run the benchmark on the test set to generate predictions.

     cd simple_agent
     python simple_benchmark.py test

     Note on Submission: After execution, a submission.zip will be automatically generated. For official leaderboard submission, a trace log (we recommend .jsonl format) is required. Please ensure you manually include your trace log in the final ZIP before submitting.

Method 2: DB-GPT (GUI-based)

Falcon is fully integrated into DB-GPT, allowing you to evaluate both Models (LLMs) and Agents through a visual interface.

1. Configuration & Execution Please refer to the official DB-GPT Evaluation Documentation for detailed steps on how to:

   • Import the Falcon benchmark dataset.
   • Configure your Model or Agent.
   • Run the evaluation pipeline via the "Models Evaluation" module.

2. Format Submission DB-GPT will generate an evaluation report in Excel (.xlsx) format. To submit your results to the Falcon leaderboard, you must convert this file into the required ZIP format using our helper script.

   # Run the formatting script
   python submission/format_submission.py --input <path_to_dbgpt_output.xlsx> --output submission.zip

   Note: The generated submission.zip will contain the required result_sql and result_csv folders formatted correctly for the leaderboard.

---

📤 Submission

Once you have generated your SQL queries (and execution results), please refer to the submission/ directory for format requirements.

• Examples: Check submission/example_submission_csv and submission/example_submission_sql for the expected file structure.
• Guidelines: Please refer to the Falcon Submission Guidelines for detailed rules.

---

📝 Citation

If you use Falcon in your research or development, please cite our paper:

@article{falcon2025,
  title={Falcon: A Comprehensive Chinese Text-to-SQL Benchmark for Enterprise-Grade Evaluation},
  author={Luo, Wenzhen and Guan, Wei and Yao, Yifan and Pan, Yimin and Wang, Feng and Yu, Zhipeng and Wen, Zhe and Chen, Liang and Zhuang, Yihong},
  journal={arXiv preprint arXiv:2510.24762},
  year={2025},
  url={https://arxiv.org/abs/2510.24762}
}

---

⚖️ License

This project is licensed under the Apache License, Version 2.0.
See the LICENSE file for the full text.

---

_{Maintained by Eosphoros AI}