SQLite NL2SQL Fine-Tuning & Evaluation Pipeline

A high-performance pipeline for fine-tuning Small Language Models (SLMs) on Text-to-SQL tasks and evaluating performance using SQLite execution accuracy and Gemini LLM Judge.

---

1. Setup

Local Environment

# Initialize virtual environment & install dependencies
python3 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt

Database & Dataset Initialization

# Seed 166 Spider SQLite databases & validation datasets
python data/setup_spider_databases.py

---

2. Training (LoRA & QLoRA)

Remote Training on Kaggle (T4 GPU)

Copy and run either script in a Kaggle notebook cell with GPU accelerator enabled:

• 16-bit LoRA: training/kaggle_train.py
• 4-bit QLoRA: training/kaggle_train_qlora.py

local Fine-Tuning

# Run local 16-bit LoRA training
python training/train_lora.py

---

3. Evaluation

Configure evaluation settings in config.json, then execute evaluations:

End-to-End Evaluation Flow

Executes reference query, generates queries for Base SLM, LoRA, and QLoRA, runs them in SQLite, compares output rows, and invokes the Gemini LLM Judge:

# Run full evaluation on custom hard dataset
python evaluation/execution_accuracy.py --dataset-path data/custom_hard_benchmark.json

# Run evaluation on Spider dev dataset
python evaluation/execution_accuracy.py --dataset spider_dev

• Detailed results are stored in execution_accuracy_results.json.

---

4. Evaluation Layers

Our pipeline implements a two-layer evaluation architecture to verify SQL accuracy:

Layer 1: SQLite Execution Verification

• Runs the model's generated query on the target .sqlite database and compares its output rows directly with the ground-truth reference query output.
• Verifies structural matches (e.g. EXACT_MATCH, BAG_MATCH, or ORDER_MISMATCH) and flags runtime exceptions (MODEL_SQL_ERROR).

Layer 2: Gemini LLM Judge

• Audits generated SQL against the schema to catch accidental execution matches (e.g., when queries return identical empty sets [] but are semantically incorrect, like using type mismatches or incorrect logic).
• Assigns a score from 0 to 10 and generates a detailed qualitative explanation of errors.

---

5. Benchmark Accuracy Results

Evaluating the fine-tuned weights against the base model on our validation sets yields the following performance benchmarks:

Model Variant	Execution Accuracy	Accuracy %	Avg Gemini Score (0-10)
LoRA Model (16-bit)	45/50	90.0%	9.2
QLoRA Model (4-bit)	42/50	84.0%	8.6
Base SLM (qwen2.5:1.5b)	40/50	80.0%	7.8