Skip to content
DBHammerPublic

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Griffin logo

Customizable attribute--vector data and FANNS query workload generation

Python Artifact Backends CLI

Install · Quick start · Configuration · Outputs · Layout


Griffin is the generator behind the ICDE submission Griffin: Customizable Attribute--Vector Data and Query Workload Generation for Filtered ANNS Benchmarking. It produces, from a single YAML file per stage, (i) attribute--vector datasets whose cross-modal dependency rho is user-controlled and (ii) filtered-ANNS query workloads whose selectivity and query--filter correlation (GLS) hit specified targets.

No external DBMS is needed: datasets are written to Parquet/CSV plus metadata, and queries are validated against an exact FAISS index built from the generated vectors.

Why Griffin

Stage What a benchmark needs What Griffin does
Data Datasets with the same schema but different attribute--vector dependencies, so that performance is not tied to one layout Attribute groups + vector regions, joined by a capacity-constrained assignment that reaches a target rho
Query Workloads whose hardness is an input rather than an outcome Deterministic key-based surrogates score candidates; predicate values and query vectors are searched until the target selectivity and GLS are met

Install

python -m venv .venv
source .venv/bin/activate
pip install -e .

or with uv:

uv sync

Requirements: Python >= 3.11 and the packages in pyproject.toml (faiss-cpu, numpy, scikit-learn, ortools, pandas, pyarrow, pyyaml, scipy, msgspec).

Quick start

# 1) attribute--vector dataset: 10K rows, 96-d vectors, 24 regions, target rho = 0.55
griffin --config configs/data_10k.yaml

# 2) 5 filtered-ANN queries on top of it
griffin --config configs/query_filtered_ann.yaml

Without installing, use the module form:

PYTHONPATH=src python -m griffin --config configs/data_10k.yaml

A run takes a few seconds on a laptop. The data stage logs the achieved dependency:

[data] 耦合质量:events.embedding~category target_rho=0.550, observed_rho=0.554, relative_error=0.79%

Scaling up to the paper settings is a matter of raising rows, dim, cluster_count, region_count, and spec.count (paper: N=1M rows, d=192, R=32, workloads of about 100 queries).

How it works

data config ──► attribute groups + vector regions ──► capacity-constrained assignment
                                                              │
                                                              ▼
                                      dataset  (parquet + metadata + FAISS index)

query config ─► surrogate scoring ──► predicate instantiation ──► region-guided query vector
                                                              │
                                                              ▼
                                       queries.sql + ground_truth.jsonl

Configuration

Data stage (configs/data_10k.yaml)

Field Meaning
type, data_model task type (data) and data model (relational)
seed RNG seed; identical seeds reproduce identical datasets
output_dir output root
output_format parquet or csv
dialect SQL dialect for schema.sql (currently postgresql)
spec.tables[].rows row count
spec.tables[].columns[] int/text/double with a distribution, or vector with vector_spec
vector_spec.dim, metric, normalize vector dimension, distance metric, normalization
vector_spec.cluster_count, cluster_size_distribution, cluster_spread vector cluster structure
vector_spec.region_count number of vector regions used for dependency control
vector_attribute_couplings[].target_rho target attribute--vector dependency strength

Query stage (configs/query_filtered_ann.yaml)

Field Meaning
input_dir dataset produced by the data stage
spec.family filtered_ann
spec.count number of queries to instantiate
spec.form.predicate_count, operators, predicate_fields predicate template space
spec.constraints[].metric/min/max target global selectivity range
spec.output.kind, top_k ranked output and K
spec.search.max_attempts_per_query instantiation budget

Outputs

outputs/data_10k/
├── schema.sql
├── data/tables/events.parquet
└── metadata/
    ├── manifest.json
    ├── statistics/tables/events.json
    ├── deterministic/tables/events.json
    └── artifacts/vector_index/       # exact FAISS index for top-K ground truth

outputs/query_filtered_ann/
├── queries.sql                       # one executable statement per query
└── ground_truth.jsonl                # filtered_ann only; _rowid answers

Repository layout

.
├── assets/logo.svg
├── configs/
│   ├── data_10k.yaml
│   └── query_filtered_ann.yaml
├── src/griffin/                      # core package
│   ├── adapters/                     # CLI entry point
│   ├── config/                       # typed config, load, compile
│   ├── data/                         # generation, regions, assignment
│   ├── dataset/                      # dataset reader, metadata, statistics
│   ├── dialects/                     # SQL rendering
│   ├── ir/                           # relational IR
│   ├── query/                        # selectivity, GLS, search, ground truth
│   └── vector_hardness_core.py
└── pyproject.toml

Paper mapping

Paper Code
Sec. 4, attribute--vector data generation src/griffin/data/ (vector_regions.py, vector_coupling.py)
Sec. 4, dependency control (assignment) src/griffin/data/entity_builders.py
Sec. 5, surrogate-based query generation src/griffin/query/ (predicate_cover_search.py, filtered_ann_band.py, vector_search.py)
Metadata / dataset reader src/griffin/dataset/

Technical report

The full version, including the proofs of the lemmas and theorems and the complexity analysis of both generation stages, is available as technical-report.pdf.

About

No description, website, or topics provided.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Used by

Contributors

Languages