A web interface (NiceGUI) to explore search results of an impact-index BOW index (BM25) against datamaestro-ir collections:
- pick topics from datamaestro IR (Adhoc) datasets, or type any query; for topics with several fields (TREC: title, description, narrative), choose the field used as query and look at all of them;
- see the top-K documents with matching terms highlighted, each marked relevant / non-relevant / unjudged (filterable), plus nDCG@K, RR@K, AP, P@K, R@K; query terms are shown in their index (stemmed) form with their document frequency;
- edit a topic's query and compare with the original: metric differences, and the original rank of every document ("was #3", "new");
- switch the index's stop list on or off for a query ("Remove stop
words"): stop words only match if the index kept them (e.g. built with
--pipeline terrier-pisaor--stop-words none); - rewrite queries with language models (e.g. the STORM query expander
Arthur-75/storm-qwen3-8B) registered in the settings, and compare the rewritten query with the original; - see and select the saved versions of a topic;
- evaluate every topic of a dataset at once, and sort topics by any metric (worst first); results are stored in the workspace, and saved reformulations of the topics are evaluated alongside;
- list the relevant documents that were not retrieved in the top-K, with their rank within the search depth when retrieved deeper;
- save queries, each linked to a collection (and optionally to its topic);
- see what an index is made of (ⓘ next to the collection): kind and
codecs, positions, value type, size, and text analysis — the build
pipeline (
pyserini,terrier,terrier-pisa) is recognized from the analyzer, with what was overridden (e.g. the stemmer); - write structured queries (
#combine,#syn,#band,#1,#uwN) with a live parser that points at syntax errors and shows how every word resolves (indexed term, stop word, unknown) and which operators are dropped. The help button (?) next to the query documents the syntax.
uv sync
uv sync --extra rewriters # local query rewriters (torch, transformers)
uv run pre-commit install # ruff + conventional commit messagesEverything lives in a workspace folder:
<workspace>/
workspace.json collections: index path, documents, datamaestro datasets
saved-queries.json saved queries
evaluations/ stored "Evaluate all" results
builds/ index builds (state, lock and log of each)
indexes/ default output of index builds
The settings dialog checks everything as it is edited: index, documents (and that they match the index), and each dataset (topics, assessments, and whether its assessed documents belong to the collection). Checks never download anything.
Both files are edited from the interface (gear icon): add a collection by giving its index path, then add the datamaestro IR datasets that provide topics and assessments. A dataset id typed in the topic panel is also added to the collection.
# Index a datamaestro collection (BOW index + document store) and register
# it in a workspace; --positions enables #1 and #uwN queries, --stop-words
# lucene|terrier|none|<file> sets the stop list (default: the pipeline's)
uv run impact-explorer index com.microsoft.msmarco.passage.documents \
data/msmarco-passage --positions \
--workspace ws --dataset com.microsoft.msmarco.passage.dev.small
# Start the interface
uv run impact-explorer serve ws --port 8080Rewriters are registered in the settings (gear icon, "Rewriters"). A
rewriter has a prompt (system prompt and user template with {query}),
generation parameters, and how the searched query is built from the
outputs:
- Generated only (
{outputs}): the raw outputs, concatenated — STORM's setting (it is trained with its outputs alone as the BM25 query, and repeats the query terms itself); - Original + generated (
{query} {outputs}); - Original + unique keywords (
{query} {keywords}): each keyword once; - a custom template with
{query},{keywords}and{outputs}.
Repeated words add up in BM25 (as in Lucene), so repetitions in the outputs
weigh more; the original query weight repeats {query}. The rewrite
panel can change both without generating again. Naming a rewriter after a
known model (e.g. Arthur-75/storm-qwen3-8B) fills in its preset.
Empty fields defer to the model: with an empty system prompt and {query}
as user template, the model's chat template builds the prompt, and the
generation parameters missing from the JSON come from its
generation_config.json ("Model's own" button). The STORM preset relies on
this: its repositories hold the prompt (system prompt, [QUERY]: …
wrapping) and the decoding (beam search, 3 outputs).
transformersbackend: the model runs locally (cuda, mps or cpu); it is downloaded and loaded the first time it is used.openaibackend: any OpenAI-compatible server, e.g.vllm serve Arthur-75/storm-qwen3-8B, with its URL (http://host:8000/v1).
Group beam search (num_beam_groups=3, diversity_penalty=1.0: STORM's
best setting, slightly above the default plain beam search) is no longer
part of transformers: it runs code from the Hub
(transformers-community/group-beam-search), so it is an explicit opt-in
("Group beam search" in the rewriter's settings). With the openai
backend, only the parameters set in the JSON are sent (plus greedy
decoding unless do_sample is true): vLLM does not do beam search through
the chat API.
A saved query can be opened directly with /?saved=<id>, and a collection
with /?collection=<name>.
An index can also be built from the interface (settings, "Index builds"): give a name, a datamaestro documents dataset and the topic datasets, and choose the index options:
- pipeline (
pyserini,terrier,terrier-pisa) and stop words; - positions, needed for phrase (
#1) and window (#uwN) queries; - compression: bitpacked block-max postings (lossless, faster search); the uncompressed index is deleted once compressed.
When the build completes, the collection is added to the workspace under that name.
The documents and topic datasets are drop-downs (type to filter) listing
the installed datamaestro repositories, read from the dataset definitions
without downloading anything: collections are the datasets whose type is
a datamaestro_ir Documents, and their topics the Adhoc datasets that
reference them (selecting documents fills those in). An Adhoc dataset
that bundles its own documents (most BEIR datasets) is listed as a
collection of its own. Any other dataset id can still be typed.
A build runs in its own process (impact-explorer build <workspace> <name>): it goes on if the interface stops, and the interface shows its
progress again when restarted. Resuming a failed, cancelled or interrupted
build skips the completed stages:
- Download documents (datamaestro);
- Document store: checkpointed every 50,000 documents, so an interrupted copy resumes from its last checkpoint. Skipped when datamaestro already stores the documents in an impact-index document store (e.g. BEIR): that store is used as is, and the collection reads its documents through datamaestro;
- BOW index, built from the document store: an interrupted index restarts from the beginning of this stage (impact-index does not checkpoint the vocabulary of BOW indexes), but nothing is downloaded or copied again;
- Compression (optional).
"Rebuild…" builds again, possibly with other index options; by default only the index is rebuilt (the documents are not copied again). "Remove" forgets the build and deletes its files, unless its collection is in the workspace. Removing a collection deletes its index and document store (proposed by default for files inside the workspace, unchecked for indexes elsewhere); datamaestro's own files are never deleted.
A collection needs an impact-index BOW index (built with
BOWIndexBuilder, so that it has an analyzer) and a source of document
texts:
- a datamaestro-ir document store dataset, indexed in its iteration order
(e.g.
com.microsoft.msmarco.passage.documents), or - an impact-index
DocumentStorewhose document numbers are the index docids (default: adocstorefolder inside or next to the index), with JSON ({"title": …, "text": …}) or plain-text contents; a store without keys uses document numbers as ids.
Index docids are mapped to external ids through the document store; qrels use external ids.
uv run pytest
uv run pre-commit run --all-filesNotes:
- The structured query parser (
query_syntax.py) mirrors impact-index's own (src/query.rs) to give positioned feedback; searching always goes through impact-index, whose errors are reported as is. - impact-index does not expose its vocabulary: index forms of query terms are recovered with the index's stemmer and checked against the analyzer; forms that cannot be checked (Porter is not idempotent) are shown in italics.