Skip to content
lamng3Public

About

Official implementation for KROMA: Ontology Matching with Knowledge Retrieval and Large Language Models (ISWC 2025)

Topics

Resources

Stars

8 stars

Watchers

1 watching

Forks

Repository files navigation

🧭 KROMA

Ontology Matching with Knowledge Retrieval and Large Language Models

ISWC 2025 arXiv Manual Python License: CC BY 4.0

⚡ Quick start · ✨ What it is · 📊 Results · 🔧 Usage · 📝 Citation

Lam Nguyen1     Erika Barcelos1     Roger French1     Yinghui Wu1
1 Case Western Reserve University

Note

Published at ISWC 2025 (Research Track, LNCS). The full version, with every table, is in this repository.

⚡ Quick start

git clone https://github.com/lamng3/kroma.git
cd kroma
./setup.sh
source .venv/bin/activate
kroma --help

kroma --help needs no model. A matching run needs the model backends and an API key:

uv pip install ".[models]"
cp .env.example .env          # TOGETHERAI_API_KEY or OPENAI_API_KEY

kroma run \
  --method_config kroma_scibert_envo_sweet \
  --llm meta-llama/Llama-3.3-70B-Instruct-Turbo-Free \
  --baseline

KROMA requires Python 3.12 and uv. The benchmark data is downloaded separately; see Datasets under Usage.

✨ What it is

KROMA matches concepts across two ontologies. For each candidate pair it retrieves the concepts' local context, asks a large language model whether the two refer to the same thing, and refines the alignment as answers arrive.

  1. Load. A track's alignment CSVs become a source graph, a target graph, and labeled source-target pairs.
  2. Retrieve. Each concept is embedded with SciBERT text vectors fused with Node2Vec graph vectors. Its context comes from the ontologies' dictionaries, the ontology graph, ConceptNet, and SPARQL endpoints for NCIT, DOID, and DBpedia.
  3. Decide. A language model answers yes or no with a confidence, alone or as a multi-agent debate.
  4. Refine. Accepted matches update the equivalence classes as they arrive (online refinement), and the whole graph is re-collapsed every 5 pairs (offline refinement). Pairs refinement cannot settle go to an expert for review.

📊 Results

Track KROMA Best baseline
Mouse–Human 94.94 92.20 MILA
NCIT–DOID 98.63 94.80 MILA
Nell–DBpedia 97.08 96.00 OLaLa
YAGO–Wikidata 95.54 92.19 LLM4OM
ENVO–SWEET 93.18 GPT-4o-mini 83.70 MILA
MI–MatOnto 61.25 32.97 MILA

F1 (%). KROMA with Llama-3.3-70B unless noted. Baselines are MILA, OLaLa, and LLM4OM with their reported models; each row shows the best of them · Table 3 of full.pdf

KROMA has the highest F1 on all six tracks. Removing knowledge retrieval or ontology refinement lowers F1 on every track reported, and with the small Qwen2.5-1.5B it still beats the best baseline on NCIT–DOID, YAGO–Wikidata, and MI–MatOnto.

Ablations and other models
Track KROMA without knowledge retrieval without refinement
Mouse–Human 94.94 91.25 86.59
NCIT–DOID 98.63 95.01 90.91
Nell–DBpedia 97.08 94.26 93.58
YAGO–Wikidata 95.54 91.98 89.74
MI–MatOnto 61.25 59.95 55.00

F1 (%), Llama-3.3-70B · Table 3 of full.pdf

Track Qwen2.5-1.5B GPT-4o-mini Llama-3.3-70B
Mouse–Human 83.6 92.0 94.9
NCIT–DOID 97.4 97.6 98.6
Nell–DBpedia 95.0 95.7 97.1
YAGO–Wikidata 95.5 95.4 95.5
ENVO–SWEET 79.8 93.2 92.0
MI–MatOnto 61.3 60.9 62.2

F1 (%) · Table 4 of full.pdf

Cost: tokens, calls, and time
Track Tokens sent vs full pairwise: Qwen2.5 GPT-4o-mini Llama-3.3 Speedup, offline vs online refinement
Mouse–Human 3.06% 2.88% 6.76% 7.17×
NCIT–DOID 54.48% 54.63% 54.48% 2.33×
Nell–DBpedia 26.09% 25.76% 26.09% 5.63×
YAGO–Wikidata 21.86% 21.93% 22.01% 18.76×
ENVO–SWEET 6.99% 6.82% 6.99% 7.00×
MI–MatOnto 43.04% 41.66% 41.81% 13.31×
Overall 25.92% 25.61% 26.36% 9.03×

Table 9 of full.pdf

Model Total tokens Output tokens Tokens per call
Llama-3.3-70B 480K 15K 470
DR1D-Llama-3-70B (deep reasoning) 620K 130K 605

All calls of one run · Table 10 of full.pdf

ENVO–SWEET, 1,000 candidate pairs Seconds
LLM inference, single agent 10,916
Offline refinement 68
Online refinement 14

§ KROMA Runtime of full.pdf

Retrieval sends the model about a quarter of what full pairwise matching would, and under 7% on the smaller ontologies; inference, not refinement, is where the time goes. New runs write a cost block next to their metrics (kroma.costs: API calls, tokens per call, and dollars at list price for hosted models).

🔧 Usage

Datasets

Place the alignment CSVs and dictionary caches under experiments/dataset/. Each track's dataset config names its CSV folder, and experiments/configs/dictionary.json names each dictionary cache. These paths are gitignored.

Track Config CSV folder
Mouse-Human mouse_human experiments/dataset/OAEI/Anatomy
NCIT-DOID ncit_doid experiments/dataset/OAEI/BioLLM
Nell-DBpedia nell_dbpedia experiments/dataset/OAEI/CommonKG
YAGO-Wikidata yago_wikidata experiments/dataset/OAEI/CommonKG
ENVO-SWEET envo_sweet experiments/dataset/OAEI/Biodiv
MI-MatOnto mi_matonto experiments/dataset/OAEI/MSE
Methods and models

A method config is a JSONL file at experiments/configs/method/<llm>/<method_config>.jsonl. It names the model backend (openai or togetherai), the embedding model, the track, and the context sources to query. Configs are included for Llama 3.3 70B, Llama 3.1 70B, Llama 3.2 3B, Llama 2 70B, DeepSeek-R1-Distill (Llama 70B and Qwen 1.5B), GPT-4o-mini, Gemma 2B, and Mistral 7B, plus debate configs such as kroma_scibert_mouse_human_debate_2A_3R (2 agents, 3 rounds).

Together models read TOGETHERAI_API_KEY, and OpenAI models read OPENAI_API_KEY, from .env.

Running an experiment

scripts/run_envo_sweet.sh runs the quick-start command.

Flag Effect
--method_config Method config name, without .jsonl (required)
--llm Model folder under experiments/configs/method/ (required)
--size Fraction of pairs to score: xsmall (20%), small, medium, large, or full (default)
--reasoning Ask the model to reason before it answers
--debate Decide each pair by a multi-agent debate
--active_learning Accept an answer only when a confidence-based acceptance score is positive
--baseline Write results under results/baseline/ instead of results/results/

The same flags work with python -m main. A run skips pairs it has already scored, so an interrupted run resumes where it stopped. From Python, kroma.pipeline.MatchingRun(args).execute() takes the same arguments and returns the metrics dictionary.

Output Contents
results/baseline/<track>/<model>_<track>.jsonl One accepted prediction per line
results/baseline/<track>/<model>_<track>.metrics.json Precision, recall, F1, token counts and API calls, and refinement timings
reviews/baseline/<track>/expert_queries.csv Pairs sent for expert review
logs/kroma.log Run log
Extending KROMA
To add Use
A chat or embedding model provider register_chat_backend, register_embedding_backend
A context source, such as a dictionary or SPARQL endpoint register_query_strategy
An alignment file format subclass OntologyLoader
A judging procedure subclass InferenceStrategy
Repository layout and development
src/kroma/         Installable package
  cli.py           kroma run
  pipeline.py      MatchingRun: prepare, score pairs, write metrics
  tasks/           Loaders, prompt builders, context sources
  agents/          Single-model and debate inference
  inference/       Chat and embedding backend factory
  algorithms/      Node2Vec fusion, online and offline refinement
  repository.py    Prediction JSONL and expert-review CSV
experiments/       Dataset and method configs (data is downloaded separately)
scripts/           Reproduction scripts
docs/              Documentation pages
full.pdf           Full version of the paper
uv pip install ".[dev]"
pytest -q

📝 Citation

@inproceedings{nguyen_2025_kroma,
    title={KROMA: Ontology Matching with Knowledge Retrieval and Large Language Models},
    author={Lam Nguyen and Erika Barcelos and Roger French and Yinghui Wu},
    booktitle={The Semantic Web -- ISWC 2025: 24th International Semantic Web Conference, Proceedings, Part I},
    series={Lecture Notes in Computer Science},
    publisher={Springer},
    pages={629--649},
    year={2025},
    doi={10.1007/978-3-032-09527-5_34},
}

License

CC BY 4.0. Copyright (c) 2025 Lam Nguyen, Erika Barcelos, Roger French, and Yinghui Wu.

About

Official implementation for KROMA: Ontology Matching with Knowledge Retrieval and Large Language Models (ISWC 2025)

Topics

Resources

Stars

8 stars

Watchers

1 watching

Forks

Releases

Packages

Used by

Contributors

Languages