Website: SPIDER-KG
SPIDER-KG integrates normalized database entities and relations into a base knowledge graph. SPIDER then extracts entities, relations, and evidence from PubMed abstracts, scores the extracted triples, matches entities, and builds a literature layer. The daily update pipeline processes new, revised, and deleted PubMed records.
This repository provides the key construction algorithms, method descriptions, initial Neo4j import, and a standalone daily update pipeline. Users supply the data, model weights, and API credentials. Base graph construction starts from normalized node and edge files. Raw database crawlers, web application code, and downstream QA evaluation pipelines are outside the scope of this repository.
On Linux with Python 3.10 or later, clone the repository and run:
git clone https://github.com/MAGIC-AI4Med/SPIDER.git
cd SPIDER
python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e '.[test]'
./scripts/test.sh
./scripts/demo.shThe example uses synthetic XML and fixed extraction results, so it requires no API, Neo4j service, or model download. It runs XML parsing, scoring, name matching, evidence publication, SQLite graph projection, and CSV export. Each invocation writes to its own artifacts/demo/example_*/ directory. Fixed extraction results test the processing pipeline; they do not represent a live LLM response.
For live daily updates, also install python -m pip install -e '.[runtime]'. For the complete construction workflow, also install python -m pip install -e '.[construction]'. The C++ programs for historical text matching require a C++17 compiler and SQLite development libraries, such as g++ libsqlite3-dev on Ubuntu.
| Task | Documentation / entry point |
|---|---|
| Understand the algorithms, thresholds, and reproduction scope | METHODS.md |
| Read the prompt templates and their source correspondence | Plain-text prompts and prompt index |
| Reproduce base graph construction, extraction, matching, and the historical literature layer | BUILD.md |
| Prepare the separately distributed data, models, and databases | DATA.md |
| Configure APIs, import the initial graph, and start daily updates | DAILY.md |
| Understand configuration migration, dependency replacements, and relative paths | CONFIGURATION.md |
| Review validation coverage and limitations | VALIDATION.md |
All shell entry points change to their own repository root. The configuration's root is relative to the directory containing that configuration file; other data, model, and state paths are relative to root. You can move the repository without copying configuration from an older project. Shell scripts also support invocation by an absolute path; the source contains no paths tied to a specific machine.
config/ SPIDER algorithm settings and daily update templates
construction/build_kg/ Type preparation, entity merging, endpoint alignment, and review rules
construction/relations/ Relation mapping, conflict classification, and provenance attachment
construction/conditions/ Portable condition-workflow input builders
construction/coverage/ Historical corpora, semantic subgraphs, matching, and L1 construction
construction/vector_store/ Optional historical Milvus vector workflow
provenance/ Adapters for recovering source evidence
prompt/ Plain-text prompt messages, one file per message
src/pubmed_daily/ Standalone daily ingestion, state management, matching, and publication
vendor/ SPIDER extraction stages, scoring, and shared algorithms
scripts/ Configuration, execution, import, and validation entry points
tests/ Unit/integration tests and synthetic fixtures
figure/ Workflow figure, PNG preview, and original PDF
data/ Separately obtained data; not committed
models/ User-downloaded model weights; not committed
artifacts/ Daily state, exports, and experiment outputs; not committed
See THIRD_PARTY_NOTICES.md for code origins and the scope of modifications. The license remains to be specified by the maintainers. Data will be released separately; see the data documentation for download and distribution information.