Scientific Paper Summarization Benchmark

A comprehensive benchmarking framework for evaluating text summarization methods on scientific papers. It systematically compares 62 summarization approaches — from classical extractive algorithms (TextRank, word-frequency) through encoder–decoder models (BART, T5, PEGASUS) to modern large language models (GPT-4.1, Claude 4, Mistral, LLaMA, DeepSeek-R1, and more) — against expert-written reference summaries using a multi-metric evaluation suite.

The benchmark is designed for reproducibility: every LLM response is cached, model training-data cutoff dates are tracked automatically, and results are exported as interactive HTML reports with per-model breakdowns.

Key features

• Multi-provider LLM support — OpenAI, Anthropic, Mistral, HuggingFace, and Ollama via a unified API layer
• Rich evaluation metrics — ROUGE-1/2/L, BERTScore, METEOR, BLEU, semantic similarity (MPNet), and factual consistency (AlignScore, SummaC, FactCC, MiniCheck)
• Length-constrained generation — configurable min/max word counts with compliance tracking
• Training cutoff tracking — automatically collects model knowledge cutoff dates from the community LLM knowledge-cutoff dataset, HuggingFace model cards, and provider documentation
• Publication date enrichment — fetches paper publication dates from CrossRef to enable temporal analyses of model knowledge vs. paper novelty
• Interactive visualisations — per-metric box plots, radar charts, rank heatmaps, correlation matrices, and sortable HTML tables

Quick Start

This project uses uv for package management.

1. Clone this repository
2. Install dependencies

   uv sync

3. Install AlignScore-large

   mkdir -p Output/llm_summarization_benchmark
   cd Output/llm_summarization_benchmark
   wget https://huggingface.co/yzha/AlignScore/resolve/main/AlignScore-large.ckpt
   cd ../..

   Note: If wget fails, download AlignScore-large manually from github.com/yuh-zha/AlignScore (download provided in section "Checkpoints") and place it in Output/llm_summarization_benchmark/.

4. Pre-download all models (the benchmark runs in offline mode)

   cd scripts && HF_HUB_OFFLINE=0 uv run python download_models.py && cd ..

5. Copy Resources/example.env to Resources/.env and fill in your API keys
6. Run

   uv run benchmark

CLI options

uv run benchmark [--clear] [--test N]
                 [--reset-all-metrics]
                 [--reset-metrics-for-models MODEL [MODEL ...]]
                 [--reset-metric-types TYPE [TYPE ...]]
                 [--gold-standard-data FILE [FILE ...]]

Flag	Description
--clear	Delete all existing results (responses + metrics) and re-run from scratch
--test N	Run on the first N publications only (useful for quick sanity checks)
--reset-all-metrics	Recompute all metrics while keeping cached LLM responses
--reset-metrics-for-models	Delete stored results for specific models and re-run from API cache (space-separated)
--reset-metric-types	Force recompute only specific metric types (default: all)
--gold-standard-data	Override the gold-standard data file(s)

By default, uv run benchmark automatically detects and recomputes any missing or empty metrics without requiring extra flags. The reset flags above can be combined to force recomputation of specific metrics for specific models:

# Recompute minicheck scores for all models
uv run benchmark --reset-metric-types minicheck_ft5_scores minicheck_7b_scores

# Recompute all metrics for two specific models
uv run benchmark --reset-metrics-for-models local:textrank local:frequency

# Recompute only minicheck scores for a specific model
uv run benchmark --reset-metric-types minicheck_ft5_scores --reset-metrics-for-models local:textrank

Configuration

LLM generation parameters (temperature, token limits, system prompt, etc.) are configured in src/llm_apis/config.py. The default word-count constraints are 15–100 words.

Resume / re-run without benchmarking

The following files must be in place to load previous results:

• Output/llm_summarization_benchmark/benchmark.pkl
• Output/llm_apis/cache.json

Run uv run benchmark again — already-processed models will be skipped and only visualisations will be regenerated.

---

Workflow

---

Gold-Standard Data

Document store in Resources/text_summarization_goldstandard_data.json, containing ID, title, abstract and reference summaries. 1–N reference summaries can be provided per paper. Multiple reference summaries improve evaluation robustness and reduce single-annotator bias.

[
  {
    "title": "Paper Title",
    "abstract": "Paper abstract text...",
    "id": "paper_001",
    "summaries": [
      "This paper analyzes ..",
      "The paper investigates .. "
    ]
  }
]

Reference summary sources

• Highlight sections of Elsevier and Cell papers, manually extracted and joined by ". ".

---

Summarization Methods

local:textrank

1. Tokenises sentences (nltk)
2. Creates TF-IDF vectors for sentence representation (sklearn)
3. Calculates cosine similarities between TF-IDF vectors (sklearn)
4. Builds similarity graph with cosine similarities as edge weights (networkx)
5. Applies PageRank to rank sentences by importance (networkx)
6. Selects highest-scoring sentences within word count limits while preserving original order

Warning: Results may be misleading when gold-standard summaries are (partial) copies from the source document rather than abstractive.

local:frequency

1. Calculates word frequency scores
2. Ranks sentences by average word frequency (excluding stopwords via nltk)
3. Selects highest-scoring sentences (in original order) within word count limits

LLM providers

• Anthropic — Claude 3.5 Haiku, Claude Sonnet 4, Claude Opus 4, Claude Opus 4.1
• OpenAI — GPT-3.5 Turbo, GPT-4o / 4o-mini, GPT-4.1 / 4.1-mini, GPT-5 / 5-mini / 5-nano
• Mistral — Mistral Large, Mistral Medium, Mistral Small, Magistral Medium
• HuggingFace — BART, T5, PEGASUS, BigBird-PEGASUS, LED, mT5, BioGPT, BioMistral, OpenBioLLM, SciLitLLM, Apertus
• Ollama — DeepSeek-R1, Gemma 3, Granite 3.3/4, LLaMA 3.1/3.2, MedLLaMA 2, Mistral 7B / Nemo / Small 3.2, Phi-3/4, Qwen 3, GPT-OSS

---

Evaluation Metrics

Each generated summary is evaluated against all available gold-standard reference summaries. For each metric, mean / min / max / std and individual scores are computed.

ROUGE

Set of metrics for evaluating summary quality by comparing n-gram overlap with reference summaries. wiki | package | paper

• ROUGE-1 — unigram overlap
• ROUGE-2 — bigram overlap
• ROUGE-L — longest common subsequence (sentence-level structure similarity)

BERTScore

Semantic similarity using contextual embeddings. paper | package

• roberta-large — default model (paper | model)
• microsoft/deberta-xlarge-mnli — proposed as improved model (paper | model)

METEOR

Matches words through exact matches, stemming, and synonyms; considers word order. paper

BLEU

N-gram overlap with brevity penalty. paper | function

Semantic Similarity (all-mpnet-base-v2)

Compares the generated summary directly against the source document (rather than reference summaries) using sentence-transformer embeddings. model

AlignScore

Factual consistency evaluation — measures whether the generated summary is entailed by the source document (title + abstract). paper | modified repository

SummaC

NLI-based sentence-level factual consistency — aggregates entailment scores between source sentences and summary sentences using a zero-shot approach (SummaC-ZS with ViTC backbone). paper | repository

FactCC

BERT-based binary factual consistency classifier — returns P(CORRECT) as a continuous score in [0, 1], indicating whether the generated summary is factually consistent with the source document. paper | model

MiniCheck

Sentence-level factual consistency via synthetic claim decomposition and verification. Each summary is split into sentences (nltk.sent_tokenize), scored individually against the source document (title + abstract), and aggregated via mean. Two variants are used: paper | repository

• MiniCheck-FT5 — fine-tuned Flan-T5-Large (770M) (model)
• MiniCheck-7B — Bespoke-MiniCheck-7B, a Mistral-7B fine-tune achieving SOTA on factual consistency benchmarks (model)

Note on factual consistency metric divergence

All four factual consistency metrics (AlignScore, SummaC, FactCC, MiniCheck) verify generated summaries against the same source document (title + abstract), matching the input the LLMs received during generation. Despite this unified input, scores are expected to diverge due to fundamental differences in model architecture and scoring granularity:

Property	AlignScore	SummaC	FactCC	MiniCheck
Architecture	RoBERTa-large regression	NLI entailment (ViTC)	BERT binary classifier	Flan-T5-Large / Mistral-7B
Granularity	Document-level	Sentence-level (internal)	Document-level	Sentence-level (explicit)
Output	Continuous [0, 1]	Continuous [0, 1]	P(CORRECT) [0, 1]	Continuous (FT5) / binary mean (7B)
Max input tokens	512 (RoBERTa)	512 (ViTC)	512 (BERT)	4096 (FT5)

Key factors driving score differences:

• Scoring granularity — AlignScore and FactCC score the entire summary as a single unit. MiniCheck decomposes summaries into sentences and averages per-sentence scores. A summary with one hallucinated sentence out of ten will score ~0.9 in MiniCheck but may score much lower in FactCC if the hallucination dominates the joint representation.
• Truncation — AlignScore, SummaC, and FactCC are limited to 512 tokens (model-internal). For the longest papers in this dataset (~550 tokens source + ~220 tokens summary), the combined input exceeds this limit and is silently truncated. MiniCheck-FT5 supports 4096 tokens and is unaffected.
• Training data and consistency definition — Each model was trained on different datasets with different operationalisations of "factual consistency," leading to systematic disagreements especially on borderline cases.
• MiniCheck-7B quantisation — The 7B variant returns binary yes/no judgments per sentence, producing a coarser score distribution (values cluster around discrete fractions like 0.0, 0.5, 0.75, 1.0) compared to the continuous scores of the other metrics.

---

Training Cutoff Dates

Model training-data cutoff dates are tracked in Resources/model_training_cutoffs.json and rendered as an interactive HTML report at Resources/model_training_cutoffs.html.

To regenerate after editing the JSON or adding new models to the benchmark:

python scripts/fetch_training_cutoffs.py

The script automatically merges data from:

• A curated seed knowledge base (built-in)
• The community LLM knowledge-cutoff dataset
• HuggingFace model cards (scraped for cutoff patterns)
• Provider documentation (OpenAI, Anthropic, Mistral)

Manual edits to the JSON are preserved across re-runs.

---

Utility Scripts

Script	Description
scripts/fetch_training_cutoffs.py	Collects model training-data cutoff dates from multiple sources and generates a JSON + interactive HTML report
scripts/add_publication_dates.py	Enriches the gold-standard dataset with publication dates fetched from the CrossRef API using DOIs
scripts/download_models.py	Parses benchmark.py and pre-downloads all required Ollama, HuggingFace, and metric models (SummaC, FactCC, MiniCheck). Required before first run since the benchmark operates in offline mode (HF_HUB_OFFLINE=1).

---

Acknowledgements & Data Sources

Resource	Usage	Link
LLM Knowledge Cutoff Dates (HaoooWang)	Primary source for model training-data cutoff dates, crowd-sourced with citations to official docs and papers	github.com/HaoooWang/llm-knowledge-cutoff-dates
CrossRef API	Publication date retrieval for gold-standard papers via DOI lookup	crossref.org
HuggingFace Hub	Model hosting, model cards used for cutoff date extraction, and inference for encoder–decoder models	huggingface.co
Ollama	Local inference runtime for open-weight LLMs (LLaMA, Gemma, DeepSeek, Phi, Qwen, etc.)	ollama.com
AlignScore	Factual consistency evaluation metric	Zha et al. 2023 · modified repo
SummaC	NLI-based sentence-level factual consistency metric	Laban et al. 2022 · repo
FactCC	BERT-based binary factual consistency classifier	Kryscinski et al. 2020 · model
MiniCheck	Sentence-level factual consistency via claim decomposition and verification (FT5 + 7B variants)	Tang et al. 2024 · repo
BERTScore	Semantic similarity evaluation using contextual embeddings	Zhang et al. 2020
ROUGE	N-gram overlap metrics for summarization evaluation	Lin 2004

---

Reference

A Systematic evaluation and benchmarking of text summarization methods for biomedical literature: From word-frequency methods to language models Baumgärtel F, Bono E, Fillinger L, Galou L, Kęska-Izworska K, Walter S, Andorfer P, Kratochwill K, Perco P, Ley M bioRxiv 2026, doi.org/10.64898/2026.01.09.697335