TextQ-German investigates how people perceive the quality of machine-generated German text and how these subjective judgments can be modeled automatically. We identified task-specific quality dimensions, quantified them through user ratings, and developed models that predict perceived quality for new generated texts.
TextQ-German is a dataset suite for studying the Quality of Experience (QoE) of machine-generated German text. It covers two Natural Language Generation (NLG) tasks:
- Machine Translation (MT)
- Automatic Text Summarization (ATS)
The resource contains human judgments of perceived text quality and supports research on:
- identifying perceptual quality dimensions of machine-generated text,
- quantifying these dimensions through subjective evaluation,
- analyzing linguistic properties associated with perceived quality, and
- developing automatic models for predicting fine-grained and overall QoE.
QoE annotations are represented as Mean Opinion Scores (MOS) obtained by averaging ratings across human annotators on a 0–6 scale.
TextQ-German consists of six subsets:
| Subset | Task | Generation | Samples | Available QoE Labels |
|---|---|---|---|---|
TextQ-ATS |
Automatic Text Summarization | Non-LLM | 91 | Linguistic Logic, Complexity, Clarity, Predictability |
TextQ-MT |
Machine Translation | Non-LLM | 106 | Precision, Complexity, Transparency, Grammaticality |
TextQ-ATS-LLM |
Automatic Text Summarization | LLM | 77 | Overall QoE, Linguistic Logic, Complexity, Clarity, Predictability |
TextQ-MT-LLM |
Machine Translation | LLM | 77 | Overall QoE, Precision, Complexity, Transparency, Grammaticality |
TextQ-ATS-Val |
Automatic Text Summarization | LLM + Non-LLM | 77 | Overall QoE, Linguistic Logic, Complexity, Clarity, Predictability |
TextQ-MT-Val |
Machine Translation | LLM + Non-LLM | 76 | Overall QoE, Precision, Complexity, Transparency, Grammaticality |
The validation subsets contain held-out, source-disjoint samples intended for final model evaluation.
The perceptual quality dimensions were derived separately for MT and ATS through crowdsourcing studies using Semantic Differential scales.
TextQ-German contains four ATS quality dimensions:
- Linguistic Logic – coherence and logical consistency of the generated text
- Complexity – perceived complexity and comprehensibility
- Clarity – clarity and unambiguity of the language
- Predictability – perceived logical and expected structure
TextQ-German contains four MT quality dimensions:
- Precision – precision, completeness, and clarity of phrasing
- Complexity – perceived complexity and comprehensibility
- Transparency – clarity and coherence of the language
- Grammaticality – grammatical correctness
All datasets are provided as semicolon-separated CSV files.
Each row corresponds to one generated text item. The QoE labels are item-level Mean Opinion Scores (MOS) obtained by averaging ratings across annotators.
TextQ-ATS-Val.csv
id;sentence;linguistic_logic;complexity;clarity;predictability;overall;batch
TextQ-MT-Val.csv
id;sentence;precision;complexity;transparency;grammaticality;overall;batch
| Column | Description |
|---|---|
id |
Unique identifier of the text item |
sentence |
Machine-generated German text |
linguistic_logic |
MOS for Linguistic Logic |
complexity |
MOS for Complexity |
clarity |
MOS for Clarity |
predictability |
MOS for Predictability |
precision |
MOS for Precision |
transparency |
MOS for Transparency |
grammaticality |
MOS for Grammaticality |
overall |
MOS for overall perceived QoE |
batch |
Generation category used in the validation subsets |
All QoE scores lie on a 0–6 scale, with higher scores indicating more positive perceived quality.
TextQ-ATS and TextQ-MT contain the original non-LLM corpora with dimension-level QoE annotations.
The ATS data are based on German automatic summaries generated using a range of extractive and abstractive summarization systems.
The MT data contain English–German machine translations sampled to cover a broad range of output quality.
TextQ-ATS-LLM and TextQ-MT-LLM extend the original resource with outputs from large language models.
In addition to the four task-specific quality dimensions, these subsets contain an overall QoE score.
TextQ-ATS-Val and TextQ-MT-Val contain both LLM-generated and non-LLM-generated outputs.
Their source texts do not overlap with the source texts used in the other TextQ-German subsets. They are intended as held-out datasets for evaluating QoE prediction models after model development.
The CSV files use ; as delimiter and can be loaded directly with pandas:
import pandas as pd
df = pd.read_csv("TextQ-ATS.csv", sep=";")
print(df.head())
print(df.columns)TextQ-German can be used for research on:
- human-centered evaluation of Natural Language Generation,
- automatic prediction of perceived text quality,
- multi-dimensional text-quality assessment,
- German text-quality modeling,
- machine translation quality assessment,
- automatic text summarization evaluation,
- linguistic feature analysis, and
- comparison of neural, feature-based, and hybrid QoE prediction models.
The resource is designed primarily as a research dataset and benchmark for perceived quality assessment.
If you use TextQ-German or the associated QoE prediction methods in your work, please cite the following papers:
Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz and Sebastian Möller. 2025. Assessing Quality of Experience in Natural Language Generation of German Text. arXiv preprint arXiv:2608.18888.
@article{pham2026assessingqualityexperiencenatural,
title={Assessing Quality of Experience in Natural Language Generation of German Text},
author={Dinh Nam Pham and Shushen Manakhimova and Vivien Macketanz and Sebastian M{\"o}ller},
year={2026},
journal={arXiv preprint arXiv:2608.18888},
doi={10.48550/arXiv.2608.18888},
}Dinh Nam Pham, Vivien Macketanz, Shushen Manakhimova, and Sebastian Möller. 2025.
Modeling Quality of Experience in German Automatic Text Summarization and Machine Translation.
In Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops, pages 169–175.
@inproceedings{pham-etal-2025-modeling,
title = "Modeling Quality of Experience in {G}erman Automatic Text Summarization and Machine Translation",
author = {Pham, Dinh Nam and
Macketanz, Vivien and
Manakhimova, Shushen and
M{\"o}ller, Sebastian},
editor = {Wartena, Christian and
Heid, Ulrich},
booktitle = "Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops",
month = sep,
year = "2025",
address = "Hannover, Germany",
publisher = "HsH Applied Academics",
url = "https://aclanthology.org/2025.konvens-2.12/",
pages = "169--175"
}TextQ-German is released under the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) license.