Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper proposes a four-part taxonomy of errors in automatic text simplification, an annotated test collection of 2,659 simplified scientific sentences, and evidence that current hallucination-detection models fail to detect most of…

desk verdict A genuinely useful ATS error taxonomy and test collection, but the central empirical claim is undercut by undefined importance and low annotator agreement; deserves peer review with heavy revision. read the letter →

arxiv 2505.16392 v1 pith:UHIPSPHC submitted 2025-05-22 cs.CL cs.AI

Benjamin Vendeville , Liana Ermakova , Pierre De Loor This is my paper · ORCID
classification cs.CLcs.AI
keywords automatictextsimplificationerrortaxonomyhallucinationtestcollectionhumanannotationlargelanguagemodelsfaithfulnessinformationdistortion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Automatic text simplification systems, especially LLM-based ones, generate errors that standard evaluation metrics miss, and the field had no shared vocabulary for naming them. This paper aims to establish, to the best of its knowledge, the first comprehensive taxonomy of ATS errors: fluency, alignment, information, and simplification, with formally defined subcategories focused on information distortion. Using that taxonomy, the authors build a test collection of 2,659 human annotations of automatically simplified scientific sentences and find a 69.16% error rate, with loss of informative content the most frequent single error. When six existing metrics and detectors are run against these labels, the best (FactCC) reaches 0.68 AUROC for binary error detection, and all of them score poorly on specific error types. If the resource is adopted, error analysis gains a shared benchmark that makes systems comparable and gives detector training a concrete target.

What carries the argument

The load-bearing object is a four-branch taxonomy (Fluency, Alignment, Information, Simplification) whose formal core is a fact-based model of information: facts are triples $(subj, rel, obj)$, with $F_{src}$, $F_{gen}$, $F_{topic}$, $F_{true}$, $F_{false}$, and $F_{imp}$ denoting the sets of source facts, generated facts, topical facts, true facts, false facts, and important facts. Errors are then defined by set operations: for example, faithfulness hallucination is $F_{gen} \cap F_{topic} \cap F_{cont}$ (generated facts that are topical but contradict the source), factuality hallucination is $F_{gen} \cap F_{topic} \cap F_{false} \setminus F_{cont}$, loss of informative content is $F_{src} \cap F_{imp} \setminus F_{gen}$, and out-of-scope generation is $F_{gen} \setminus F_{imp}$. This formal core is what turns the taxonomy into an annotation scheme: annotators label each simplified sentence with the error types from the tree, producing the test collection that grounds the evaluation.

What would settle it

Re-annotate the same 104 instances with five annotators trained on the paper's scheme: if Fleiss kappa for Information and Simplification errors stays near the reported 0.02 and 0.26 instead of rising into the substantial range, the benchmark's labels—not the detection models—are the limiting factor.

Watch

Extended reading notes

Core claim

The paper's central claim is that the first comprehensive error taxonomy for automatic text simplification can be built from a formal, fact-based definition of information distortion, and that a test collection annotated with this taxonomy reveals both a high prevalence of errors and the unreliability of existing detectors. Following the taxonomy, facts are modeled as triples (subject, relation, object); information errors such as topic shift, faithfulness hallucination, and factuality hallucination are defined by set operations over source facts, generated facts, and important facts, while simplification errors such as loss of informative content and out-of-scope generation depend on the goal-dependent set of important facts. On 2,659 annotations, the paper reports that 69.16% of simplified sentences contain at least one error, with loss of informative content (19.56%), out-of-scope generation (15.72%), and faithfulness hallucination (13.54%) among the most frequent. The showcase evaluation then shows that existing methods—FactCC, FEQA, QAGS, FactAcc, LENS, and BERTScore—do not detect or classify these errors reliably, which the paper takes as evidence that a taxonomy-grounded benchmark is needed.

Load-bearing premise

The whole benchmark rests on treating 2,659 human annotations as reliable reference labels even though agreement among annotators was near zero for Information errors and low for Simplification errors, and the definition of what counts as 'important information' is left to the simplification goal rather than specified.

Editorial extensions

If this is right

  • Researchers obtain a public CSV test collection, an annotation scheme, and analysis code, so error-detection results across different simplification systems become directly comparable.
  • The benchmark quantifies the gap between current practice and useful evaluation: no evaluated detector shows reliable performance on any of the four error families, so improving detection is an open problem.
  • The 69.16% annotated error rate, led by loss of informative content and out-of-scope generation, shifts attention from fluency metrics to information-distortion and simplification-specific failures.
  • Because the test collection is planned to anchor the shared task on error detection described in the paper, upcoming systems will have a concrete, public target to optimize against.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The low inter-annotator agreement on Information ($\kappa=0.02$) and Simplification ($\kappa=0.26$) errors suggests a testable bottleneck: re-annotating with trained annotators would show whether the taxonomy's definitions or annotator inexperience explain the noise, a distinction the authors leave open.
  • Because the formal definitions of Simplification errors depend on $F_{imp}$, which is goal-dependent, a natural extension is to instantiate $F_{imp}$ for specific audiences (non-experts vs. language learners) and measure how the error distribution shifts; the released annotation scheme makes this possible without new data collection.
  • The same fact-triple machinery could transfer to neighbouring generation tasks such as summarization or medical text simplification, where faithfulness hallucinations and loss of informative content are also central concerns.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper presents a taxonomy of errors in automatic text simplification (ATS), organized into four top-level categories (Fluency, Alignment, Information, Simplification) with formal set-theoretic definitions for information-related errors. The authors annotate 2,659 sentence-level simplifications from CLEF 2024 SimpleText runs, measure inter-annotator agreement, release the test collection, and evaluate six existing metrics/models (BERTScore, QAGS, FEQA, FactCC, FactAcc, LENS) for binary error detection, coarse-grained detection, and fine-grained error-type detection. The central claims are that this is the first comprehensive ATS error taxonomy and that current methods fail to detect these errors reliably.

Significance. If the taxonomy and test collection were fully reliable, this would fill a real gap: a shared vocabulary and benchmark for error analysis in ATS, with formal definitions going beyond ad-hoc typologies. The paper is honest about annotation challenges and releases its resources, and its planned use as the basis for the CLEF 2025 SimpleText shared task is a concrete form of impact. However, the current annotation reliability for the two most prevalent error categories is too low to support the strength of the empirical claims as stated.

major comments (4)
  1. [Section 2.5.1 / Appendix D2.1-D2.2] The definitions of Loss of Informative Content and Out-of-Scope Generation rely on F_imp, the set of important facts, which is explicitly goal-dependent and is not operationally specified. The authors themselves write that these errors "will need to be more properly defined based on a proper definition of the needs of each target audience." Without an operational criterion for importance, the prevalence figures in Table 1 (19.56% and 15.72%) and the model scores in Tables 4-5 for these categories are not anchored to a shared judgment. Please provide the annotator instructions for identifying important facts, or re-annotate with a defined importance criterion and adjudication.
  2. [Section 3, Tables 2-3] The inter-annotator agreement is very low for the two categories that carry the empirical conclusion: Fleiss kappa is 0.02 for Information errors and 0.26 for Simplification errors, with only 25.2% unanimous agreement for Simplification, and annotator C was retained despite a self-consistency rate of 0.56. Since the conclusion that "existing methods fail to detect them reliably" depends on these gold labels, the low agreement is load-bearing: the reported AUPRC values may reflect annotator disagreement or label noise as much as model failure. Please report per-error-type agreement, provide adjudicated or majority-vote labels, or explicitly analyze how label noise affects the AUPRC estimates.
  3. [Section 2.5.2 and Appendix D1.2] The prose definition of Overspecification says "Replacing a concept with a more general one," which directly contradicts the formal condition e_src ⊃ e_gen and the Appendix D1.2 definition of replacing a broad entity with a specific subcategory. This internal contradiction appears twice in Section 2.5.2 and must be corrected, since the formal condition is the one that matches the intended meaning and the appendix example.
  4. [Appendix D1.2 and D2.1-D2.2] The bullet list under D1.2 is copied verbatim from D1.1 (e.g., "Replacing entities with the greater category of entities"), which is inconsistent with the definition of overspecification. In addition, D2.1 examples 1-3 duplicate the D1.1 examples, and the two examples under D2.2 are identical. This duplication obscures the annotation scheme and should be fixed before the resource is released.
minor comments (4)
  1. [Section 4.1] The text says "since about half of our dataset is error-free," but Table 1 reports 30.84% "No error" (820 of 2,659), and Table 4 also lists 30.84% for No Error; this discrepancy should be corrected.
  2. [Throughout] There are several typos and repeated phrases, including "misalignement" (Tables and Appendix B1), "accross" (Section 3), and the duplicated "In particular, we will use In particular" in Section 4.1.
  3. [Table 3] Undefined Cohen's kappa values are set to 1 in cases of complete agreement; this practice should be stated explicitly or the undefined values should be reported as missing, since setting them to 1 can inflate the impression of agreement.
  4. [Section 2] The paper alternates between "greater categories" and "top-level categories" for the four main error types; using one consistent term would improve readability.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the taxonomy is a definitional proposal, the test collection is human-annotated, and the evaluation scores are measured against external checkpoints rather than fitted to the labels.

full rationale

The paper's derivation chain is: (1) propose a set-theoretic taxonomy of ATS errors; (2) collect simplifications from CLEF 2024 shared-task runs; (3) have annotators label those simplifications using the taxonomy; (4) evaluate off-the-shelf error-detection models against the resulting labels. None of these steps reduces to its own output. The formal definitions, e.g., Error_faithfulness = F_gen ∩ F_topic ∩ F_cont and Error_loss = F_src ∩ F_imp \ F_gen, are definitions built from openly stated primitives; they do not generate empirical predictions from fitted constants. F_imp is admittedly goal-dependent and not fully operationalized, and the inter-annotator agreement for Information and Simplification errors is low, but these are annotation-quality and construct-validity limitations, not circular reasoning. The empirical claim that existing methods 'fail to detect them reliably' follows directly from measured AUROC/AUPRC values on held-out labels; the evaluated models are pretrained systems with no parameters fitted to this dataset. The paper's self-citations (CLEF SimpleText overviews) are used only for data provenance and prior analysis; they are not load-bearing justifications for the central conclusion. The overspecification definition in Section 2.5.2 contains a prose/formal mismatch, but that is an internal consistency issue, not a circular reduction. Overall, the resource-paper structure is standard benchmark construction and is not circular.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The formal taxonomy rests on several unproven assumptions about fact representation, source truthfulness, and importance. The most consequential are the reliability of subjective importance judgments and the treatment of annotations as gold despite low inter-annotator agreement; these assumptions directly affect the error prevalence and detection results.

assumptions (5)
  • domain assumption The source document is entirely truthful.
    Invoked in Section 2.4.2 to define faithfulness and factuality hallucinations; if the source itself contains errors, the formal distinction between source-contradicting and world-knowledge-contradicting facts collapses.
  • domain assumption Information in text can be represented as facts (subject, relation, object) triples.
    Section 2.4 defines F_src and F_gen this way; this representation may not capture pragmatics, modality, hedging, or implicit information in real scientific text.
  • domain assumption Every important fact is true and on-topic (F_imp subset of F_true and F_imp subset of F_topic).
    Stated in Section 2.5.1; this assumption makes simplification errors well-defined but is not guaranteed for every target audience.
  • domain assumption What counts as important information depends on the simplification goal and target audience, and annotators can apply it consistently.
    Section 2.5.1 and D2.1 note importance is goal-dependent; low inter-annotator agreement for Simplification errors (Fleiss kappa 0.26) suggests this assumption is fragile.
  • domain assumption The 2,659 human annotations constitute reliable gold labels for evaluating error detectors.
    All analysis in Sections 3 and 4 treats annotations as ground truth, yet one annotator has self-consistency 0.56 and Information errors have Fleiss kappa 0.02.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection." pith.science (2026). https://pith.science/paper/UHIPSPHC

@misc{pith2026250516392,
  author       = {Pith},
  title        = {Pith review of: Resource for Error Analysis in Text Simplification: New Taxonomy and Test Collection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UHIPSPHC}},
  note         = {Machine review of arXiv:2505.16392}
}
read the original abstract

The general public often encounters complex texts but does not have the time or expertise to fully understand them, leading to the spread of misinformation. Automatic Text Simplification (ATS) helps make information more accessible, but its evaluation methods have not kept up with advances in text generation, especially with Large Language Models (LLMs). In particular, recent studies have shown that current ATS metrics do not correlate with the presence of errors. Manual inspections have further revealed a variety of errors, underscoring the need for a more nuanced evaluation framework, which is currently lacking. This resource paper addresses this gap by introducing a test collection for detecting and classifying errors in simplified texts. First, we propose a taxonomy of errors, with a formal focus on information distortion. Next, we introduce a parallel dataset of automatically simplified scientific texts. This dataset has been human-annotated with labels based on our proposed taxonomy. Finally, we analyze the quality of the dataset, and we study the performance of existing models to detect and classify errors from that taxonomy. These contributions give researchers the tools to better evaluate errors in ATS, develop more reliable models, and ultimately improve the quality of automatically simplified texts.

Figures

Figures reproduced from arXiv: 2505.16392 by the authors.

Figure 1
Figure 1. Structure of our typology of errors in text simplification. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Diagram illustrating the sets and intersections in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Diagram illustrating the sets and intersections in [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Hallucination Detection and Mitigation in Scientific Text Simplification using Ensemble Approaches: DS@GT at CLEF 2025 SimpleText

    cs.CL 2025-08 conditional novelty 3.0 of 10

    An ensemble of BERT, cosine similarity, NLI, and LLM judgment with a meta-classifier detects distorted simplifications, while an LLM post-editor trades simplicity for faithfulness.

Reference graph

Works this paper leans on

22 extracted references · 10 canonical work pages · cited by 1 Pith paper

  1. [1]

    Dennis Davari, Liana Ermakova, and Ralf Krestel. 2024. Comparative Analysis of Evaluation Measures for Scientific Text Simplification. In Linking Theory and Practice of Digital Libraries - 28th International Conference on Theory and Practice of Digital Libraries, TPDL 2024, Ljubljana, Slovenia, September 24-27, SIGIR ’25, July 13–18,2025, Padua, Italy Ben...

    work page 2024

  2. [2]

    Esin Durmus, He He, and Mona Diab. 2020. FEQA: A Question Answer- ing Evaluation Framework for Faithfulness Assessment in Abstractive Sum- marization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (Eds.). Association for Computational Linguistics, ...

    doi:10.18653/v1/2020.acl-main.454 2020

  3. [3]

    Liana Ermakova, Hosein Azarbonyad, Jan Bakker, Benjamin Vendeville, and Jaap Kamps. 2025. CLEF 2025 SimpleText Track - Simplify Scientific Text (and Nothing More). In Advances in Information Retrieval - 47th European Conference on Information Retrieval, ECIR 2025, Lucca, Italy, April 6-10, 2025, Proceedings, Part V (Lecture Notes in Computer Science, Vol....

    work page 2025

  4. [4]

    Liana Ermakova, Sarah Bertin, Helen McCombie, and Jaap Kamps. 2023. Overview of the CLEF 2023 SimpleText Task 3: Simplification of Scientific Texts. InWorking Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), Thessaloniki, Greece, September 18th to 21st, 2023 (CEUR Workshop Proceedings, Vol. 3497) , Mohammad Aliannejadi, Guglielmo Fagg...

    work page 2023

  5. [5]

    Liana Ermakova, Valentin Laimé, Helen McCombie, and Jaap Kamps. 2024. Overview of the CLEF 2024 SimpleText Task 3: Simplify Scientific Text. InWork- ing Notes of the Conference and Labs of the Evaluation Forum (CLEF 2024), Grenoble, France, 9-12 September, 2024 (CEUR Workshop Proceedings, Vol. 3740) , Guglielmo Faggioli, Nicola Ferro, Petra Galuscáková, a...

    work page 2024

  6. [6]

    Liana Ermakova, Eric SanJuan, Stéphane Huet, Hosein Azarbonyad, Olivier Augereau, and Jaap Kamps. 2023. Overview of the CLEF 2023 SimpleText Lab: Au- tomatic Simplification of Scientific Texts. InExperimental IR Meets Multilinguality, Multimodality, and Interaction, Avi Arampatzis, Evangelos Kanoulas, Theodora Tsikrika, Stefanos Vrochidis, Anastasia Giach...

    work page 2023

  7. [7]

    Liana Ermakova, Eric SanJuan, Stéphane Huet, Hosein Azarbonyad, Gior- gio Maria Di Nunzio, Federica Vezzani, Jennifer D’Souza, and Jaap Kamps. 2024. Overview of the CLEF 2024 SimpleText Track: Improving Access to Scientific Texts for Everyone. In Experimental IR Meets Multilinguality, Multimodality, and Interaction: 15th International Conference of the CL...

    work page 2024

  8. [8]

    Liu, and Mohammad Saleh

    Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assess- ing The Factual Accuracy of Generated Text. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD ’19). Association for Computing Machinery, New York, NY, USA, 166–175. https://doi.org/10.1145/3292500.3330955

    arXiv 2019

Show all 22 references
  1. [9]

    David Heineman, Yao Dou, Mounica Maddela, and Wei Xu. 2023. Dancing Between Success and Failure: Edit-level Simplification Evaluation Using SALSA. https://doi.org/10.48550/arXiv.2305.14458 arXiv:2305.14458 [cs]

    2023 doi arXiv

  2. [10]

    Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kuk- liansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating Factual Consistency Evaluation. https: //doi.org/10.48550/arXiv.2204.04991 arXiv:2204.04991 [cs]

    2022 doi arXiv

  3. [11]

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions. https://doi.org/1...

    2023 doi arXiv

  4. [12]

    Sebastian Joseph, Kathryn Kazanas, Keziah Reina, Vishnesh Ramanathan, Wei Xu, Byron Wallace, and Junyi Jessy Li. 2023. Multilingual Simplification of Medical Texts. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Houda Bouamor, Juan P...

    2023 doi

  5. [13]

    Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher

  6. [14]

    J. R. Landis and G. G. Koch. 1977. The Measurement of Observer Agreement for Categorical Data. Biometrics 33, 1 (March 1977), 159–174

    1977

  7. [15]

    Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has Machine Translation Achieved Human Parity? A Case for Document-level Evaluation. https://doi. org/10.48550/arXiv.1808.07048 arXiv:1808.07048 [cs]

    2018 doi arXiv

  8. [16]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, et al. 2024. GPT-4 Tech- nical Report. https://doi.org/10.48550/arXiv.2303.08774 arXiv:2303.08774 [cs]

    2024 doi arXiv

  9. [17]

    Advaith Siddharthan. 2014. A Survey of Research on Text Simplification. ITL - International Journal of Applied Linguistics 165, 2 (Dec. 2014), 259–298. https: //doi.org/10.1075/itl.165.2.06sid

    2014 doi

  10. [18]

    Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and Answering Questions to Evaluate the Factual Consistency of Summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Te...

    2020 doi

  11. [19]

    Xuanxin Wu and Yuki Arase. 2024. An In-depth Evaluation of GPT-4 in Sentence Simplification with Error-based Human Assessment. arXiv:2403.04963 [cs]

    2024 arXiv

  12. [20]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi

  13. [22]

    arXiv:1904.09675 [cs]

    BERTScore: Evaluating Text Generation with BERT. arXiv:1904.09675 [cs]

    1904 arXiv

  14. [2020]

    In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.)

    Evaluating the Factual Consistency of Abstractive Text Summariza- tion. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Onl...

    2020 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.