{"id":"ca486113-316d-4775-8847-2fbdf55d9b6a","arxiv_id":"2505.16392","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new taxonomy and annotated test collection for errors in automatic text simplification, with benchmarks showing current detectors rarely identify these errors.","lead":"This resource paper introduces a four-part taxonomy of errors in automatically simplified scientific text and a 2,659-annotation test collection built on it. It also measures how well existing hallucination and quality detectors catch these errors, finding that most perform poorly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central empirical claim rests on gold labels for Simplification errors, but D2.1/D2.2 depend on an unspecified goal-dependent F_imp and show low inter-annotator agreement; reported model failure may partly be label noise.","rationale":"The single most load-bearing assumption is the reliability of the gold labels, specifically for the D. Simplification categories that drive the empirical conclusion. The paper's formal apparatus does not actually define F_imp, and the authors acknowledge this; given Fleiss kappa 0.26 for D and 0.02 for C, the labels cannot be treated as a stable ground truth. My proposed test directly checks whether an operationalized F_imp rescues the taxonomy; if it does not, the 'existing methods fail to detect them reliably' claim should be downgraded to 'existing methods do not agree with these annotations.' I agree with the reader's weakest_assumption and keep CONDITIONAL because the taxonomy and resource may still be useful once definitions are tightened and annotations are validated; the internal Overspecification definition error reinforces the need for a revision before the benchmark is used.","tokens_in":14135,"tokens_out":6723,"duration_ms":57940,"concrete_test":"Perform a controlled re-annotation study: take the 104 multi-annotated instances, provide two fresh annotators with an explicit operationalization of F_imp (for example, important facts are those needed to answer a fixed set of five comprehension questions about the source), and have them relabel D2.1 and D2.2. If Cohen's kappa between the new labels and the original majority labels stays below 0.6, the taxonomy's Simplification categories are not operationalizable and the reported failure of existing models is confounded by label noise; if kappa exceeds 0.6, the low IAA in Table 3 stems from annotator training, and the empirical claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that current ATS methods fail to detect errors reliably under the new taxonomy, and that claim is only as strong as the gold labels. The weakest link is the definition of Simplification errors, which are the most prevalent categories in the dataset. In Section 2.5.1, D2.1 (Loss of Informative Content) and D2.2 (Out-of-Scope Generation) are defined relative to F_imp, the set of important facts. The paper itself says this set 'depends on the goal of the simplification' and that these errors 'will need to be more properly defined based on a proper definition of the needs of each target audience.' No operational definition of F_imp was given to annotators, so the 19.56% and 15.72% prevalence figures in Table 1 are not anchored to a shared criterion. The inter-annotator agreement is consistent with this: Fleiss kappa for D. Simplification is 0.26, with only 25.2% unanimous agreement, and for C. Information it is 0.02 (Table 3). Annotator C, who had a self-consistency rate of only 0.56, was retained to increase data size. In addition, the formal taxonomy contains an internal contradiction: Section 2.5.2 defines Overspecification as 'Replacing a concept with a more general one' (twice), although the formal condition e_src ⊃ e_gen and Appendix D1.2 both describe replacing a general concept with a more specific one. If the labels for the two most prevalent error categories are noisy, the low AUPRC scores in Tables 4 and 5 may partly measure annotation disagreement rather than model failure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a taxonomy of errors in automatic text simplification (ATS), organized into four top-level categories (Fluency, Alignment, Information, Simplification) with formal set-theoretic definitions for information-related errors. The authors annotate 2,659 sentence-level simplifications from CLEF 2024 SimpleText runs, measure inter-annotator agreement, release the test collection, and evaluate six existing metrics/models (BERTScore, QAGS, FEQA, FactCC, FactAcc, LENS) for binary error detection, coarse-grained detection, and fine-grained error-type detection. The central claims are that this is the first comprehensive ATS error taxonomy and that current methods fail to detect these errors reliably.","tokens_in":14482,"tokens_out":4611,"duration_ms":36640,"significance":"If the taxonomy and test collection were fully reliable, this would fill a real gap: a shared vocabulary and benchmark for error analysis in ATS, with formal definitions going beyond ad-hoc typologies. The paper is honest about annotation challenges and releases its resources, and its planned use as the basis for the CLEF 2025 SimpleText shared task is a concrete form of impact. However, the current annotation reliability for the two most prevalent error categories is too low to support the strength of the empirical claims as stated.","major_comments":[{"comment":"The definitions of Loss of Informative Content and Out-of-Scope Generation rely on F_imp, the set of important facts, which is explicitly goal-dependent and is not operationally specified. The authors themselves write that these errors \"will need to be more properly defined based on a proper definition of the needs of each target audience.\" Without an operational criterion for importance, the prevalence figures in Table 1 (19.56% and 15.72%) and the model scores in Tables 4-5 for these categories are not anchored to a shared judgment. Please provide the annotator instructions for identifying important facts, or re-annotate with a defined importance criterion and adjudication.","section":"Section 2.5.1 / Appendix D2.1-D2.2"},{"comment":"The inter-annotator agreement is very low for the two categories that carry the empirical conclusion: Fleiss kappa is 0.02 for Information errors and 0.26 for Simplification errors, with only 25.2% unanimous agreement for Simplification, and annotator C was retained despite a self-consistency rate of 0.56. Since the conclusion that \"existing methods fail to detect them reliably\" depends on these gold labels, the low agreement is load-bearing: the reported AUPRC values may reflect annotator disagreement or label noise as much as model failure. Please report per-error-type agreement, provide adjudicated or majority-vote labels, or explicitly analyze how label noise affects the AUPRC estimates.","section":"Section 3, Tables 2-3"},{"comment":"The prose definition of Overspecification says \"Replacing a concept with a more general one,\" which directly contradicts the formal condition e_src ⊃ e_gen and the Appendix D1.2 definition of replacing a broad entity with a specific subcategory. This internal contradiction appears twice in Section 2.5.2 and must be corrected, since the formal condition is the one that matches the intended meaning and the appendix example.","section":"Section 2.5.2 and Appendix D1.2"},{"comment":"The bullet list under D1.2 is copied verbatim from D1.1 (e.g., \"Replacing entities with the greater category of entities\"), which is inconsistent with the definition of overspecification. In addition, D2.1 examples 1-3 duplicate the D1.1 examples, and the two examples under D2.2 are identical. This duplication obscures the annotation scheme and should be fixed before the resource is released.","section":"Appendix D1.2 and D2.1-D2.2"}],"minor_comments":[{"comment":"The text says \"since about half of our dataset is error-free,\" but Table 1 reports 30.84% \"No error\" (820 of 2,659), and Table 4 also lists 30.84% for No Error; this discrepancy should be corrected.","section":"Section 4.1"},{"comment":"There are several typos and repeated phrases, including \"misalignement\" (Tables and Appendix B1), \"accross\" (Section 3), and the duplicated \"In particular, we will use In particular\" in Section 4.1.","section":"Throughout"},{"comment":"Undefined Cohen's kappa values are set to 1 in cases of complete agreement; this practice should be stated explicitly or the undefined values should be reported as missing, since setting them to 1 can inflate the impression of agreement.","section":"Table 3"},{"comment":"The paper alternates between \"greater categories\" and \"top-level categories\" for the four main error types; using one consistent term would improve readability.","section":"Section 2"}],"recommendation":"major_revision","confidential_remarks":"The resource is credible in intent, but the low inter-annotator agreement and the unspecified importance criterion make the current empirical claims premature for a published resource paper. A revision that either provides adjudicated labels, operationalizes F_imp, or substantially weakens the conclusion about existing methods failing would make the contribution solid. The decision to retain annotator C for data size is particularly problematic for a test collection that will be used as shared-task gold data."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is worth a serious look despite its wobbly empirical legs. It gives the field its first ATS-specific error taxonomy with formal, fact-based definitions of information distortion, and an annotated test collection built from real SimpleText runs. The taxonomy is mostly a reorganization of known categories, but making the definition formal and separating information from simplification errors is a real step forward. The decision to run existing hallucination detectors against the collection and report AUPRC per error type is also useful, even if the results are predictably poor.\n\nWhat I trust: the data collection is traceable to CLEF 2024 runs, the annotation schema is detailed, and the authors report inter-annotator agreement honestly. They do not hide the fact that Information errors have Fleiss kappa 0.02 and Simplification errors 0.26. The transparency is a point in their favor.\n\nWhat worries me: the two most prevalent Simplification error types (Loss of Informative Content, Out-of-Scope Generation: together about 35%) are defined relative to F_imp, the set of important facts, which the paper itself says depends on the simplification goal and \"will need to be more properly defined.\" Annotators were not given an operational definition, so the prevalence figures are not anchored to a shared criterion. Given D.Simplification kappa is 0.26, the low AUPRC scores of the detection models may partly measure label noise, not model failure. The stress-test note holds up on this point.\n\nThere are also small internal errors: Section 2.5.2's Overspecification definition says \"more general\" while the formal condition e_src > e_gen and the Appendix say \"more specific.\" Section 4.1 says \"about half\" the dataset is error-free, but Table 1 shows 30.84% no-error. Appendix D2.1 examples are duplicated from D1.1. None of these sink the resource, but they need cleaning.\n\nBottom line: this is a promising resource, not a finished benchmark. The paper deserves peer review and a request for major revision: operationalize F_imp for annotation, reconcile the Overspecification typo, fix the internal inconsistencies, and present the model evaluation with appropriate caveats about label reliability. Who benefits: anyone working on ATS evaluation, error analysis, or hallucination detection for simplification. I'd cite it once the dataset is actually public. Bring it to reading group when the data lands.","headline":"A genuinely useful ATS error taxonomy and test collection, but the central empirical claim is undercut by undefined importance and low annotator agreement; deserves peer review with heavy revision.","tokens_in":14972,"tokens_out":1963,"would_cite":true,"duration_ms":15520,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a four-part taxonomy of errors in automatic text simplification, an annotated test collection of 2,659 simplified scientific sentences, and evidence that current hallucination-detection models fail to detect most of…","keywords":["automatic text simplification","error taxonomy","hallucination","test collection","human annotation","large language models","faithfulness","information distortion"],"falsifier":"Re-annotate the same 104 instances with five annotators trained on the paper's scheme: if Fleiss kappa for Information and Simplification errors stays near the reported 0.02 and 0.26 instead of rising into the substantial range, the benchmark's labels—not the detection models—are the limiting factor.","tokens_in":13957,"feed_emoji":"🧠","tokens_out":12836,"duration_ms":90125,"temperature":0.7,"pith_summary":"Automatic text simplification systems, especially LLM-based ones, generate errors that standard evaluation metrics miss, and the field had no shared vocabulary for naming them. This paper aims to establish, to the best of its knowledge, the first comprehensive taxonomy of ATS errors: fluency, alignment, information, and simplification, with formally defined subcategories focused on information distortion. Using that taxonomy, the authors build a test collection of 2,659 human annotations of automatically simplified scientific sentences and find a 69.16% error rate, with loss of informative content the most frequent single error. When six existing metrics and detectors are run against these labels, the best (FactCC) reaches 0.68 AUROC for binary error detection, and all of them score poorly on specific error types. If the resource is adopted, error analysis gains a shared benchmark that makes systems comparable and gives detector training a concrete target.","feed_headline":"New taxonomy: 69% of simplified texts have errors; detectors miss them","feed_subtitle":"2,659 annotated simplifications give the field a shared error vocabulary and a benchmark current detectors fail to meet.","key_machinery":"The load-bearing object is a four-branch taxonomy (Fluency, Alignment, Information, Simplification) whose formal core is a fact-based model of information: facts are triples $(subj, rel, obj)$, with $F_{src}$, $F_{gen}$, $F_{topic}$, $F_{true}$, $F_{false}$, and $F_{imp}$ denoting the sets of source facts, generated facts, topical facts, true facts, false facts, and important facts. Errors are then defined by set operations: for example, faithfulness hallucination is $F_{gen} \\cap F_{topic} \\cap F_{cont}$ (generated facts that are topical but contradict the source), factuality hallucination is $F_{gen} \\cap F_{topic} \\cap F_{false} \\setminus F_{cont}$, loss of informative content is $F_{src} \\cap F_{imp} \\setminus F_{gen}$, and out-of-scope generation is $F_{gen} \\setminus F_{imp}$. This formal core is what turns the taxonomy into an annotation scheme: annotators label each simplified sentence with the error types from the tree, producing the test collection that grounds the evaluation.","core_discovery":"The paper's central claim is that the first comprehensive error taxonomy for automatic text simplification can be built from a formal, fact-based definition of information distortion, and that a test collection annotated with this taxonomy reveals both a high prevalence of errors and the unreliability of existing detectors. Following the taxonomy, facts are modeled as triples (subject, relation, object); information errors such as topic shift, faithfulness hallucination, and factuality hallucination are defined by set operations over source facts, generated facts, and important facts, while simplification errors such as loss of informative content and out-of-scope generation depend on the goal-dependent set of important facts. On 2,659 annotations, the paper reports that 69.16% of simplified sentences contain at least one error, with loss of informative content (19.56%), out-of-scope generation (15.72%), and faithfulness hallucination (13.54%) among the most frequent. The showcase evaluation then shows that existing methods—FactCC, FEQA, QAGS, FactAcc, LENS, and BERTScore—do not detect or classify these errors reliably, which the paper takes as evidence that a taxonomy-grounded benchmark is needed.","pith_inferences":["The low inter-annotator agreement on Information ($\\kappa=0.02$) and Simplification ($\\kappa=0.26$) errors suggests a testable bottleneck: re-annotating with trained annotators would show whether the taxonomy's definitions or annotator inexperience explain the noise, a distinction the authors leave open.","Because the formal definitions of Simplification errors depend on $F_{imp}$, which is goal-dependent, a natural extension is to instantiate $F_{imp}$ for specific audiences (non-experts vs. language learners) and measure how the error distribution shifts; the released annotation scheme makes this possible without new data collection.","The same fact-triple machinery could transfer to neighbouring generation tasks such as summarization or medical text simplification, where faithfulness hallucinations and loss of informative content are also central concerns."],"forward_implications":["Researchers obtain a public CSV test collection, an annotation scheme, and analysis code, so error-detection results across different simplification systems become directly comparable.","The benchmark quantifies the gap between current practice and useful evaluation: no evaluated detector shows reliable performance on any of the four error families, so improving detection is an open problem.","The 69.16% annotated error rate, led by loss of informative content and out-of-scope generation, shifts attention from fluency metrics to information-distortion and simplification-specific failures.","Because the test collection is planned to anchor the shared task on error detection described in the paper, upcoming systems will have a concrete, public target to optimize against."],"supporting_citations":[{"why":"It documents that existing evaluation measures for scientific text simplification do not agree with human quality labels, which motivates an error-based framework.","marker":"[1]"},{"why":"FEQA is one of the question-answering faithfulness models evaluated in the showcase, and it reaches only 0.52 AUROC on binary error detection.","marker":"[2]"},{"why":"It describes the shared task that will use the proposed test collection to anchor an error-detection challenge.","marker":"[3]"},{"why":"It is the shared-task overview whose system runs supply the simplified sentences that were annotated in the test collection.","marker":"[5]"},{"why":"It introduces FactAcc, a fact-based factual-consistency model that the showcase evaluates, and supplies hallucination terminology the taxonomy engages with.","marker":"[8]"},{"why":"SALSA is the only prior edit-level simplification evaluation typology, and the paper argues it is incomplete and misaligned with faithfulness terminology.","marker":"[9]"},{"why":"FactCC is the BERT-based consistency model that achieves the highest binary error-detection AUROC (0.68) in the showcase.","marker":"[13]"},{"why":"QAGS is one of the question-generation/question-answering consistency models evaluated in the showcase, with moderate binary AUROC of 0.51.","marker":"[18]"},{"why":"BERTScore, a standard text-generation metric, shows very low error-detection AUROC (0.23), supporting the claim that current ATS metrics miss errors.","marker":"[20]"}],"fun_headline_variants":["Error taxonomy: 69% of simplifications flawed, detectors fail","Simplified texts: 69% have errors per first distortion taxonomy","Detectors fail: 69% of simplified texts contain errors (new taxonomy)","New error taxonomy: 69% of simplified sentences are flawed, detectors miss","First distortion taxonomy: 69% error rate, detectors unreliable"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole benchmark rests on treating 2,659 human annotations as reliable reference labels even though agreement among annotators was near zero for Information errors and low for Simplification errors, and the definition of what counts as 'important information' is left to the simplification goal rather than specified.","fun_headline_variants_meta":{"raw":{"variants":["Error taxonomy: 69% of simplifications flawed, detectors fail","Simplified texts: 69% have errors per first distortion taxonomy","Detectors fail: 69% of simplified texts contain errors (new taxonomy)","New error taxonomy: 69% of simplified sentences are flawed, detectors miss","First distortion taxonomy: 69% error rate, detectors unreliable"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000699,"raw_usage":{"total_tokens":3180,"prompt_tokens":988,"completion_tokens":2192,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2111}},"tokens_in":604,"tokens_out":2192,"duration_ms":12115,"temperature":1.0,"reasoning_tokens":2111,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:01:17.394363+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the same 104 instances with five annotators trained on the paper's scheme: if Fleiss kappa for Information and Simplification errors stays near the reported 0.02 and 0.26 instead of rising into the substantial range, the benchmark's labels—not the detection models—are the limiting factor.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It documents that existing evaluation measures for scientific text simplification do not agree with human quality labels, which motivates an error-based framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It describes the shared task that will use the proposed test collection to anchor an error-detection challenge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the shared-task overview whose system runs supply the simplified sentences that were annotated in the test collection."},{"cited_title":"Dancing Between Success and Failure: Edit-level Simplification Evaluation using SALSA","cited_arxiv_id":"2305.14458","evidence_quote":"SALSA is the only prior edit-level simplification evaluation typology, and the paper argues it is incomplete and misaligned with faithfulness terminology."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FactCC is the BERT-based consistency model that achieves the highest binary error-detection AUROC (0.68) in the showcase."}],"review_version":1}