Pith. sign in

REVIEW 5 major objections 5 minor 12 references

Hallucination Detection and Mitigation in Scientific Text Simplification using Ensemble Approaches: DS@GT at CLEF 2025 SimpleText

T0 review · 5 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a four-signal ensemble—a fine-tuned BERT classifier, source-similarity scoring, entailment checks, and an LLM judge—detects spurious content in scientific text simplification at 0.95 F1, and that LLM post-editing impr

desk verdict A competent shared-task system description whose central ensemble claim is not supported by its own tables; the Task 2.2 ensemble is the strongest part, and the Task 2.3 BLEU result is partly circular. read the letter →

arxiv 2508.11823 v1 pith:YQCXHBKH submitted 2025-08-15 cs.CL

Krishna Chaitanya Marturi , Heba H. Elwazzan This is my paper
classification cs.CL
keywords textsimplificationhallucinationdetectionensemblelearningBERTnaturallanguageinferenceLLMasjudgegroundedgenerationCLEFSimple
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a system for the CLEF 2025 SimpleText Task 2, which asks whether simplified scientific sentences are hallucinated or distorted and how to keep generated simplifications grounded. The system combines a fine-tuned BERT classifier, a semantic similarity score computed over chunked source abstracts, a natural-language-inference entailment model, and an LLM judge, with outputs fused by a small neural-network meta-classifier. The authors claim this ensemble reaches an F1 of 0.95 on spurious-sentence detection and remains stable when the source abstract is withheld, although a BERT-only baseline matches its F1 and even edges it out in accuracy without the source. For grounded generation, the same LLM post-edits simplified text against the source, raising BLEU and lexical similarity but lowering SARI—a trade-off the paper frames as fidelity versus bold simplification. If right, the paper offers a practical, off-the-shelf recipe for catching and correcting LLM hallucinations in scientific text simplification.

What carries the argument

The load-bearing object is the ensemble meta-classifier: a three-layer neural network that ingests eight probabilistic features from the four detectors—BERT score, chunked cosine similarity, entailment/contradiction probabilities, and the four LLM-judge scores—and outputs a spuriousness decision. Around it sit the source-chunking scheme (100-word passages with 50-word overlap) and, in the post-hoc setting, dense passage retrieval that selects the top-5 similar chunks before computing similarity, entailment, and LLM judgment. For grounding, the machinery is a prompt-driven LLM post-editor that returns the input unchanged when faithful and otherwise rewrites it against the reference.

What would settle it

Retrain the meta-classifier and the BERT classifier on the official SimpleText training split only (with validation held out), then run them on the released test set with labels withheld. If the ensemble's ROC AUC is no longer greater than the BERT-only baseline's—as already happens for accuracy in the post-hoc setting—the claimed ensemble advantage collapses.

Watch

Extended reading notes

Core claim

The central claim is that hallucination detection in scientific simplification is best served by combining qualitatively different signals rather than relying on a single model. The paper fuses eight probabilistic features—a BERT classifier score, maximum chunk-level cosine similarity, maximum entailment and contradiction probabilities from an NLI model, and four LLM-judge scores (spuriousness, over-generalization, contradiction, vagueness)—into a three-layer neural classifier. The authors report 0.95 F1 with the source available (ROC AUC 0.68) and 0.95 F1 without it (ROC AUC 0.64), with the ensemble balancing precision and recall better than the conservative LLM judge alone. For distortion-

Load-bearing premise

The load-bearing premise is that the meta-classifier and the fine-tuned BERT classifier were trained on examples drawn independently of the CLEF 2025 test set; the paper never documents this split, so the ensemble's reported edge over the BERT-only baseline could be an artifact of test-set influence rather than a genuine gain from combining signals.

Editorial extensions

If this is right

  • If the ensemble claim holds, hallucination detection in simplification need not require a purpose-built model; off-the-shelf BERT, NLI, and an LLM judge plus a small meta-classifier already reach 0.95 F1.
  • The near-match of the BERT-only baseline implies that supervised lexical and syntactic patterns alone are a strong signal for spuriousness, with semantic and reasoning components adding robustness rather than raw F1.
  • Grounded post-editing consistently raises BLEU and lexical similarity across both 37-abstract and 217-summary test sets, suggesting the approach transfers across dataset sizes.
  • The consistent SARI decline under grounding means that adopting this method means accepting less aggressive simplification in exchange for semantic fidelity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A reader should not infer that the ensemble is strictly better than a single BERT classifier: in the post-hoc subtask, BERT alone has higher accuracy, so the ensemble benefit is conditional on the setting and metric.
  • Because the paper never specifies the training split for the meta-classifier or the fine-tuned BERT, the reported ensemble advantage could partly reflect fitting to the test distribution; a straightforward check is to retrain on a standard train/dev split and compare ROC AUC against BERT alone.
  • The LLM judge's four scores enter the meta-classifier, so the ensemble's behavior depends on the judge's calibration; swapping in a smaller or cheaper model would reveal how much of the gain is model-specific.
  • The fidelity-versus-simplicity trade-off suggests a testable extension: constraining the post-editing prompt to delete fabricated content but forbid longer, more complex rewrites might recover SARI without sacrificing BLEU.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper describes the DS@GT submission to CLEF 2025 SimpleText Task 2, covering hallucination detection and mitigation in scientific text simplification. For Tasks 2.1/2.2, the authors propose ensemble detectors that combine a fine-tuned BERT (or DeBERTa) classifier, semantic similarity, an NLI model, and an LLM judge, aggregated by a small meta-classifier. For Task 2.3, they use an LLM post-editor that revises a baseline simplification to be 'fully grounded' in the source. The paper evaluates these systems on the official task data and reports high F1 values (0.95) for spurious text detection, improved multi-label error classification, and higher BLEU for grounded generation; it also acknowledges a trade-off between groundedness and simplification quality (lower SARI, higher FKGL).

Significance. If the central claims held, the paper would provide a useful working system for hallucination detection in simplification, with a documented combination of semantic, entailment, and LLM-based signals, and an empirical trade-off analysis. The presentation of prompt templates and component-wise comparisons is a strength. However, the experimental evidence does not establish the headline claims: in Task 2.1 the ensemble does not reliably beat its BERT-only component, the post-hoc setting is internally inconsistent about source access, and the Task 2.3 evaluation uses BLEU as a fidelity proxy in a way that is partly circular. The paper therefore currently reads as a system description with over-stated conclusions rather than a validated methodology.

major comments (5)
  1. [Section 4.1.1 / Tables 1-2] The central ensemble claim is not supported. In the sourced setting (Table 1), the ensemble matches the BERT-only classifier on accuracy (0.91), F1 (0.95), and AUPRC (0.93), and differs only in AUROC (0.68 vs 0.65, +0.03). In the post-hoc setting (Table 2), the ensemble is slightly worse on accuracy (0.90 vs 0.91) and identical in AUROC (0.64); the LLM-alone AUROC is higher (0.70). No confidence intervals, bootstrap resamples, or paired tests (McNemar/DeLong) are reported, so the single positive AUROC gap could be noise. The claim in §4.1.1 that the ensemble 'demonstrates the effectiveness of combining shallow, semantic, and reasoning-based components' is therefore not established by the data shown. Please provide statistical tests and clarify the criterion for ensemble advantage.
  2. [Section 3.1.1 / 4.1] The training/evaluation split is never specified. The meta-classifier, BERT classifier, and RoBERTa/DeBERTa models are described as 'trained' and 'fine-tuned', but the paper does not state whether the training data are disjoint from the 3,379/3,336 evaluation instances used in Tables 1-2. Without this, the reported ensemble performance could reflect tuning on the test distribution rather than genuine generalization, and the comparison between ensemble and components is not interpretable. Please give an explicit description of the train/dev/test partition and, if the same data are used for evaluation, report a proper cross-validation or held-out evaluation.
  3. [Section 3.1.2] There is a direct contradiction in the post-hoc setting. The text states 'The input texts are without access to their source in the post-hoc setting,' but the first bullet under §3.1.2 describes Dense Passage Retrieval that 'encodes both the input text and the chunks derived from the source abstracts' and uses these chunks for NLI and LLM scoring. If the system has access to source abstracts during inference, this is not a post-hoc setting; if it does not, the method description is wrong. This affects the validity of all post-hoc results in Table 2 and the comparison in §4.1.3. Please clarify the actual information available at inference time and align the description accordingly.
  4. [Section 4.3 / Table 4] The claim that grounded systems exhibit 'higher semantic fidelity' is largely based on BLEU scores, but BLEU is a lexical n-gram overlap measure, not a semantic fidelity metric. The post-editor prompt explicitly instructs the LLM to 'revise the input text so that it is fully grounded in the reference document' and 'return only the corrected version ... otherwise return the same input text.' Reporting higher BLEU against that same reference is therefore expected by construction, not evidence of improved factual consistency. Moreover, the grounded outputs show large FKGL increases (e.g., 9.56 to 12.74 and 7.77 to 12.87), indicating reduced readability, and the paper does not provide any human judgment, factual-consistency evaluation, or semantic metric besides BLEU/Levenshtein. The trade-off discussion is plausible, but the evidence as presented conflates prompt-induced lexical alignment w
  5. [Section 4.2 / Table 3] The Task 2.2 ensemble gains are small and are reported without uncertainty quantification. For example, the No-error F1 is 0.763 for the ensemble vs 0.694 for RoBERTa and 0.680 for LLaMA; on some categories the gap is tiny (A F1 0.283 vs 0.282). No confidence intervals, significance tests, or per-category variance are shown. In addition, §3.2 inconsistently names the underlying transformer as both 'RoBERTa-large' and 'DeBERTa,' and the fusion mechanism of the 'three-layer neural network meta-classifier' is described only superficially. Please specify the exact inputs, architecture, training details, and validation procedure so that the reported improvement can be evaluated.
minor comments (5)
  1. [Section 3.2] The model names are inconsistent: §3.2 says 'fine-tuning a RoBERTa-large model' but the ensemble is called 'DebertaLlmensemble' and is attributed to a 'DeBERTa model.' Please correct the model names to be consistent throughout.
  2. [Figure 2] Figure 2 is labeled 'Spurious Text Detection without access to Source' but the component list includes Dense Passage Retrieval over source abstracts. The figure and the text should be reconciled with the intended post-hoc protocol.
  3. [Author affiliation] The corresponding author's email string in the footnote is garbled: '/envel⌢pe-⌢penkmarturi3@gatech.edu.' It should be a standard email address.
  4. [Section 4.1.3] The statement that 'the absence of the source abstract does not significantly impair the model’s detection capabilities' is not supported by any statistical test. The AUROC drop from 0.68 to 0.64 is simply stated. Please avoid the word 'significantly' without a significance test, or supply one.
  5. [Section A.1 / A.2] The prompt templates are a useful contribution, but some typographical artifacts remain (e.g., double braces in the JSON examples). Minor cleaning would help reproducibility.

Circularity Check

1 steps flagged · score 4.0 of 10

Task 2.3 fidelity evidence reduces to the post-editing prompt's own objective; otherwise no significant circularity.

  1. self definitional [Section 3.3.1 (grounded generation prompt) and Section 4.3 (evaluation)]
    "Your task is to revise the input text so that it is fully grounded in the reference document. The corrected version must: Be factually consistent with the reference ... Return only the corrected version of the input text if it is needed, otherwise return the same input text. ... grounded systems generally exhibit higher semantic fidelity, as evidenced by consistently higher BLEU scores."

    The grounded post-editor is explicitly prompted to make the output 'fully grounded in the reference document.' Section 4.3 then reports that grounded outputs have higher BLEU and Levenshtein similarity to that same reference and interprets this as evidence of 'semantic fidelity.' Because the prompt directly instructs the model to align with the reference, the measured overlap is the immediate effect of the instruction, not an independent confirmation. The claim 'grounded generation preserves source fidelity' is thus equivalent to the objective already encoded in the post-editing prompt; the evaluation metric is essentially measuring whether the model obeyed the prompt.

full rationale

Apart from the Task 2.3 evaluation, no significant circularity is present. The Task 2.1/2.2 ensemble results are empirical comparisons; the fact that BERT-only matches or beats the ensemble in Tables 1 and 2 is a statistical/evidentiary weakness, not a circularity. The paper contains no self-citations, no imported uniqueness theorems, and no ansatz smuggled via citation. The only step that reduces by construction is the grounded-generation evaluation: the prompt instructs the model to align output to the reference, and the evaluation uses overlap with that reference (BLEU/Levenshtein) as evidence of fidelity. I rate this as partial circularity (4) because it affects one of the three subtask evaluations but does not undermine the ensemble detection claims. The unspecified meta-classifier training split is a reporting gap, not evidence of circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

No new physical or mathematical entities are introduced. The system relies on existing models and a trained meta-classifier; the main hidden inputs are data split assumptions and unstated training details.

free parameters (5)
  • meta-classifier weights = not reported
    Three-layer neural network trained to aggregate eight probabilistic features from BERT, cosine similarity, NLI, and LLM judge; architecture and training details omitted.
  • document chunk size = 100 words with 50-word overlap
    Chosen in Section 3.1.1 without ablation; affects all source-dependent features.
  • retrieval top-k = 5 chunks
    Chosen in Section 3.1.2 without ablation; determines NLI and LLM inputs in post-hoc setting.
  • BERT classifier fine-tuning = binary spuriousness label
    Fine-tuned on unstated training data; serves as primary signal in both settings.
  • LLM judge score thresholds = not reported
    The ensemble uses four LLM scores (0-1); how they are thresholded or normalized is not stated.
assumptions (4)
  • domain assumption CLEF 2025 labels are reliable ground truth for spuriousness and distortion categories.
    All metrics in Section 4 treat the shared task labels as correct.
  • domain assumption Pre-trained models (multi-qa-MiniLM-L6-cos-v1, facebook/bart-large-mnli, llama-3.3-70b-versatile) are used as prescribed and produce calibrated probability outputs.
    Scores from these models are directly fed into the meta-classifier without calibration mention.
  • domain assumption The meta-classifier is trained on a data split disjoint from the test set.
    This premise is not stated in the paper; it underlies the validity of the ensemble comparison.
  • domain assumption The LLM judge's JSON responses are parseable and follow the requested 0-1 range.
    The paper gives prompts but no failure-rate or post-processing details for malformed outputs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hallucination Detection and Mitigation in Scientific Text Simplification using Ensemble Approaches: DS@GT at CLEF 2025 SimpleText." pith.science (2026). https://pith.science/paper/YQCXHBKH

@misc{pith2026250811823,
  author       = {Pith},
  title        = {Pith review of: Hallucination Detection and Mitigation in Scientific Text Simplification using Ensemble Approaches: DS@GT at CLEF 2025 SimpleText},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YQCXHBKH}},
  note         = {Machine review of arXiv:2508.11823}
}
read the original abstract

In this paper, we describe our methodology for the CLEF 2025 SimpleText Task 2, which focuses on detecting and evaluating creative generation and information distortion in scientific text simplification. Our solution integrates multiple strategies: we construct an ensemble framework that leverages BERT-based classifier, semantic similarity measure, natural language inference model, and large language model (LLM) reasoning. These diverse signals are combined using meta-classifiers to enhance the robustness of spurious and distortion detection. Additionally, for grounded generation, we employ an LLM-based post-editing system that revises simplifications based on the original input texts.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

12 extracted references · 11 canonical work pages

  1. [1]

    Ermakova, et al., Overview of clef 2025 simpletext track: Simplify scientific texts (and nothing more), in: J

    L. Ermakova, et al., Overview of clef 2025 simpletext track: Simplify scientific texts (and nothing more), in: J. Carillo de Albornoz, et al. (Eds.), Experimental IR Meets Multilinguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference of the CLEF Association (CLEF 2025), LNCS, Springer-Verlag, 2025

    work page 2025

  2. [2]

    Vendeville, et al., Overview of the CLEF 2025 SimpleText Task 2: Identify and Avoid Hallucination, in: G

    B. Vendeville, et al., Overview of the CLEF 2025 SimpleText Task 2: Identify and Avoid Hallucination, in: G. Faggioli, et al. (Eds.), Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2025), CEUR Workshop Proceedings, CEUR-WS.org, 2025. To appear

    work page 2025

  3. [3]

    Z. Ji, N. Lee, J. A. Fries, Y. Goldberg, M. Sachan, C. D. Manning, Survey of hallucination in natural language generation, ACM Computing Surveys (2023)

    work page 2023

  4. [4]

    Maynez, S

    J. Maynez, S. Narayan, B. Bohnet, R. McDonald, On faithfulness and factuality in abstractive summarization, in: ACL, 2020

    work page 2020

  5. [5]

    Kryściński, B

    W. Kryściński, B. McCann, C. Xiong, R. Socher, Evaluating the factual consistency of abstractive text summarization, in: EMNLP, 2020

    work page 2020

  6. [6]

    Y. Dong, J. Zhang, M. Lapata, Multi-fact correction in abstractive text summarization, in: ACL, 2020

    work page 2020

  7. [7]

    M. Reid, N. Duan, B. Dolan, Z. Liu, Revisiting factual evaluation of summarization via question answering, in: ACL Findings, 2022

    work page 2022

  8. [8]

    Y. Gao, X. Li, G. Neubig, Factual error correction for abstractive summarization via reinforcement learning, in: NAACL, 2022

    work page 2022

Show all 12 references
  1. [9]

    Nishino, A

    R. Nishino, A. Cohan, I. Augenstein, Scitldr: Extreme summarization of scientific documents, in: EMNLP, 2023

    2023

  2. [10]

    Vendeville, L

    B. Vendeville, L. Ermakova, P. De Loor, Resource for error analysis in text simplification: New taxonomy and test collection, arXiv preprint arXiv:2505.16392 (2025). URL: https://arxiv.org/ pdf/2505.16392.pdf, to appear in SIGIR ’25: Proceedings of the 48th International ACM S...

    2025 arXiv

  3. [11]

    Ziems, et al., Can large language models be consistently trusted for factuality detection?, arXiv preprint arXiv:2305.15005 (2023)

    C. Ziems, et al., Can large language models be consistently trusted for factuality detection?, arXiv preprint arXiv:2305.15005 (2023)

    2023 arXiv

  4. [12]

    s p u r i o u s n e s s

    PACE, Partnership for an Advanced Computing Environment (PACE), 2017. URL: http://www. pace.gatech.edu. A. Prompt Templates A.1. LLM as Judge Prompt You a r e an e x p e r t a n n o t a t o r t a s k e d with e v a l u a t i n g whether an i n p u t t e x t i s s p u r i o u s...

    2017

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.