Pith. sign in

REVIEW 2 major objections 6 minor 18 references

This paper builds a human-in-the-loop corpus and workflow for simplifying scientific summaries with LLMs, claiming that non-specialist readers prefer LLM simplifications for comprehension while expert editing is needed to preserve scientifi

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-01 01:48 UTC pith:KC6PEQGZ

load-bearing objection A transparent, small-scale resource paper whose Phase 1 reader data is solid, but the expert-edited references—the would-be benchmark component—are unvalidated for the target audience and all automatic metrics favor the GPT baselines. the 2 major comments →

arxiv 2607.25630 v1 pith:KC6PEQGZ submitted 2026-07-28 cs.CL cs.AIcs.HC

A Human-in-the-Loop Corpus for LLM-Based Simplification of Scientific Summaries

Kyuri Im , Michael F\"arber This is my paper
classification cs.CL cs.AIcs.HC
keywords text simplificationscientific communicationhuman-in-the-looplarge language modelscorpusSciSummNetreadabilityexpert annotation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper builds and releases a corpus for simplifying scientific summaries for non-specialist readers. It starts with expert-written abstracts from computer science papers, asks an LLM to simplify them, collects difficulty annotations and preference judgments from STEM readers outside the field, and then has computer science experts produce reference simplifications informed by those annotations. The central finding is that readers overwhelmingly prefer the LLM versions for understanding and simplicity, while expert editing is what preserves domain terminology and the strength of scientific claims. The corpus is meant to support training and benchmarking of simplification systems that balance readability with fidelity.

Core claim

The paper's contribution is a human-in-the-loop workflow and the resulting corpus: 47 expert-edited reference simplifications, each anchored to an original SciSummNet summary and a GPT-4o-mini baseline simplification, with sentence- and phrase-level difficulty annotations and 92 comparative reader judgments. The Phase 1 results show a descriptive preference for the LLM outputs in understanding (73/92) and simplicity (70/92), with more balanced naturalness judgments. Phase 2 shows that expert post-editing substantially rewords the LLM output (lower BLEU/BERTScore), reduces surface readability by conventional formulas, but is qualitatively necessary for keeping technical terms and calibrated c

What carries the argument

The load-bearing mechanism is the two-phase, human-in-the-loop editing loop. In Phase 1, cross-disciplinary STEM readers flag difficult sentences/phrases and judge original vs. LLM-simplified summaries on understanding, naturalness, and simplicity. In Phase 2, computer science experts use these flags to edit the LLM output under four guided scenarios, with explicit guidelines to preserve domain terminology and claim strength. The resulting set of original, LLM-simplified, and expert-edited triples is the corpus.

Load-bearing premise

The central resource's ground truth value depends on the assumption that the expert-edited summaries are actually better for the target readers, which the paper explicitly did not test.

What would settle it

Run a comprehension or preference study with the same cross-disciplinary STEM population, presenting original, GPT-simplified, and expert-edited versions of the same 47 summaries. If expert-edited versions are not judged more comprehensible or more faithful than the GPT baseline, the claim that expert editing is crucial is weakened.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • The corpus provides a training and benchmarking resource for simplification models that must balance readability with terminology fidelity.
  • The preference data suggests LLM simplifications are acceptable starting points for non-specialists, but they should be treated as drafts rather than final scientific text.
  • Expert post-editing can be guided by reader annotations to target exactly where simplification breaks down, making the process more efficient.
  • Automatic metrics like BLEU and readability formulas favor GPT outputs, which underscores why they are insufficient for evaluating scientific simplification quality.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If expert-edited references are later judged by the same reader population, they may or may not beat the LLM baseline; the paper leaves this open, and a reader study is the obvious next test.
  • The workflow could be extended to other domains and languages, and the reader-annotation signal could be used to train a reward model for simplification.
  • Claim-strength calibration is a hidden dimension here; the corpus could support metrics that detect overstatement or dropped qualifications, not just readability.
  • The released corpus could enable a new evaluation paradigm that asks readers to verify factual consistency rather than just rate simplicity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper presents a human-in-the-loop workflow for simplifying scientific summaries. Using SciSummNet as the source, the authors generate baseline simplifications with GPT-4o-mini, collect sentence- and phrase-level difficulty annotations and comparative judgments (understanding, naturalness, simplicity) from STEM readers outside computer science (Phase 1), and then have CS experts post-edit the LLM outputs into reference simplifications (Phase 2). The released corpus includes 92 reader judgments and 47 expert-edited summaries. Phase 1 shows a clear descriptive preference for the GPT outputs in understanding and simplicity. Phase 2 automatic metrics (BLEU, BERTScore, SARI, Flesch-Kincaid) favor the GPT outputs on source overlap and surface readability, while qualitative examples illustrate that expert editing preserves domain terminology and claim strength. The authors are consistently modest: they report no significance tests, label the SARI variant as reference-free, and explicitly state that the expert-edited summaries were not evaluated by target readers. The paper positions the corpus as supporting training and benchmarking of simplification systems for cross-disciplinary scientific communication.

Significance. If the released corpus is of sufficient quality, it addresses a real gap: existing simplification benchmarks focus on news/Wikipedia, and few include cross-disciplinary reader judgments. The human-in-the-loop methodology is a practical contribution, and the paper ships data and code, which is valuable. The Phase 1 results provide useful descriptive evidence that LLM simplifications improve perceived comprehensibility for STEM non-CS readers. The main weakness is that the expert-edited references—a key component of the corpus—are not validated with the target audience, so the resource's ground-truth value for benchmarking fidelity and readability is only partially established. Still, the authors explicitly acknowledge this, and the corpus can support future work even if the expert references require further validation. Overall, the contribution is appropriate for a resource paper, provided the claims are kept proportionate to the evidence.

major comments (2)
  1. [§4.2, Table 1, Abstract] The claim that expert post-editing is 'crucial' for domain terminology and claim strength is supported only by a few qualitative examples and by automatic metrics that actually favor the GPT outputs on every reported measure (BLEU, BERTScore, Flesch-Kincaid, reference-free SARI). The expert-edited summaries were not independently evaluated by target readers, as the authors note in §5. This is load-bearing for the resource's benchmarking value: if a downstream system trains to match these references, it may optimize for editorial preferences rather than target-reader utility. I recommend either adding a small-scale reader evaluation of expert-edited vs. GPT outputs on a subset of items, or softening the abstract and contribution statements to 'qualitative evidence suggests potential importance' and explicitly stating that expert-edit quality was not reader-validated in the main text.
  2. [§3.3, §4.1] The Phase 1 study reports only the total number of judgments per dimension (92), not the number of participants, their specific STEM disciplines, how many summaries each participant judged, or any annotation reliability measures. Section 5 lists annotation reliability as a limitation but provides no agreement statistics (e.g., Cohen's kappa for highlighted spans or sentence-level difficulty flags). For a corpus paper with human annotations, these details are essential for assessing data quality and reproducibility. Please report participant counts, demographics, task assignment structure, and any inter-annotator agreement metrics computed, or explain why they are not applicable.
minor comments (6)
  1. [Figure 6] The 'Original' text in Figure 6 contains a duplicated sentence: 'Since the notion of correctness is not intuitively clear in this context, different ways of defining it are proposed.' appears twice consecutively. Please fix.
  2. [§3.2, Appendix] The exact GPT-4o-mini prompt is only available in the repository. To make the paper self-contained, include the full prompt in an appendix and report the model configuration (temperature, max tokens, etc.).
  3. [§3.4, §4.2] The Phase 2 section does not report how many expert editors participated, their level of expertise (e.g., PhD vs. MSc), or how many summaries each editor handled. This information should be added for transparency.
  4. [Table 1] The 'reference-free SARI' variant is non-standard: SARI normally requires human references and a 'keep' component. The paper acknowledges this, but a brief description or citation of the variant would help readers interpret the numbers. Also clarify whether all metrics are computed on the 47-item subset consistently.
  5. [Abstract/Intro] The paper uses 'understanding' and 'comprehension' interchangeably (e.g., abstract says 'comprehensibility' and 'understanding'; §1 says 'understanding'). Pick one term and use it consistently.
  6. [§4.1] Figure 5 shows raw counts but no percentages. Given the total is 92 per dimension, percentages would improve readability. Also consider reporting the 73/92 preference for understanding as a proportion (e.g., 79%).

Circularity Check

0 steps flagged

No circularity: corpus construction and descriptive evaluations are empirical, not a self-referential derivation.

full rationale

This paper does not present a derived chain where an output is equivalent to an input by construction. The contributions are an empirically constructed corpus (original SciSummNet summaries, GPT-4o-mini outputs, Phase 1 reader annotations, Phase 2 expert edits) and descriptive evaluations. No parameter is fitted to a subset and then 'predicted' on a closely related quantity: the Phase 1 preference counts (e.g., 73/92 for Understanding) are directly reported human judgments, and the Phase 2 automatic metrics (Table 1) are standard overlap/readability scores computed on the released texts. The claimed value of the corpus for training and benchmarking is a forward-looking resource claim, not a prediction validated against its own annotations. The authors explicitly acknowledge the key limitation that expert-edited references 'were not independently evaluated by the target readers' and that 'these observations should be treated as complementary evidence rather than proof that either condition is uniformly superior.' That is an external-validity caveat, not circularity: the expert edits are not defined in terms of the target-reader judgments, and no conclusion is forced by self-citation or by definition. The manuscript is therefore self-contained against external benchmarks and warrants a score of 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

The paper introduces no new theoretical entities or fitted parameters; its contribution is empirical. The key assumptions are about the validity of the annotation protocol and the superiority of expert edits, both acknowledged as limitations.

axioms (4)
  • domain assumption SciSummNet expert-written summaries are appropriate source texts for measuring simplification quality.
    The corpus uses only 150-word expert summaries, not full papers; results may not extend to other text types (Section 3.1).
  • domain assumption Cross-disciplinary STEM readers' difficulty annotations are reliable signals of comprehension barriers.
    The authors note participants may conflate unfamiliar terminology with linguistic complexity (Section 5).
  • domain assumption Expert-edited simplifications are better references than GPT baselines.
    Expert-edited outputs were not independently evaluated by target readers; the paper treats them as references based on qualitative inspection (Section 4.2/5).
  • domain assumption GPT-4o-mini zero-shot outputs are a representative LLM baseline.
    Single model and prompt; the paper acknowledges model dependence (Section 5).

reviewed 2026-08-01 · how reviews work

0 comments
Cite this review

Pith. "Pith review of A Human-in-the-Loop Corpus for LLM-Based Simplification of Scientific Summaries." pith.science (2026). https://pith.science/paper/KC6PEQGZ

@misc{pith2026260725630,
  author       = {Pith},
  title        = {Pith review of: A Human-in-the-Loop Corpus for LLM-Based Simplification of Scientific Summaries},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KC6PEQGZ}},
  note         = {Machine review of arXiv:2607.25630}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Interdisciplinary research is accelerating, yet scientific papers remain difficult to understand outside their home fields. We study large language model (LLM)-based simplification of scientific texts and present a human-in-the-loop workflow that transforms expert summaries into more accessible versions for non-specialists. Using SciSummNet as the source corpus, we first generate baseline simplifications with GPT-4o-mini. In Phase 1, readers from STEM fields outside computer science identify difficult sentences and phrases and compare the original and GPT-simplified summaries in terms of comprehensibility, naturalness, and simplicity. In Phase 2, computer science experts use this feedback to create expert-edited reference simplifications. We release the resulting corpus together with human judgments and automatic evaluation results. The Phase 1 judgments show a clear preference for the GPT-generated summaries in terms of comprehensibility and simplicity, while qualitative analysis of the Phase 2 edits highlights the importance of preserving domain-specific terminology and the strength of scientific claims. The resulting resource supports the training and benchmarking of simplification systems for cross-disciplinary scientific communication.

Figures

Figures reproduced from arXiv: 2607.25630 by Kyuri Im, Michael F\"arber.

Figure 1
Figure 1. Figure 1: Interface for selecting difficult sentences in Phase 1. 3.2. Baseline Simplification We generate initial simplified summaries using GPT-4o-mini in a zero-shot setting, i.e., without providing input-output demonstrations. The system prompt asks the model to simplify the text for readers without expertise in the scientific field and for the general public, identify complex words or key phrases, and preserve … view at source ↗
Figure 2
Figure 2. Figure 2: Interface for comparing summaries by understanding, naturalness, and simplicity in Phase 1 [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Interface for highlighting difficult words and phrases in Phase 1. Phase 1 indicating difficult words or phrases. Procedure. Experts created an expert-edited reference simplification following four guided scenarios: 1. Both summaries contain difficult spans: refine and merge content for clarity and fidelity. 2. Only Original flagged: verify and edit GPT simplifications where necessary. 3. Only GPT flagged:… view at source ↗
Figure 4
Figure 4. Figure 4: Interface for creating expert-edited reference simplifications in Phase 2. Understanding Naturalness Simplicity 0 20 40 60 80 73 36 70 11 14 8 8 42 Number of Judgments 14 GPT-Simplified Original No Difference [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Comparative judgments across the three Phase 1 evaluation dimensions. Each dimension comprises 92 judgments; no inferential significance test was conducted. 4. Results We evaluate (i) how cross-disciplinary STEM readers perceive LLM-generated simplifications of scientific summaries in Phase 1 and (ii) how expert-edited reference simplifications compare with the LLM outputs according to automatic metrics in… view at source ↗
Figure 6
Figure 6. Figure 6: Example participant feedback on language complexity in scientific summaries. Qualitative Insights. Participant comments and highlights, shown in Figures 6 and 7, reveal two recurring tensions: (1) Precision vs. Approachability: replacing terms such as latent variables, PCFG, or NP-hard with generic paraphrases can improve readability but reduce technical precision; and (2) Style and Naturalness: the LLM of… view at source ↗
Figure 7
Figure 7. Figure 7: Example participant feedback highlighting clarity issues in scientific summaries. Category Metric and scale GPT-Simplified Expert-Edited Source similarity Mean sentence BLEU (0–1, ↑) 0.949 0.349 Mean BERTScore F1 (−1–1, ↑) 0.9907 0.9031 Simplification Reference-free SARI (0–100, ↑) 46.76 33.72 Readability FK Grade Level (no fixed bounds, ↓) 12.10 13.00 FK Reading Ease (no fixed bounds, ↑) 48.54 40.89 [PIT… view at source ↗
Figure 8
Figure 8. Figure 8: Comparison of an original, GPT-simplified, and expert-edited summary from Phase 2. Readability. The GPT-simplified summaries receive a lower FKGL value (12.10 vs. 13.00) and a higher FKRE value (48.54 vs. 40.89). The readability formulas therefore favor the GPT outputs in terms of surface-level features. However, these formulas do not assess terminological accuracy, conceptual clarity, or scientific claim … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 6 canonical work pages · 2 internal anchors

  1. [1]

    The advantages of interdisciplinarity in modern science

    M. Bonaventura, V. Latora, V. Nicosia, P. Panzarasa, The advantages of interdisciplinarity in modern science, arXiv preprint (2017). URL: https://doi.org/10.48550/arXiv.1712.07910

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1712.07910 2017

  2. [2]

    Cunningham, B

    E. Cunningham, B. Smyth, D. Greene, Collaboration in the time of COVID: A scientometric analysis of multidisciplinary SARS-CoV-2 research, Humanities and Social Sciences Communications 8 (2021) 240. URL: https://doi.org/10.1057/s41599-021-00922-7

    work page doi:10.1057/s41599-021-00922-7 2021

  3. [3]

    S. S. Al-Thanyyan, A. M. Azmi, Automated text simplification: A survey, ACM Computing Surveys 54 (2022). URL: https://doi.org/10.1145/3442695. 10 Kyuri Im et al.CEUR Workshop Proceedings1–11

    work page doi:10.1145/3442695 2022

  4. [4]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in Neural Information Processing Systems 33 (2020) 1877–1901

    2020

  5. [5]

    J. Wei, M. Bosma, V. Y. Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, Q. V. Le, Finetuned language models are zero-shot learners, arXiv preprint (2021). URL: https://doi.org/10.48550/arXiv .2109.01652

    Pith/arXiv arXiv doi:10.48550/arxiv.2109.01652 2021

  6. [6]

    Engelmann, F

    B. Engelmann, F. Haak, C. K. Kreutz, N. Nikzad-Khasmakhi, P. Schaer, Text simplification of scientific texts for non-expert readers, in: Working Notes of the Conference and Labs of the Evaluation Forum (CLEF 2023), 2023, pp. 2987–2998. URL: https://ceur-ws.org/Vol-3497/paper-250 .pdf

    2023

  7. [7]

    LLM-based Text Simplification and its Effect on User Comprehension and Cognitive Load

    T. Guidroz, D. Ardila, J. Li, A. Mansour, P. Jhun, N. Gonzalez, X. Ji, M. Sanchez, S. Kakarmath, M. Bellaiche, M. A. Garrido, F. Ahmed, D. Choudhary, J. Hartford, C. Xu, H. J. Serrano Echeverria, Y. Wang, J. Shaffer, E. Y. Cao, Y. Matias, A. Hassidim, D. R. Webster, Y. Liu, S. Fujiwara, P. Bui, Q. Duong, LLM-based text simplification and its effect on use...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2505.01980 2025

  8. [8]

    Qiang, M

    J. Qiang, M. Huang, Y. Zhu, Y. Yuan, C. Zhang, K. Yu, Redefining simplicity: Benchmarking large language models from lexical to document simplification, arXiv preprint (2025). URL: https://doi.org/10.48550/arXiv.2502.08281

    Pith/arXiv arXiv doi:10.48550/arxiv.2502.08281 2025

  9. [9]

    Yasunaga, J

    M. Yasunaga, J. Kasai, R. Zhang, A. R. Fabbri, I. Li, D. Friedman, D. R. Radev, ScisummNet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks, Proceedings of the AAAI Conference on Artificial Intelligence 33 (2019) 7386–7393. URL: https://doi.org/10.1609/aaai.v33i01.33017386

    doi:10.1609/aaai.v33i01.33017386 2019

  10. [10]

    Papineni, S

    K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318

    2002

  11. [11]

    Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, 2004, pp

    C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries, in: Text Summarization Branches Out, 2004, pp. 74–81. URL: https://aclanthology.org/W04-1013/

    2004

  12. [12]

    J. P. Kincaid, J. Fishburne, Robert P., R. L. Rogers, B. S. Chissom, Derivation of New Readability Formulas (Automated Readability Index, Fog Count and Flesch Reading Ease Formula) for Navy Enlisted Personnel, Technical Report Research Branch Report 8-75, Chief of Naval Technical Training, 1975. URL: https://stars.library.ucf.edu/istlibrary/56/

    1975

  13. [13]

    Sulem, O

    E. Sulem, O. Abend, A. Rappoport, BLEU is not suitable for the evaluation of text simplification, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, pp. 738–744. URL: https://doi.org/10.18653/v1/D18-1081

    doi:10.18653/v1/d18-1081 2018

  14. [14]

    Siddharthan, Syntactic simplification and text cohesion, Research on Language and Computation 4 (2006) 77–109

    A. Siddharthan, Syntactic simplification and text cohesion, Research on Language and Computation 4 (2006) 77–109. URL: https://doi.org/10.1007/s11168-006-9011-1

    work page doi:10.1007/s11168-006-9011-1 2006

  15. [15]

    Vajjala, D

    S. Vajjala, D. Meurers, Readability assessment for text simplification: From analysing documents to identifying sentential simplifications, ITL - International Journal of Applied Linguistics 165 (2014) 194–222. URL: https://doi.org/10.1075/itl.165.2.04vaj

    work page doi:10.1075/itl.165.2.04vaj 2014

  16. [16]

    W. Xu, C. Callison-Burch, C. Napoles, Problems in current text simplification research: New data can help, Transactions of the Association for Computational Linguistics 3 (2015) 283–297. URL: https://doi.org/10.1162/tacl_a_00139

    doi:10.1162/tacl_a_00139 2015

  17. [17]

    D. Kauchak, Improving text simplification language modeling using unsimplified text data, in: Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, 2013, pp. 1537–1546. URL: https://aclanthology.org/P13-1151/

    2013

  18. [18]

    Zaman, F

    F. Zaman, F. Kamiran, M. Shardlow, S.-U. Hassan, A. Karim, N. R. Aljohani, SATS: Simplification- aware text summarization of scientific documents, Frontiers in Artificial Intelligence 7 (2024) 1375419. URL: https://doi.org/10.3389/frai.2024.1375419. 11

    arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.