REVIEW 4 major objections 3 minor 1 cited by
JEBS: A Fine-grained Biomedical Lexical Simplification Task
T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper presents JEBS, a fine-grained biomedical lexical simplification dataset of 21,595 hand-labeled replacements for 10,314 expert terms, and defines a three-part task—identify, classify, generate—to train systems that replace or…
desk verdict JEBS is a genuinely useful fine-grained lexical simplification resource; the identification benchmark's coverage bias and the self-performed human evaluation are the main things to fix before it becomes a standard. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the JEBS task decomposition itself: identification of expert terms, classification into five replacement types, and generation of replacement spans. The machinery that ties the dataset together is the annotation of in situ term–replacement pairs within parallel abstracts, linking each expert term to the specific span in the simplified version that handles it. This design lets each sub-task be trained and measured independently, and it also provides context that disambiguates acronyms and supports context-dependent decisions such as when generalization is appropriate.
What would settle it
A direct check would be to take a fresh sample of biomedical abstracts, have independent annotators apply the JEBS annotation protocol, and compare their term spans and replacement-type labels to the published JEBS labels; if agreement falls near chance or if annotators systematically disagree on which replacement type applies, the gold standard's reliability is not established. Alternatively, one could test whether models trained on JEBS outperform end-to-end models on a held-out clinical reading-comprehension task; if they do not, the fine-grained decomposition may not translate into practical gains.
Extended reading notes
Core claim
The paper's central claim is that fine-grained, term-level supervision for biomedical simplification is feasible and useful, and that the JEBS dataset makes it possible. JEBS is built by annotating the PLABA parallel corpus, where each complex term in an original abstract is linked to its replacement span in a human simplification, and that replacement is labeled as one of five types: substitution, explanation, generalization, exemplification, or omission. The paper further argues that this decomposition supports targeted training and evaluation that end-to-end corpora cannot, and it demonstrates the task by reporting baselines for rule-based and transformer models on each sub-task, including an end-to-end pipeline.
Load-bearing premise
The load-bearing premise is that the human simplifications in the source corpus can be reliably decomposed into the five replacement types, and that the annotations—which show only moderate inter-annotator agreement—are accurate enough to serve as gold-standard training and evaluation data.
Editorial extensions
If this is right
- Systems trained on JEBS can be evaluated on each simplification step in isolation, yielding diagnostics for whether a failure is due to missing a term, choosing the wrong strategy, or generating a bad replacement.
- The five-type taxonomy gives a standard vocabulary for describing and comparing biomedical simplification quality, replacing coarse similarity measures that mix unrelated edit types.
- The dataset's context-rich annotation enables work on acronym disambiguation and context-dependent simplification decisions, such as choosing generalization only when a term is peripheral.
- Baseline results provide a reference point for future work: transformer-based identification, BERT-based NER classification, and prompted LLMs for generation.
Reading between the lines
- If JEBS is adopted, fine-grained simplification datasets may enable controllable simplification where users choose which strategy to apply, rather than accepting an end-to-end rewrite.
- The moderate inter-annotator agreement suggests that term identification and classification are genuinely ambiguous; a promising extension would be to model simplification as a distribution over types rather than a single label, or to use the annotator disagreement as a signal for difficulty.
- The class imbalance (substitutions dominate) may push future work toward few-shot or synthetic-data methods for rare types like exemplification, which the paper itself notes as a future direction.
- One could test whether training on JEBS improves downstream readability of full abstracts compared with end-to-end simplification, using a human-preference or comprehension study.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces JEBS, a fine-grained biomedical lexical simplification dataset derived from 400 PLABA abstracts and their human simplifications. The JEBS task decomposes lexical simplification into three sub-tasks: identification of expert terms, classification of the simplification type (substitution, explanation, generalization, exemplification, omission), and generation of replacement text. The dataset contains 10,314 expert terms and 21,595 labeled simplifications. The authors provide baseline experiments for rule-based and transformer-based models on each sub-task, including an end-to-end system, and report automated metrics for identification/classification and manual evaluations for generation. The paper claims that JEBS enables rigorous evaluation of biomedical lexical simplification systems.
Significance. The dataset is a substantial contribution. It is larger than prior fine-grained lexical simplification resources (e.g., Med-EASi) and offers a finer-grained taxonomy that separates explanations from exemplifications and substitutions from generalizations. The task decomposition is a useful step toward targeted evaluation of lexical simplification, rather than end-to-end simplification. The authors provide a range of baselines, including encoder-only, encoder-decoder, and decoder-only models, and the data and code are publicly available. If the identified methodological issues are addressed, JEBS would be a valuable resource for the community. The paper is honest in reporting moderate inter-annotator agreement and acknowledges several limitations, which is commendable.
major comments (4)
- [Section 3.1 vs. Section 4] There is a mismatch between the definition of the identification sub-task and the gold standard used to evaluate it. Section 3.1 defines identification as 'labeling terms in a given text as expert terms,' which implies an exhaustive annotation of complex terms. However, Section 4 states that expert terms were selected and 'linked with their respective simplifications, as found in the PLABA adaptations.' This procedure records only terms that the PLABA simplifiers actually replaced or explained; complex terms left unmodified by the simplifiers are absent from the gold set. As a result, a system that correctly identifies a complex term that PLABA did not simplify is penalized as a false positive. The paper provides no evidence that the PLABA simplifications are exhaustive in their treatment of complex terms, and the moderate identification IAA (F1 = 0.52) is consistent with annotators agreeing on terms that happen to have simplifications rather than on an independent notion of complexity. This coverage bias means the reported identification F1 does not measure the stated task, and the claim that JEBS enables 'rigorous evaluation' of the identification stage is not supported as written. The authors should either redefine the task to 'identify terms that need simplification' and frame the gold standard accordingly, or augment the data with annotations of unsimplified complex terms.
- [Section 5.2] The NER-based classification baselines are trained with a single label per term ('the simplification method assigned to them most often in the training data'), but evaluated against the union of both annotators' labels using average F1 and union F1. Since JEBS contains 21,595 simplifications for 10,314 terms, a substantial fraction of terms have more than one simplification type. A model trained to output one label per term cannot match a multi-label gold standard, and the reported scores are therefore depressed in a way that does not reflect the model's true ability on the stated classification task. The evaluation protocol should be aligned with the training setup, e.g., by using a multi-label objective and metric, or by evaluating on a single-label version of the gold data with clear handling of terms with multiple types.
- [Section 5.3 and Table 5] The manual evaluation of the generation sub-task was conducted entirely by the authors, with a single expert per output (5,760 judgments). No inter-annotator agreement or variance statistics are reported, and the 5-point Likert ratings are linearly interpolated to 0-1 scores without validation that the intervals are equal. The conclusion that GPT-4o-mini generally outperforms the other baselines (Table 5) rests on ratings from a single rater per item, so the ranking may not be reliable. Section 9 acknowledges that multiple experts would yield more reliable results, but the acknowledgment does not mitigate the fact that the baseline comparisons in Table 5 are presented without reliability evidence. The authors should either provide multiple annotations per output with agreement statistics, or clearly present the results as informal case studies rather than rigorous benchmark scores.
- [Appendix A, Algorithm 1] Algorithm 1 as written initializes D' to the empty set and only iterates over entries in S (terms with synonyms). Consequently, any term that is not itself listed as a synonym of another term is omitted from the output dictionary D'. If this algorithm is exactly what was used to construct the dataset, the published construction is incorrect; if the implementation instead copies D into D' first, the pseudocode does not match the procedure. This is a load-bearing detail because the synonym-linking step directly affects which simplifications are associated with each expert term in the released dataset. The authors should correct the pseudocode (e.g., initialize D' with a copy of D) and clarify the intended behavior for terms without synonyms.
minor comments (3)
- [Section 3.2.2] The text contains a typo: 'a definition' should be 'a definition' in the sentence 'which may be long and/or complicated' (referring to a definition).
- [Section 5] The phrase 'split was performed at the question-level' is unclear to readers not familiar with PLABA. Please briefly explain what a 'question' is in the PLABA structure and how this split avoids contamination.
- [Table 5] The caption says 'Aggregate evaluation results' but the aggregation method across the four axes (simplicity, accuracy, completeness, brevity) is not defined in the text or caption. Readers should be told that the aggregate is the mean of the four 0-1 scores, or the caption should be revised.
Circularity Check
No significant circularity: JEBS is a new annotation layer over the published PLABA corpus, and no baseline prediction reduces to a fitted input.
full rationale
The paper's derivation chain is self-contained in the relevant sense. JEBS adds a new manual annotation layer (expert-term spans, replacement-type labels, synonym links) on top of the published PLABA corpus; the five-type taxonomy is defined from the lexical-simplification literature, and the baselines use external resources (MetaMapLite/UMLS, pretrained transformers, off-the-shelf LLMs) evaluated on held-out data. The PLABA citations (Attal et al. 2023; Ondov et al. 2022) involve overlapping authors, but PLABA is a peer-reviewed published dataset and the JEBS annotations are new, so the self-citation is not load-bearing in the sense of a uniqueness theorem or an unverified premise. The identification gold set is constructed by selecting terms that the PLABA simplifiers actually replaced or explained (Section 4), so unsimplified complex terms are unlabeled; this is a real coverage/validity limitation for the identification sub-task, as is the moderate inter-annotator agreement (identification F1 0.52, classification F1 0.46), but neither makes a reported result equivalent to its input by construction: the models never see the gold simplifications at prediction time, and the classification and generation sub-tasks have independent gold labels and human evaluation. These caveats belong under data-quality risk rather than circularity.
Assumptions & free parameters
assumptions (3)
- domain assumption PLABA simplifications are high-quality expert rewrites suitable as a gold-standard source.
- ad hoc to paper The five simplification types (substitution, explanation, generalization, exemplification, omission) are exhaustive and mutually exclusive for the annotation task.
- ad hoc to paper The synonym-linking algorithm (Appendix A) correctly merges simplifications of synonymous terms.
Cite this review
Pith. "Pith review of JEBS: A Fine-grained Biomedical Lexical Simplification Task." pith.science (2026). https://pith.science/paper/T6KTBUHY
@misc{pith2026250612898,
author = {Pith},
title = {Pith review of: JEBS: A Fine-grained Biomedical Lexical Simplification Task},
year = {2026},
howpublished = {\url{https://pith.science/paper/T6KTBUHY}},
note = {Machine review of arXiv:2506.12898}
}
read the original abstract
Online medical literature has made health information more available than ever, however, the barrier of complex medical jargon prevents the general public from understanding it. Though parallel and comparable corpora for Biomedical Text Simplification have been introduced, these conflate the many syntactic and lexical operations involved in simplification. To enable more targeted development and evaluation, we present a fine-grained lexical simplification task and dataset, Jargon Explanations for Biomedical Simplification (JEBS, https://github.com/bill-from-ri/JEBS-data ). The JEBS task involves identifying complex terms, classifying how to replace them, and generating replacement text. The JEBS dataset contains 21,595 replacements for 10,314 terms across 400 biomedical abstracts and their manually simplified versions. Additionally, we provide baseline results for a variety of rule-based and transformer-based systems for the three sub-tasks. The JEBS task, data, and baseline results pave the way for development and rigorous evaluation of systems for replacing or explaining complex biomedical terms.
Figures
Forward citations
Cited by 1 Pith paper
-
Lessons from the TREC Plain Language Adaptation of Biomedical Abstracts (PLABA) track
Across two TREC shared-task years, top LLM systems matched human writers on factual accuracy and completeness but not on simplicity or brevity, while common automatic metrics correlated poorly with manual judgments.
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. https://doi.org/10.1162/coli_a_00418 The (un)suitability of automatic evaluation metrics for text simplification . Computational Linguistics, 47(4):861--889
-
[4]
Kush Attal, Brian Ondov, and Dina Demner-Fushman. 2023. https://doi.org/10.1038/s41597-022-01920-3 A dataset for plain language adaptation of biomedical abstracts . Scientific Data, 10(1)
-
[5]
Chandrayee Basu, Rosni Vasu, Michihiro Yasunaga, and Qian Yang. 2023. https://arxiv.org/abs/2302.09155 Med-easi: Finely annotated dataset and models for controllable simplification of medical texts . Preprint, arXiv:2302.09155
work page Pith review arXiv 2023
-
[6]
Nancy D Berkman, Stacey L Sheridan, Katrina E Donahue, David J Halpern, and Karen Crotty. 2011. Low health literacy and health outcomes: an updated systematic review. Annals of internal medicine, 155(2):97--107
work page 2011
-
[7]
Sleeman, Jatinder Palta, Rishabh Kapoor, and Preetam Ghosh
Priyankar Bose, Sriram Srinivasan, William C. Sleeman, Jatinder Palta, Rishabh Kapoor, and Preetam Ghosh. 2021. https://doi.org/10.3390/app11188319 A survey on recent named entity recognition and relationship extraction techniques on clinical texts . Applied Sciences, 11(18):8319
-
[8]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...
arXiv 2020
Show all 40 references
-
[9]
Yixin Cao, Ruihao Shui, Liangming Pan, Min-Yen Kan, Zhiyuan Liu, and Tat-Seng Chua. 2020. Expertise style transfer: A new task towards better communication between experts and laymen. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, p...
2020
-
[10]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \' a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . C...
2019 arXiv
-
[11]
Todd Cook. 2020. https://www.kaggle.com/datasets/toddcook/bert-english-uncased-bigrams Bert english uncased bigrams
2020
-
[12]
Dina Demner-Fushman, Willie J Rogers, and Alan R Aronson. 2017. https://doi.org/10.1093/jamia/ocw177 MetaMap Lite: an evaluation of a new Java implementation of MetaMap . Journal of the American Medical Informatics Association, 24(4):841--844
2017 doi
-
[13]
Ashwin Devaraj, Iain Marshall, Byron Wallace, and Junyi Jessy Li. 2021. https://doi.org/10.18653/v1/2021.naacl-main.395 Paragraph-level simplification of medical texts . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Li...
2021 doi
-
[14]
Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. https://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805
2018 arXiv
-
[15]
Yue Dong, Zichao Li, Mehdi Rezagholizadeh, and Jackie Chi Kit Cheung. 2019. https://arxiv.org/abs/1906.08104 Editnts: An neural programmer-interpreter model for sentence simplification through explicit editing . Preprint, arXiv:1906.08104
2019 arXiv
-
[16]
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783
2024 arXiv
-
[17]
Noemie Elhadad. 2006. Comprehending technical texts: predicting and defining unfamiliar terms. AMIA Annu. Symp. Proc., pages 239--243
2006
-
[18]
Gary N Fox and Nashat S Moawad. 2003. Uptodate: a comprehensive clinical database. Journal of family practice, 52(9):706--710
2003
-
[19]
Yue Guo, Wei Qiu, Gondy Leroy, Sheng Wang, and Trevor Cohen. 2022. Cells: A parallel corpus for biomedical lay language generation. arXiv preprint arXiv:2211.03818
2022 arXiv
-
[20]
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://openreview.net/forum?id=XPZIaotutsD Deberta: Decoding-enhanced bert with disentangled attention . In International Conference on Learning Representations
2021
-
[21]
Sasikiran Kandula, Dorothy Curtis, and Qing Zeng-Treitler. 2010. A semantic and syntactic text simplification tool for health content. AMIA Annu. Symp. Proc., 2010:366--370
2010
-
[22]
David A Kindig, Allison M Panzer, and Lynn Nielsen-Bohlman. 2004. Health Literacy: A Prescription to End Confusion. National Academies Press
2004
-
[23]
Alexandra King. 2010. Poor health literacy: a 'hidden' risk factor. Nature Reviews Cardiology, 7(9):473--474
2010
-
[24]
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. https://arxiv.org/abs/1901.08746 Biobert: a pre-trained biomedical language representation model for biomedical text mining . CoRR, abs/1901.08746
2019 arXiv
-
[25]
Donald AB Lindberg, Betsy L Humphreys, and Alexa T McCray. 1993. The unified medical language system. Yearbook of medical informatics, 2(01):41--51
1993
-
[26]
Zequn Liu, Shukai Wang, Yiyang Gu, Ruiyi Zhang, Ming Zhang, and Sheng Wang. 2021. https://arxiv.org/abs/2109.04018 Graphine: A dataset for graph-aware terminology definition generation . Preprint, arXiv:2109.04018
2021 arXiv
-
[27]
Meta. 2025. https://huggingface.co/meta-llama/Llama-4-Scout-17B-16E-Instruct Llama 4 scout instruct
2025
-
[28]
Naomi Miller, Eve-Marie Lacroix, and Joyce EB Backus. 2000. Medlineplus: building and maintaining the national library of medicine's consumer health web service. Bulletin of the Medical Library Association, 88(1):11
2000
-
[29]
Ani Nenkova and Rebecca Passonneau. 2004. https://aclanthology.org/N04-1019 Evaluating content selection in summarization: The pyramid method . In Proceedings of the Human Language Technology Conference of the North A merican Chapter of the Association for Computational Lingui...
2004
-
[30]
Sergiu Nisioi, Sanja S tajner, Simone Paolo Ponzetto, and Liviu P. Dinu. 2017. https://doi.org/10.18653/v1/P17-2014 Exploring neural text simplification models . In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)...
2017 doi
-
[31]
Brian Ondov, Kush Attal, and Dina Demner-Fushman. 2022. A survey of automated methods for biomedical text simplification. J. Am. Med. Inform. Assoc., 29(11):1976--1988
2022
-
[32]
OpenAI. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276
2024 arXiv
-
[33]
Atharva Phatak, David W Savage, Robert Ohle, Jonathan Smith, and Vijay Mago. 2022. https://doi.org/10.2196/38095 Medical text simplification using reinforcement learning (teslea): Deep learning–based text simplification approach . JMIR Medical Informatics, 10(11):e38095
2022 doi
-
[34]
Neha Srikanth and Junyi Jessy Li. 2020. https://arxiv.org/abs/2010.10035 Elaborative simplification: Content addition and explanation generation in text simplification . CoRR, abs/2010.10035
2020 arXiv
-
[35]
Pontus Stenetorp, Sampo Pyysalo, Goran Topi\' c , Tomoko Ohta, Sophia Ananiadou, and Jun'ichi Tsujii. 2012. brat : a web-based tool for NLP -assisted text annotation. In Proceedings of the Demonstrations Session at EACL 2012 , Avignon, France. Association for Computational Linguistics
2012
-
[36]
Rachael Tatman. 2020. https://www.kaggle.com/datasets/rtatman/english-word-frequency English word frequency
2020
-
[37]
Laurens Van den Bercken, Robert-Jan Sips, and Christoph Lofi. 2019. Evaluating neural text simplification in the medical domain. In The World Wide Web Conference, pages 3286--3292
2019
-
[38]
David L Wheeler, Tanya Barrett, Dennis A Benson, Stephen H Bryant, Kathi Canese, Vyacheslav Chetvernin, Deanna M Church, Michael DiCuccio, Ron Edgar, Scott Federhen, et al. 2007. Database resources of the national center for biotechnology information. Nucleic acids research, 3...
2007
-
[39]
Ryen W White and Eric Horvitz. 2009. Cyberchondria: studies of the escalation of medical concerns in web search. ACM Transactions on Information Systems (TOIS), 27(4):1--37
2009
-
[40]
Qing T Zeng, Tony Tse, Jon Crowell, Guy Divita, Laura Roth, and Allen C Browne. 2005. Identifying consumer-friendly display ( CFD ) names for health concepts. AMIA Annu. Symp. Proc., pages 859--863
2005
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.