{"id":"2eef80bc-fa42-4c61-9d62-46e4ebcd3e81","arxiv_id":"2506.12898","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"JEBS is a fine-grained dataset of expert-annotated replacements for biomedical terms, supporting identification, classification, and generation sub-tasks for lexical simplification.","lead":"This paper introduces JEBS, a new dataset of 21,595 plain-language replacements for 10,314 complex medical terms in 400 research abstracts. It splits the simplification task into identifying difficult terms, deciding how to simplify them, and generating the replacement, which allows more accurate training and testing of simplification systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The JEBS identification gold set only contains terms that the PLABA simplifiers chose to rewrite; unsimplified complex terms are unlabeled, making the identification benchmark biased.","rationale":"The reader flagged moderate inter-annotator agreement and decomposability of the PLABA simplifications as the weakest assumption. I agree those are real concerns, but the more fundamental, less discussed issue is coverage: the gold standard for identification is constructed from the set of terms that were simplified in PLABA, so it may exclude complex terms that no simplifier touched. This is a correctness risk in the benchmark itself, not just noise in the labels. It directly affects the central claim that JEBS supports rigorous evaluation of the identification sub-task. The proposed concrete test would settle whether the coverage gap is large enough to matter. I do not think this warrants rejecting the paper—the resource is still useful, especially for the generation and classification sub-tasks—but it strengthens the existing conditional verdict: the identification benchmark should be reframed as detecting terms simplified in PLABA, or supplemented with exhaustive complexity annotations, before being adopted as a general-purpose benchmark.","tokens_in":12791,"tokens_out":6560,"duration_ms":74159,"concrete_test":"Randomly select 40 of the 400 JEBS abstracts. Have two independent annotators (not authors, blind to PLABA simplifications) mark all non-consumer biomedical terms in each original abstract. Compute recall of the JEBS gold term set against the union of these independent annotations (i.e., |JEBS ∩ independent| / |independent|). If recall is substantially below 1.0 (e.g., <0.85), the gold standard omits many complex terms and the identification task measures a biased subset rather than expert-term identification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4 describes constructing JEBS by 'selecting expert terms and linking them with their respective simplifications, as found in the PLABA adaptations.' This procedure records only terms that the PLABA simplifiers actually replaced or explained; any genuinely complex term that the simplifier left untouched is absent from the gold set. Yet Section 3.1 defines identification as 'labeling terms in a given text as expert terms'—an exhaustive judgment of complexity. Consequently, the identification sub-task is not evaluated against an independent, exhaustive annotation of expert terms: a system that correctly identifies a complex term that PLABA did not simplify is penalized as a false positive. The paper never demonstrates that the PLABA simplifications are exhaustive in their treatment of complex terms, and the moderate identification IAA (F1 0.52) is consistent with annotators agreeing on the terms that happen to have simplifications rather than on an independent notion of complexity. This coverage bias undermines the central claim that JEBS enables 'rigorous evaluation' of the identification stage.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces JEBS, a fine-grained biomedical lexical simplification dataset derived from 400 PLABA abstracts and their human simplifications. The JEBS task decomposes lexical simplification into three sub-tasks: identification of expert terms, classification of the simplification type (substitution, explanation, generalization, exemplification, omission), and generation of replacement text. The dataset contains 10,314 expert terms and 21,595 labeled simplifications. The authors provide baseline experiments for rule-based and transformer-based models on each sub-task, including an end-to-end system, and report automated metrics for identification/classification and manual evaluations for generation. The paper claims that JEBS enables rigorous evaluation of biomedical lexical simplification systems.","tokens_in":13017,"tokens_out":6206,"duration_ms":70033,"significance":"The dataset is a substantial contribution. It is larger than prior fine-grained lexical simplification resources (e.g., Med-EASi) and offers a finer-grained taxonomy that separates explanations from exemplifications and substitutions from generalizations. The task decomposition is a useful step toward targeted evaluation of lexical simplification, rather than end-to-end simplification. The authors provide a range of baselines, including encoder-only, encoder-decoder, and decoder-only models, and the data and code are publicly available. If the identified methodological issues are addressed, JEBS would be a valuable resource for the community. The paper is honest in reporting moderate inter-annotator agreement and acknowledges several limitations, which is commendable.","major_comments":[{"comment":"There is a mismatch between the definition of the identification sub-task and the gold standard used to evaluate it. Section 3.1 defines identification as 'labeling terms in a given text as expert terms,' which implies an exhaustive annotation of complex terms. However, Section 4 states that expert terms were selected and 'linked with their respective simplifications, as found in the PLABA adaptations.' This procedure records only terms that the PLABA simplifiers actually replaced or explained; complex terms left unmodified by the simplifiers are absent from the gold set. As a result, a system that correctly identifies a complex term that PLABA did not simplify is penalized as a false positive. The paper provides no evidence that the PLABA simplifications are exhaustive in their treatment of complex terms, and the moderate identification IAA (F1 = 0.52) is consistent with annotators agreeing on terms that happen to have simplifications rather than on an independent notion of complexity. This coverage bias means the reported identification F1 does not measure the stated task, and the claim that JEBS enables 'rigorous evaluation' of the identification stage is not supported as written. The authors should either redefine the task to 'identify terms that need simplification' and frame the gold standard accordingly, or augment the data with annotations of unsimplified complex terms.","section":"Section 3.1 vs. Section 4"},{"comment":"The NER-based classification baselines are trained with a single label per term ('the simplification method assigned to them most often in the training data'), but evaluated against the union of both annotators' labels using average F1 and union F1. Since JEBS contains 21,595 simplifications for 10,314 terms, a substantial fraction of terms have more than one simplification type. A model trained to output one label per term cannot match a multi-label gold standard, and the reported scores are therefore depressed in a way that does not reflect the model's true ability on the stated classification task. The evaluation protocol should be aligned with the training setup, e.g., by using a multi-label objective and metric, or by evaluating on a single-label version of the gold data with clear handling of terms with multiple types.","section":"Section 5.2"},{"comment":"The manual evaluation of the generation sub-task was conducted entirely by the authors, with a single expert per output (5,760 judgments). No inter-annotator agreement or variance statistics are reported, and the 5-point Likert ratings are linearly interpolated to 0-1 scores without validation that the intervals are equal. The conclusion that GPT-4o-mini generally outperforms the other baselines (Table 5) rests on ratings from a single rater per item, so the ranking may not be reliable. Section 9 acknowledges that multiple experts would yield more reliable results, but the acknowledgment does not mitigate the fact that the baseline comparisons in Table 5 are presented without reliability evidence. The authors should either provide multiple annotations per output with agreement statistics, or clearly present the results as informal case studies rather than rigorous benchmark scores.","section":"Section 5.3 and Table 5"},{"comment":"Algorithm 1 as written initializes D' to the empty set and only iterates over entries in S (terms with synonyms). Consequently, any term that is not itself listed as a synonym of another term is omitted from the output dictionary D'. If this algorithm is exactly what was used to construct the dataset, the published construction is incorrect; if the implementation instead copies D into D' first, the pseudocode does not match the procedure. This is a load-bearing detail because the synonym-linking step directly affects which simplifications are associated with each expert term in the released dataset. The authors should correct the pseudocode (e.g., initialize D' with a copy of D) and clarify the intended behavior for terms without synonyms.","section":"Appendix A, Algorithm 1"}],"minor_comments":[{"comment":"The text contains a typo: 'a definition' should be 'a definition' in the sentence 'which may be long and/or complicated' (referring to a definition).","section":"Section 3.2.2"},{"comment":"The phrase 'split was performed at the question-level' is unclear to readers not familiar with PLABA. Please briefly explain what a 'question' is in the PLABA structure and how this split avoids contamination.","section":"Section 5"},{"comment":"The caption says 'Aggregate evaluation results' but the aggregation method across the four axes (simplicity, accuracy, completeness, brevity) is not defined in the text or caption. Readers should be told that the aggregate is the mean of the four 0-1 scores, or the caption should be revised.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The identification coverage bias is the most serious concern and directly affects the paper's central claim of enabling rigorous evaluation. The classification evaluation mismatch and the manual evaluation reliability are also important. The paper is otherwise well-organized and the dataset is a potential contribution. I would be willing to review a revised version that addresses these issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: JEBS is worth taking seriously as a new resource for biomedical lexical simplification. It's larger and more fine-grained than Med-EASi, with 21,595 labeled replacements and a five-type taxonomy that separates explanation from exemplification and substitution from generalization. The three sub-task breakdown is a real contribution, and the data is released. The baseline suite is thorough, covering rule-based, fine-tuned encoder, and in-context learning approaches. The paper also does a decent job documenting annotation and reporting IAA.\n\nThe soft spots are in proportion to how much they matter. The stress-test note about identification is correct: the gold set only contains terms that the PLABA simplifiers actually replaced or explained. The paper defines identification as labeling all expert terms, but the evaluation set is a subset of terms that happen to have simplifications. So a system that flags a genuinely complex term the simplifier skipped is falsely penalized. That's a real coverage bias, and it undermines the \"rigorous evaluation\" claim for the identification sub-task. It doesn't wreck the classification or generation sub-tasks, which are conditioned on reference terms, but it does mean the identification numbers should be read as 'predicting what PLABA simplifiers chose to edit,' not 'finding all complex terms.' The paper should either narrow the task definition accordingly or re-annotate with an exhaustive complexity judgment.\n\nThe human evaluation is author-run, single expert per output, with no reliability statistics. For a benchmark paper, that's a minor but real weakness. The class imbalance (0.6% exemplifications) is acknowledged and is a fair limitation.\n\nThe moderate IAA (identification F1 0.52) is worth noting. It may partly reflect the coverage issue: annotators agree better on terms that have clear simplifications.\n\nBottom line: this is a solid resource paper that deserves a serious referee. The dataset fills a real gap. The identification evaluation needs clarification or re-annotation, and the human eval needs a bit more rigor, but neither issue is fatal to the resource's value. I'd cite it and would bring it to a reading group focused on text simplification or health NLP.\n\nRecommendation: send to peer review with a request for the authors to address the identification gold-set coverage and provide some reliability analysis for the manual eval.","headline":"JEBS is a genuinely useful fine-grained lexical simplification resource; the identification benchmark's coverage bias and the self-performed human evaluation are the main things to fix before it becomes a standard.","tokens_in":13467,"tokens_out":3256,"would_cite":true,"duration_ms":31279,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents JEBS, a fine-grained biomedical lexical simplification dataset of 21,595 hand-labeled replacements for 10,314 expert terms, and defines a three-part task—identify, classify, generate—to train systems that replace or…","keywords":["biomedical lexical simplification","text simplification dataset","expert term identification","replacement classification","term replacement generation","medical jargon","plain language adaptation","fine-grained annotation"],"falsifier":"A direct check would be to take a fresh sample of biomedical abstracts, have independent annotators apply the JEBS annotation protocol, and compare their term spans and replacement-type labels to the published JEBS labels; if agreement falls near chance or if annotators systematically disagree on which replacement type applies, the gold standard's reliability is not established. Alternatively, one could test whether models trained on JEBS outperform end-to-end models on a held-out clinical reading-comprehension task; if they do not, the fine-grained decomposition may not translate into practical gains.","tokens_in":12619,"feed_emoji":"🩺","tokens_out":3957,"duration_ms":37598,"temperature":0.7,"pith_summary":"The paper introduces JEBS, a fine-grained lexical simplification task for biomedical text, along with a dataset of 21,595 hand-labeled replacements for 10,314 complex terms drawn from 400 biomedical abstracts. The central proposal is that automatically simplifying medical literature should be broken into three sub-tasks: identifying expert terms, deciding which of five replacement strategies to use, and generating the replacement text. Existing parallel corpora only support end-to-end simplification, which mixes syntactic and lexical edits and makes evaluation coarse; JEBS isolates lexical operations so systems can be trained and evaluated on each step directly. A sympathetic reader should care because progress on this task would give lay readers usable plain-language versions of dense medical research, with explicit explanations instead of black-box rewriting.","feed_headline":"Fine-grained dataset isolates five ways to simplify medical jargon","feed_subtitle":"JEBS labels 21,595 replacements for 10,314 complex terms to train systems that explain medical text clearly.","key_machinery":"The central object is the JEBS task decomposition itself: identification of expert terms, classification into five replacement types, and generation of replacement spans. The machinery that ties the dataset together is the annotation of in situ term–replacement pairs within parallel abstracts, linking each expert term to the specific span in the simplified version that handles it. This design lets each sub-task be trained and measured independently, and it also provides context that disambiguates acronyms and supports context-dependent decisions such as when generalization is appropriate.","core_discovery":"The paper's central claim is that fine-grained, term-level supervision for biomedical simplification is feasible and useful, and that the JEBS dataset makes it possible. JEBS is built by annotating the PLABA parallel corpus, where each complex term in an original abstract is linked to its replacement span in a human simplification, and that replacement is labeled as one of five types: substitution, explanation, generalization, exemplification, or omission. The paper further argues that this decomposition supports targeted training and evaluation that end-to-end corpora cannot, and it demonstrates the task by reporting baselines for rule-based and transformer models on each sub-task, including an end-to-end pipeline.","pith_inferences":["If JEBS is adopted, fine-grained simplification datasets may enable controllable simplification where users choose which strategy to apply, rather than accepting an end-to-end rewrite.","The moderate inter-annotator agreement suggests that term identification and classification are genuinely ambiguous; a promising extension would be to model simplification as a distribution over types rather than a single label, or to use the annotator disagreement as a signal for difficulty.","The class imbalance (substitutions dominate) may push future work toward few-shot or synthetic-data methods for rare types like exemplification, which the paper itself notes as a future direction.","One could test whether training on JEBS improves downstream readability of full abstracts compared with end-to-end simplification, using a human-preference or comprehension study."],"forward_implications":["Systems trained on JEBS can be evaluated on each simplification step in isolation, yielding diagnostics for whether a failure is due to missing a term, choosing the wrong strategy, or generating a bad replacement.","The five-type taxonomy gives a standard vocabulary for describing and comparing biomedical simplification quality, replacing coarse similarity measures that mix unrelated edit types.","The dataset's context-rich annotation enables work on acronym disambiguation and context-dependent simplification decisions, such as choosing generalization only when a term is peripheral.","Baseline results provide a reference point for future work: transformer-based identification, BERT-based NER classification, and prompted LLMs for generation."],"supporting_citations":[{"why":"Supplies the PLABA parallel corpus of biomedical abstracts paired with expert-written simplifications, from which JEBS is derived.","marker":"Attal et al., 2023"},{"why":"Med-EASi is the closest prior dataset with deletions, elaborations, and replacements, which JEBS extends with a larger size and a finer-grained taxonomy.","marker":"Basu et al., 2023"},{"why":"Argues that automatic metrics conflate the distinct operations in text simplification, motivating the fine-grained task design.","marker":"Alva-Manchego et al., 2021"},{"why":"Survey linking term-level simplification to faithfulness, supporting the paper's choice to focus on lexical operations rather than full rewrites.","marker":"Ondov et al., 2022"},{"why":"MetaMapLite is the rule-based tool used in the identification baseline to find candidate expert terms.","marker":"Demner-Fushman et al., 2017"},{"why":"The Unified Medical Language System (UMLS) provides the lexicon for the rule-based identification and substitution components.","marker":"Lindberg et al., 1993"},{"why":"BERT-Large serves as a fine-tuned transformer baseline for identification and classification.","marker":"Devlin et al., 2018"},{"why":"BioBERT-Large is evaluated as a domain-pretrained identification model, testing whether biomedical pretraining helps the sub-task.","marker":"Lee et al., 2019"},{"why":"Llama 3 Instruct is used as a prompted decoder-only baseline for identification and generation, including few-shot in-context learning.","marker":"Dubey et al., 2024"}],"fun_headline_variants":["Five-way breakdown of medical jargon in one dataset","JEBS: 21,595 replacements to demystify biomed text","Fine-grained medical simplification: five action types","Split medical jargon into five clear simplification moves","New dataset isolates five ways to simplify bio jargon"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the human simplifications in the source corpus can be reliably decomposed into the five replacement types, and that the annotations—which show only moderate inter-annotator agreement—are accurate enough to serve as gold-standard training and evaluation data.","fun_headline_variants_meta":{"raw":{"variants":["Five-way breakdown of medical jargon in one dataset","JEBS: 21,595 replacements to demystify biomed text","Fine-grained medical simplification: five action types","Split medical jargon into five clear simplification moves","New dataset isolates five ways to simplify bio jargon"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00094,"raw_usage":{"total_tokens":3967,"prompt_tokens":842,"completion_tokens":3125,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":458,"completion_tokens_details":{"reasoning_tokens":3060}},"tokens_in":458,"tokens_out":3125,"duration_ms":25029,"temperature":1.0,"reasoning_tokens":3060,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:35:49.533923+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check would be to take a fresh sample of biomedical abstracts, have independent annotators apply the JEBS annotation protocol, and compare their term spans and replacement-type labels to the published JEBS labels; if agreement falls near chance or if annotators systematically disagree on which replacement type applies, the gold standard's reliability is not established. Alternatively, one could test whether models trained on JEBS outperform end-to-end models on a held-out clinical reading-comprehension task; if they do not, the fine-grained decomposition may not translate into practical gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the PLABA parallel corpus of biomedical abstracts paired with expert-written simplifications, from which JEBS is derived."},{"cited_title":"Med-EASi: Finely Annotated Dataset and Models for Controllable Simplification of Medical Texts","cited_arxiv_id":"2302.09155","evidence_quote":"Med-EASi is the closest prior dataset with deletions, elaborations, and replacements, which JEBS extends with a larger size and a finer-grained taxonomy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Survey linking term-level simplification to faithfulness, supporting the paper's choice to focus on lexical operations rather than full rewrites."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MetaMapLite is the rule-based tool used in the identification baseline to find candidate expert terms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Unified Medical Language System (UMLS) provides the lexicon for the rule-based identification and substitution components."}],"review_version":1}