{"id":"46017837-bef1-424c-a9cc-1141c4281e4b","arxiv_id":"2501.15700","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"A one-shot GPT-4 prompt driven by distilled PLABA annotation guidelines ranked first on simplicity and third on accuracy in the plain-language biomedical abstract adaptation task.","lead":"This paper tested several large language models, including GPT-4, for rewriting medical abstracts into simpler language for the general public. Its best GPT-4 system ranked first among seven teams on simplicity and third on accuracy in the 2023 PLABA competition.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 1st-place simplicity ranking may be within annotation noise; the paper gives no confidence intervals or inter-annotator agreement, so the 2.42-point lead over the median is not established as robust.","rationale":"After reading the paper in good faith, the central claim is a report of an official shared-task ranking. The most load-bearing condition is that the official human evaluation scores are precise enough to make the 1st-versus-others distinction meaningful. The paper's own evidence is a single row of averages in Table 4 with no uncertainty. The sample size (430 sentences for simplicity, 117 for accuracy) and the coarse rating scale (-1/0/1 transformed to 1-100) make random noise a plausible alternative explanation for the observed 2.42-point gap over the median. The reader's weakest_assumption identified exactly this: the external evaluation is assumed valid and comparable, but no inter-annotator agreement or confidence intervals are reported. I found no internal mathematical inconsistency; the reported numbers are internally coherent. The other plausible concern—that the exact prompt is not released—affects reproducibility but not the truth of the ranking. Therefore the single decisive check is to obtain the per-sentence scores and compute confidence intervals. If the CIs overlap, the claim should be treated as an unverified observation rather than a robust result; if they do not overlap, the concern is resolved. The appropriate verdict remains CONDITIONAL, matching the reader, because the paper should either supply the uncertainty analysis or soften the generalizing conclusions.","tokens_in":7564,"tokens_out":7535,"duration_ms":65037,"concrete_test":"Request from the PLABA 2023 organizers (or reconstruct from released evaluation data) the per-sentence simplicity scores for all seven top-ranked systems. Compute bootstrap 95% confidence intervals for each system's mean simplicity and a paired bootstrap CI for the difference between the top-ranked and second-ranked systems. If the difference CI includes zero, the 'ranked first' claim is not statistically supported; if it excludes zero, the concern is settled. Also compute inter-annotator agreement (e.g., Krippendorff's alpha) if double-annotated items exist.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the GPT-4 one-shot system 'ranked first in the avg. simplicity measure' (abstract; §4.2, Table 4) depends on the external human evaluation being sufficiently precise to distinguish the top system from the others. The paper reports only the average simplicity score (91.28), the top score (91.28, a tie), and the median (88.86) across 7 teams, with no standard deviations, confidence intervals, per-sentence scores, or inter-annotator agreement. The evaluation was on a small sample: 40 abstracts, 430 sentences for simplicity, and the scores were transformed from -1/0/1 ratings. With a gap of only 2.42 points to the median, and presumably smaller gaps to the second-place system, the rank could easily flip under random annotation noise. The conclusions 'LLMs hold a lot of promise' and 'the best-performing system was the annotation-guideline-instructed GPT-4...' overstate what the data can support without uncertainty quantification. This is not an internal inconsistency, but it is a load-bearing weakness: if the official scores are noisy, the headline result is not a reproducible or generalizable finding.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper describes the authors' submissions to the TAC-2023 PLABA shared task on adapting biomedical abstracts into plain language. The authors fine-tune T5-Large, LLaMa2 (13B and 70B chat), and GPT-3.5, and also use in-context learning with GPT-4, comparing systems on automatic metrics and a small internal human evaluation. They report that their GPT-4-based one-shot in-context learning system, prompted with a distilled version of the PLABA annotation guideline and one training example, ranked first in the official external human evaluation on the average simplicity axis and third on the average accuracy axis.","tokens_in":7816,"tokens_out":8167,"duration_ms":69231,"significance":"If the reported ranking is robust, the paper provides a practically relevant finding: a simple one-shot GPT-4 prompt with annotation guidelines can outperform more expensive fine-tuning approaches on plain-language adaptation of biomedical text. The use of the official external human evaluation is a strength, as is the observation that high BLEU/ROUGE scores can reflect superficial repetition rather than genuine simplification. However, the paper does not provide uncertainty quantification for the external evaluation, and the exact prompt and example selection are not specified, which limits the strength of the headline conclusions. The work is a solid shared-task system description but needs additional evidence to support its central claims at the journal level.","major_comments":[{"comment":"The claim of ranking first in the average simplicity measure rests on an external human evaluation of 430 sentences, but the paper reports no inter-annotator agreement, confidence intervals, or per-item score distributions. The reported average of 91.28 is only 2.42 points above the median of 88.86 on a scale derived from ternary (-1/0/1) ratings; roughly 20 sentence-level one-step rating changes could account for this gap, so the first-place ranking is not established as robust. Please add uncertainty quantification (e.g., bootstrap confidence intervals if per-sentence scores are available) or explicitly state that the observed difference may be within annotation noise.","section":"§4.2, Table 4"},{"comment":"The internal human evaluation used to select the GPT-4 submission is the only evidence that GPT-4 outperformed the fine-tuned systems in human judgment, but it is based on approximately 100 sentences per model, evaluated by the lead authors with no inter-annotator agreement, no detailed rating criteria, and only a preference ranking. This is a small and potentially biased sample; the paper should either provide a more rigorous evaluation protocol or temper the claim that GPT-4 was the best-performing of the authors' own systems.","section":"§3.4.1"},{"comment":"The top-performing GPT-4 system uses a 'distilled version of the PLABA annotation guideline' with one training example, but the full prompt and the method for selecting that example are not provided. Since the paper's central conclusion attributes the result to this specific annotation-guideline-instructed one-shot prompt, the omission prevents replication and independent verification. Please include the exact prompt and describe the example-selection procedure.","section":"§3.2, Figure 2"}],"minor_comments":[{"comment":"The abstract says the authors 'leveraged the best open-source Large Language Models,' but GPT-4 is proprietary; this should be corrected to avoid a factual error.","section":"Abstract"},{"comment":"The column headers in Table 3 are garbled, so the reader cannot tell which system corresponds to each score or which score is the rank-1 submission in the external SARI comparison; please use a clearly labeled table.","section":"§4.1, Table 3"},{"comment":"The transformation from the -1/0/1 rating scale to the 1-100 scale is not specified; please clarify the mapping.","section":"§3.4.2"},{"comment":"The observation that T5 and LLaMa2 'tend to frequently repeat the abstract sentence as is' is qualitative; including example outputs or a quantitative measure (e.g., percentage of unchanged sentences) would strengthen the point.","section":"§4.1"},{"comment":"Calling the systems 'state-of-the-art' is an overclaim given the third-place accuracy ranking; consider using 'competitive' or 'well-performing' instead.","section":"§5"},{"comment":"There are several typos and formatting issues (e.g., 'Simplicty axis' in Table 4, the garbled author name 'Y etisgen', and 'V olume' in the reference list); these should be fixed in a revision.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a good shared-task system description with a valuable observation about automatic metrics. The main concerns are the lack of uncertainty quantification for the headline ranking and the missing prompt details; both are fixable within the manuscript's scope. I would not reject, but the current version overstates the robustness of the first-place claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"What you should know first: this is a TAC-2023 PLABA system description, not a research breakthrough. The genuinely useful piece is the GPT-4 one-shot prompt built from a distilled version of the annotation guidelines plus one hand-picked training example. That simple recipe beat six other teams on the official external human evaluation for average simplicity (91.28 vs. 88.86 median, tied with the top score) and came in 3rd on accuracy. If you work on plain-language adaptation or health literacy, that prompt recipe is worth stealing.\n\nThe paper does some things well. It compares a range of models fairly: fine-tuned T5, LLaMa-2 (LoRA), GPT-3.5, and GPT-4 ICL, and it reports both automatic and human evaluations. The observation that T5 and LLaMa-2 inflate BLEU/ROUGE by copying sentences rather than simplifying is a useful caution for the field. The choice of GPT-4 as the top submission was made on internal human judgment before external scores came out, so there is no circularity in the headline ranking. That is clean.\n\nSoft spots, in proportion. The load-bearing simplicity ranking rests on an external human evaluation with no confidence intervals, no inter-annotator agreement, and no per-sentence score distributions. The gap to the median is 2.42 points on a 1-100 scale; the gap to the second-place system is likely smaller. With 430 sentences scored by an unknown number of annotators and a -1/0/1 scale transformed to 1-100, that rank could plausibly flip under noise. The paper should report IAA or at least acknowledge the uncertainty. The internal human evaluation used to select GPT-4 was only about 100 sentences per model with no IAA either, and the prose calls GPT-4's internal performance \"remarkably higher\" without numbers backing that drama. Also, the conclusion calls these \"state-of-the-art\" systems, which is unsupported; they are competitive systems in one shared task. No code or exact prompt is released, so the main artifact is only partially reproducible.\n\nThe stress-test note lands: the rank is not established as robust. But that is a limitation rather than a fatal flaw, because the paper's primary job is to report official shared-task results, and it does so transparently.\n\nWho this is for: researchers building plain-language adaptation systems, especially those curious how far a well-engineered GPT-4 prompt can go. It deserves a serious referee for a shared-task venue or workshop, with requests for uncertainty quantification and artifact release.","headline":"A useful shared-task system report: a distilled one-shot GPT-4 prompt wins PLABA simplicity, but the win is not shown to be statistically robust.","tokens_in":8359,"tokens_out":1262,"would_cite":true,"duration_ms":13600,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single GPT-4 prompt with distilled guidelines ranked first for simplicity in a shared task adapting medical abstracts to plain language.","keywords":["plain language adaptation","biomedical abstracts","large language models","in-context learning","GPT-4","text simplification","health literacy","human evaluation"],"falsifier":"Re-score the top system alongside its closest competitors with a new set of expert annotators, or bootstrap confidence intervals from the per-sentence evaluation scores; if the simplicity ranking flips or the intervals overlap substantially, the claim of ranking first is not supported.","tokens_in":7404,"feed_emoji":"🩺","tokens_out":4666,"duration_ms":38788,"temperature":0.7,"pith_summary":"Health information in biomedical abstracts is too technical for many readers, and this paper asks whether large language models can bridge that gap automatically. The authors built several systems for a shared plain-language adaptation task and found that their best system was not a heavily fine-tuned one but a GPT-4 model prompted with a distilled version of the adaptation guidelines plus one worked example. In the external human evaluation, this one-shot prompt tied for the top average simplicity score and placed third on average accuracy among seven participating teams. The result matters because it suggests that carefully encoding expert guidelines into a prompt can achieve strong simplification without expensive fine-tuning.","feed_headline":"A single prompt wins plain-language medical abstract task","feed_subtitle":"Distilled guidelines plus one example beat seven teams on simplicity while staying third in accuracy.","key_machinery":"The load-bearing mechanism is the one-shot in-context learning prompt given to GPT-4: it contains a distilled version of the official annotation guidelines for plain-language adaptation plus one example sentence pair chosen to illustrate most of those rules. The guidelines tell the model to split long sentences, expand abbreviations, explain expert terms in parentheses or clauses at first mention, omit statistical details such as p-values, and drop or retain sentences as needed. The single example demonstrates how those rules apply, letting the model infer the transformation style without weight updates. The paper contrasts this with fine-tuned open models (T5, LLaMa-2, GPT-3.5), which scored better on automatic metrics but often repeated abstract sentences with minimal simplification.","core_discovery":"The paper's central claim is that an 'annotation-guideline-instructed GPT-4-based one-shot in-context learning model' produces plain-language adaptations that human evaluators judged simpler than those of the other six teams, while remaining reasonably faithful and complete. In the task's external evaluation, this system scored 91.28 on the average simplicity axis, matching the top score among all rank-one submissions and exceeding the 88.86 median; on the average accuracy axis it scored 92.52, ranking third. The authors attribute the result to the prompt, which carried a condensed version of the annotation guidelines and a single training example, rather than to any fine-tuning of the model.","pith_inferences":["The reported margins are small: a 2.42-point simplicity gap over the median and ties at the top, with no inter-annotator agreement or confidence intervals reported, so the ranking may not replicate under a different panel of annotators.","The prompt is tuned to a proprietary model whose behavior can change across versions; testing whether open-weight models with the same guidelines-plus-example prompt reproduce the result would say whether the method generalizes.","The 'distilled guidelines plus one example' pattern is a general template for any rubric-driven rewriting task; one could test it on other domains such as legal, technical, or financial text."],"forward_implications":["A guideline-distilled one-shot prompt can outperform fine-tuned open models on human-judged simplicity for plain-language adaptation.","Automatic metrics such as BLEU, ROUGE, and SARI may reward conservative repetition, so system ranking should include human evaluation.","The same prompting recipe could be applied to other consumer health texts, such as clinical trial summaries or discharge instructions.","Because the model needs no task-specific fine-tuning, the approach is cheap to deploy and update as new biomedical abstracts appear."],"supporting_citations":[{"why":"Defines the shared task and the external test setting whose human evaluation produced the ranking.","marker":"[1]"},{"why":"Supplies the PLABA dataset of abstracts and manual plain-language adaptations used for training and evaluation.","marker":"[2]"},{"why":"Establishes few-shot in-context learning, the mechanism the top GPT-4 system relies on.","marker":"[32]"},{"why":"Provides the GPT-4 model used for the top-performing system.","marker":"[33]"},{"why":"Gives the official annotation guidelines that the authors distilled into the prompt, the core of the method.","marker":"[34]"},{"why":"Defines the simplicity and accuracy evaluation axes and the scoring that produced the rank.","marker":"[38]"}],"fun_headline_variants":["One-shot GPT-4 prompt tops medical abstract simplicity","Guideline-packed prompt beats fine-tuning for plain-language abstracts","No fine-tuning: GPT-4 prompt ranks first in simplicity","Single prompt yields simplest medical abstracts"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking rests on the assumption that the official external human evaluation is a valid, unbiased measure of adaptation quality, and that small differences like 91.28 versus 88.86 on the simplicity average are real rather than annotator noise.","fun_headline_variants_meta":{"raw":{"variants":["One-shot GPT-4 prompt tops medical abstract simplicity","Guideline-packed prompt beats fine-tuning for plain-language abstracts","No fine-tuning: GPT-4 prompt ranks first in simplicity","Single prompt yields simplest medical abstracts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002032,"raw_usage":{"total_tokens":7882,"prompt_tokens":874,"completion_tokens":7008,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":490,"completion_tokens_details":{"reasoning_tokens":6946}},"tokens_in":490,"tokens_out":7008,"duration_ms":46212,"temperature":1.0,"reasoning_tokens":6946,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:01:50.308815+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-score the top system alongside its closest competitors with a new set of expert annotators, or bootstrap confidence intervals from the per-sentence evaluation scores; if the simplicity ranking flips or the intervals overlap substantially, the claim of ranking first is not supported.","supporting_citations":[{"cited_title":"Plain Language Adaptation of Biomedical Abstracts - A track of TAc 2023; 2023","cited_arxiv_id":null,"evidence_quote":"Defines the shared task and the external test setting whose human evaluation produced the ranking."},{"cited_title":"A Dataset for Plain Language Adaptation of Biomedical Abstracts","cited_arxiv_id":null,"evidence_quote":"Supplies the PLABA dataset of abstracts and manual plain-language adaptations used for training and evaluation."},{"cited_title":"Language Models are Few-Shot Learners","cited_arxiv_id":null,"evidence_quote":"Establishes few-shot in-context learning, the mechanism the top GPT-4 system relies on."},{"cited_title":"PLABA data annotation guidelines; 2023","cited_arxiv_id":null,"evidence_quote":"Gives the official annotation guidelines that the authors distilled into the prompt, the core of the method."},{"cited_title":"PLABA evaluation notes; 2023","cited_arxiv_id":null,"evidence_quote":"Defines the simplicity and accuracy evaluation axes and the scoring that produced the rank."}],"review_version":1}