{"id":"1b99ac93-80d0-488a-8454-7addfba0fcfa","arxiv_id":"2502.03004","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Fine-tuning GPT-4o on biomedical QA datasets improves accuracy on MedQA, PubMedQA, and BioASQ, while RAG adds little once fine-tuning is applied.","lead":"MedBioLM is a GPT-4o model fine-tuned on biomedical QA datasets, reporting 88% on MedQA and 96% on BioASQ, plus improved long-form and short-form answer quality versus base GPT-4o. The paper tests whether domain-specific fine-tuning and retrieval augmentation help medical question answering, a question relevant to anyone building clinical AI tools.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No evaluation is shown to be independent of the training signal: Table 3 reports MedBioLM on a dataset described as 'curated for training' (§3.1), and the RAG index stores the answer field (§3.3), so retrieval can supply the gold answer.","rationale":"The reader's weakest assumption was that evaluation might be on training data because no train/test splits are stated; the paper text supports that directly. My read adds the closely related RAG circularity: the index described in §3.3 contains answer fields, so retrieval can hand the gold answer to the generator. Both are instances of the same missing independence condition, and both are load-bearing because the claimed fine-tuning gains and RAG effects depend on them. The reader also noted RAG stores gold answers, so there is substantial agreement. I do not see a separate objection that changes the verdict; the paper lacks the verification necessary to accept the specific numbers, and the self-admitted limitation in §5 (overfitting risk, inconsistent RAG impact) reinforces this. A REJECT verdict is appropriate, and my analysis does not alter it.","tokens_in":12573,"tokens_out":5322,"duration_ms":57349,"concrete_test":"Ask the authors for the exact evaluation subset used for every row in Tables 1, 3, and 4, then run one decisive check: take the short-form MedQA evaluation questions, build the RAG index exactly as described in §3.3 but with the gold answer field removed for those questions, and rerun the k=1..5 top-k sweep of Figure 4. If retrieval no longer returns the gold answer and the RAG metrics collapse to approximately the no-RAG row (ROUGE-1 ≈43, BLEU ≈12), the reported RAG benefit is an artifact of answer-key retrieval. Verdict remains unchanged.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that fine-tuning improves biomedical QA and that RAG contributes to factual consistency. Both halves require an evaluation setting that does not leak the answer. That independence condition is missing in the paper as written. Section 3.1 describes the Combined Custom Dataset as 'curated for training,' yet Table 3 reports MedBioLM's long-form ROUGE-1/BLEU/BERTScore on exactly that dataset, with no train/test split stated anywhere. Because the same dataset was used to fine-tune MedBioLM, the row can reflect memorization of training targets rather than generalization. The same absence of stated splits affects every row in Tables 1, 3, and 4: Table 5 lists only training samples and durations, never a held-out test set. Separately, §3.3 says the retrieval index stores an ID, a question field, and an answer field, with a keyword-based search strategy. For any evaluation question present in that index, retrieval can return the gold answer, so the RAG rows in Tables 3-4 and Figure 4 measure how well the generator copies from the answer key, not how well retrieval supplies external knowledge. The paper's own §5 acknowledges overfitting risk and inconsistent RAG impact. Either issue alone would prevent accepting the reported numbers; together they undercut both the fine-tuning and RAG conclusions. This is a verification gap, not a disagreement with the general possibility that fine-tuning helps; the needed evidence is simply absent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MedBioLM, a GPT-4o model fine-tuned on several biomedical QA datasets, optionally combined with a keyword-based retrieval-augmented generation (RAG) pipeline. The authors report accuracy gains on closed-ended benchmarks (MedQA 88.0%, PubMedQA 78.9%, BioASQ 96.0%) and improvements in ROUGE, BLEU, BERTScore, and BLEURT on long-form and short-form QA, and they conclude that fine-tuning is the dominant contributor while RAG provides inconsistent or marginal gains. The manuscript also includes training configurations, prompting strategies, and an analysis of top-k retrieval behavior.","tokens_in":12887,"tokens_out":4747,"duration_ms":39496,"significance":"If the reported numbers were valid, the paper would offer a useful practical recipe for adapting a general-purpose LLM to biomedical QA across multiple answer formats, and its negative results on RAG would be informative. The study addresses a relevant and active problem, and the authors appropriately acknowledge limitations such as single-physician evaluation and overfitting risk. However, the empirical evaluation as presented is not verifiable: the paper does not specify train/test splits, the RAG index stores gold answers, and several reported numbers contradict each other. These issues go beyond presentation and affect the central claims, so the current contribution cannot be assessed on the evidence provided.","major_comments":[{"comment":"The paper never states whether any evaluation was performed on held-out test splits. Table 5 enumerates only training samples and durations, and §3.1 describes the combined custom dataset as \"curated for training,\" yet Table 3 reports MedBioLM's long-form scores on exactly that dataset. Without stated test splits, the numbers in Tables 1, 3, and 4 cannot be distinguished from memorization of training targets, which invalidates the abstract's claim that fine-tuning \"significantly improves\" performance.","section":"§3.1, Tables 1, 3, 4, 5"},{"comment":"The retrieval index stores an answer field containing \"corresponding responses,\" and both question and answer fields are searchable. For any evaluation question present in the index, keyword retrieval can return the gold answer, and the generator can then copy it. Consequently, the MedBioLM+RAG rows in Table 4 and the top-k analysis in Figure 4 do not measure retrieval of external knowledge; they measure copying from the answer key. The manuscript must state that test queries and their gold answers were excluded from the index, and the methodology must actually implement that exclusion.","section":"§3.3, Table 4, Figure 4"},{"comment":"The reported numbers are internally inconsistent. Table 4 gives MedBioLM+RAG ROUGE-1=40.51 and BLEU=11.55, while §4.3 reports top-k=1 values of ROUGE-1=11.33 and BLEU=3.26 for the same short-form MedQA setup; either Figure 4 refers to a different configuration or one of the reported values is wrong. In addition, §4.3 states that \"the fine-tuned GPT-4o model achieves a BLEURT score of 8.63,\" whereas Table 4 lists MedBioLM BLEURT=33.43 and MedBioLM+RAG BLEURT=8.63. The text and the table cannot both be correct, making the central RAG comparison unverifiable.","section":"§4.3, Table 4, Figure 4"},{"comment":"No error bars, confidence intervals, significance tests, or multiple training runs are reported. For example, Table 1 shows MedBioLM at 88.0% versus GPT-4o at 87.0% on MedQA; a one-point difference with no variance or significance assessment does not support the claim of \"significant\" improvement. The same issue applies to the small ROUGE/BLEU differences in Tables 3 and 4, where the relative gains are on the order of a few points or less.","section":"§4, Tables 1, 3, 4"},{"comment":"The \"Combined Custom Dataset\" used to train and evaluate MedBioLM in Table 3 is not described in sufficient detail and is not released. The text says it \"consists of a combination of long-form QA data from LiveQA, MedQA, PubMedQA, and BioASQ,\" but Table 5 lists 6,652 samples without explaining how the constituent datasets were filtered, split, or balanced. Without a precise description and release, the Table 3 results are not reproducible, particularly because the same dataset is described as being \"curated for training.\"","section":"§3.1, §5, Table 5"}],"minor_comments":[{"comment":"Table 2 has broken column headers such as \"Y es (%)\" and row entries that run together (e.g., \"78.9 64 36 96\"); the table needs to be reformatted.","section":"Table 2"},{"comment":"The dataset name is inconsistently spelled as both \"BioASQ\" and \"BioSQA\" in the text and tables; please unify the terminology.","section":"Throughout"},{"comment":"The sentence beginning \"The short-form question evaluation results indicate...\" is repeated verbatim in consecutive paragraphs and should be removed.","section":"§4.3"},{"comment":"There is a typo: \"expolored\" should be \"explored.\"","section":"§3.4"},{"comment":"Figure 4's axes and configuration are not defined clearly; the text reports specific k=1 and k=5 values but does not say whether these correspond to the base model or the fine-tuned model, and the values do not match Table 4.","section":"Figure 4"},{"comment":"The pairwise evaluation by a single physician is acknowledged as subjective, but the number of questions evaluated and the selection criteria are not stated; please provide this information.","section":"§4.2"}],"recommendation":"reject","confidential_remarks":"The verification gap here is decisive: the evaluation protocol is not described in a way that rules out training-set contamination, and the RAG index as described leaks gold answers into retrieval. Combined with internal numerical contradictions, the reported results are not usable. I would not encourage a minor revision; if the authors were to resubmit with a properly held-out evaluation, a sanitized RAG index, corrected tables, and released or fully specified data, the work could be reconsidered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe gist: MedBioLM is a fine-tuned GPT-4o for biomedical QA, and the paper reports accuracy gains on MedQA, PubMedQA, and BioASQ, plus ROUGE/BLEU gains on long-form and short-form. The fine-tuning direction is plausible and matches prior work. But the evaluation as written has two load-bearing problems: the RAG index stores the answer field, and the custom dataset used for Table 3 is described as 'curated for training' with no holdout split. Those aren't minor edits—they undermine the RAG and the generalization claims.\n\nWhat's genuinely useful: the systematic comparison across three QA formats on a current model, with a battery of metrics. Table 4's contrast between fine-tuning and RAG on short-form QA is a nice datapoint: retrieval helps little after fine-tuning. The paper also acknowledges its own limitations in §5 (inconsistent RAG impact, overfitting risk, single-physician pairwise eval). That honesty counts for something.\n\nThe soft spots are serious. Section 3.3 says the index stores 'an answer field' and both question and answer fields are searchable. With keyword retrieval, any indexed question can return the gold answer, so the RAG rows in Tables 3 and 4 are not an external-knowledge test. That's circular. Second, no train/test splits are given for any dataset. Table 3 reports MedBioLM on a dataset the authors say was 'curated for training' (Section 3.1). Evaluating on training data would inflate the fine-tuning scores. No error bars or significance tests either, though the gaps on PubMedQA (78.9 vs 44.7) are large enough that the direction likely holds.\n\nI don't think the authors are being dishonest—the limitations section is candid—but the paper as written doesn't support its claimed numbers. A revision that releases splits, removes the answer field from the RAG index, and reports variance could make this a decent empirical note. As is, the central numbers aren't verifiable.\n\nWho's this for? Someone tracking fine-tuning vs RAG on medical QA might read it as a cautionary example. It's not a serious contribution to method, but it's not junk either. My recommendation: if the venue can handle revision, send it out but with a clear request for splits and a corrected retrieval setup. If it's for a one-shot conference, desk reject is defensible.","headline":"Plausible fine-tuning results, but the RAG evaluation is circular and the splits are missing, so the headline numbers aren't verifiable.","tokens_in":13388,"tokens_out":2970,"would_cite":false,"duration_ms":27833,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning GPT-4o on biomedical QA data yields MedBioLM, which outperforms the base model on MedQA, PubMedQA, and BioASQ.","keywords":["biomedical question answering","large language models","fine-tuning","retrieval-augmented generation","MedQA","PubMedQA","BioASQ","long-form QA"],"falsifier":"Run MedBioLM's exact evaluation on the official held-out test splits of MedQA, PubMedQA, and BioASQ, with training sets explicitly excluded, and check whether the accuracy margins over base GPT-4o (88.0 vs 87.0, 78.9 vs 44.74, 96.0 vs 92.0) and the ROUGE/BLEU gains still appear; the claim collapses if the margins vanish.","tokens_in":1713,"feed_emoji":"🩺","tokens_out":2984,"duration_ms":200107,"temperature":0.7,"pith_summary":"This paper claims that fine-tuning a general large language model, GPT-4o, on biomedical question-answering data yields a domain-specialized model, MedBioLM, that beats the base model on standard medical benchmarks. The reported gains are large: 88% accuracy on MedQA versus 87% for the base model, 78.9% versus 44.74% on PubMedQA, and 96% versus 92% on BioASQ. Fine-tuning also improves lexical and semantic similarity scores on long-form and short-form answers. The paper further argues that retrieval-augmented generation helps short-form QA modestly but adds little once fine-tuning is in place. A sympathetic reader would care because the result suggests that domain adaptation, not just scale, is what pushes LLMs toward reliable medical answers.","feed_headline":"MedBioLM hits 96% on BioASQ and 88% on MedQA after fine-tuning","feed_subtitle":"Domain-specific tuning lifts accuracy and answer quality over general GPT-4o across three biomedical QA formats.","key_machinery":"The load-bearing object is the fine-tuned model itself, MedBioLM, built by supervised fine-tuning of GPT-4o on task-specific datasets (MedQA, PubMedQA, BioASQ, MedicationQA, LiveQA, and a custom combined set) with prompt-specific decoding parameters. The fine-tuning pipeline is the mechanism: adaptive optimization with batch size set to 0.2% of training examples, a dynamic learning-rate scaling factor, and automatic seed assignment. RAG is implemented as a keyword-based search index with document chunking and structured field mapping, but the experiments show that fine-tuning, not retrieval, carries the accuracy gains. Prompt engineering (system messages, max tokens, temperature, top-p, penalties) is tuned per QA format, and evaluation uses accuracy for closed-ended tasks and ROUGE, BLEU, BERTScore, and BLEURT for open-ended ones.","core_discovery":"The central discovery is that supervised fine-tuning of GPT-4o on medical and biological QA datasets produces a model—MedBioLM—that substantially outperforms the original base model on closed-ended reasoning tasks (MedQA 88.0% vs 87.0%, PubMedQA 78.9% vs 44.74%, BioASQ 96.0% vs 92.0%) and improves long-form and short-form answer quality as measured by ROUGE, BLEU, BERTScore, and BLEURT. The paper also finds that RAG provides only marginal gains for the base model on short-form QA and no significant gain once fine-tuning is applied, and that increasing retrieval top-k beyond one hurts short-form answer quality. Fine-tuning reduces the model's tendency to answer 'Maybe' on PubMedQA, indicating improved confidence and reasoning. The authors present this as evidence that domain-specific adaptation is the primary driver of biomedical QA performance, with retrieval playing a secondary role.","pith_inferences":["If the reported gains hold on truly held-out splits, the implication is that fine-tuning GPT-4o is a cheap, practical route to specialized medical assistants without building a new model from scratch.","The negative top-k result hints that for short medical answers, precision-oriented retrieval with a single best document outperforms broad recall—a design principle that could generalize to other closed-domain QA settings.","The single-physician pairwise evaluation leaves open whether human readers would actually prefer MedBioLM's long answers; a multi-expert blind study would be the natural next test.","The paper does not report train/test splits, so a clean re-evaluation with explicitly disjoint data would separate genuine domain adaptation from memorization of benchmark questions."],"forward_implications":["Domain-specific fine-tuning of a strong general LLM can lift medical exam performance by 10–30 percentage points over the base model.","Retrieval-augmented generation built on keyword search gives only marginal short-form QA gains and can hurt performance as top-k grows.","Fine-tuned medical QA models answer decisively instead of hedging with 'Maybe', which matters for clinical decision support.","Combining multiple QA datasets during fine-tuning produces the best long-form scores, implying data diversity helps generalization."],"supporting_citations":[{"why":"Supplies the MedQA multiple-choice dataset from USMLE, the primary closed-ended benchmark.","marker":"Jin et al., 2020"},{"why":"Supplies PubMedQA, the yes/no/maybe reasoning benchmark where fine-tuning shows the largest gain.","marker":"Jin et al., 2019"},{"why":"Supplies BioASQ, the biomedical yes/no QA benchmark.","marker":"Nentidis et al., 2023"},{"why":"Supplies MedicationQA, the long-form medication question benchmark.","marker":"Abacha et al., 2019"},{"why":"Supplies LiveQA, the long-form consumer health question benchmark.","marker":"Abacha et al., 2017"},{"why":"Defines retrieval-augmented generation, the retrieval pipeline this paper adapts to keyword search.","marker":"Lewis et al., 2020"},{"why":"Grounds the instruction-following fine-tuning approach used to adapt GPT-4o.","marker":"Ouyang et al., 2022"},{"why":"Provides the OLAPH evaluation methodology with BERTScore and BLEURT and pairwise evaluation.","marker":"Jeong et al., 2024"},{"why":"Provides the Med-Gemini medical reasoning baseline and the pairwise evaluation framing.","marker":"Saab et al., 2024"},{"why":"Documents GPT-4o, the base model used for fine-tuning and comparison.","marker":"OpenAI, 2024a"}],"fun_headline_variants":["MedBioLM: GPT-4o fine-tuned to 96% BioASQ, 88% MedQA","Fine-tuning beats RAG for biomedical QA accuracy","SFT lifts BioASQ to 96%, MedQA to 88% from GPT-4o base","Domain tuning, not retrieval, powers MedBioLM QA gains"],"cache_read_input_tokens":15488,"weakest_assumption_plain":"The reported gains assume the benchmark questions were not seen during fine-tuning, yet no train/test split is stated anywhere in the paper.","fun_headline_variants_meta":{"raw":{"variants":["MedBioLM: GPT-4o fine-tuned to 96% BioASQ, 88% MedQA","Fine-tuning beats RAG for biomedical QA accuracy","SFT lifts BioASQ to 96%, MedQA to 88% from GPT-4o base","Domain tuning, not retrieval, powers MedBioLM QA gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001193,"raw_usage":{"total_tokens":4897,"prompt_tokens":899,"completion_tokens":3998,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":3908}},"tokens_in":515,"tokens_out":3998,"duration_ms":34148,"temperature":1.0,"reasoning_tokens":3908,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:14:57.431929+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run MedBioLM's exact evaluation on the official held-out test splits of MedQA, PubMedQA, and BioASQ, with training sets explicitly excluded, and check whether the accuracy margins over base GPT-4o (88.0 vs 87.0, 78.9 vs 44.74, 96.0 vs 92.0) and the ROUGE/BLEU gains still appear; the claim collapses if the margins vanish.","supporting_citations":[{"cited_title":"B., Agichtein, E., Pinter, Y., and Demner-Fushman, D","cited_arxiv_id":null,"evidence_quote":"Supplies LiveQA, the long-form consumer health question benchmark."},{"cited_title":"Retrieval-augmented generation for knowledge-intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Defines retrieval-augmented generation, the retrieval pipeline this paper adapts to keyword search."}],"review_version":1}