{"id":"5a875315-4945-4770-bb01-7cdf68927038","arxiv_id":"2501.17190","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuned BERT Large Uncased reaches 100% accuracy on a small, template-heavy Healthline-derived medical question classification dataset, while LoRA-tuned RoBERTa-large reaches only 78%.","lead":"This paper fine-tunes four BERT and RoBERTa variants to classify medical questions into predefined disease-related labels, then returns a fixed answer per label. The authors report near-perfect accuracy on their Healthline-based dataset, while the smaller LoRA model performs clearly worse.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 100% claim is undermined by likely cross-fold leakage from templated, non-deduplicated Healthline questions, and the paper itself flags possible overfitting.","rationale":"The reader's weakest assumption—that the Healthline-derived dataset permits a fair evaluation with independent cross-validation folds free of near-duplicate templated questions—is exactly the load-bearing vulnerability I identify. The claim of 100% metrics is only meaningful if the validation folds are independent samples of real medical questions; the paper provides no evidence of that, and Table 1 shows the opposite. A single controlled retraining experiment with grouped or deduplicated folds would settle whether the perfect scores survive a leakage-free evaluation. The self-flagged caution in Section 6 and the arithmetic inconsistency in the 70/30 split independently corroborate the concern. I therefore see no reason to adjust the reader's reject verdict, though the strength of the rejection would increase if the grouped-fold experiment confirms the leakage.","tokens_in":12716,"tokens_out":3375,"duration_ms":32295,"concrete_test":"Download the HealifyAI dataset referenced as [5]; normalize each question by stripping template phrases such as 'What is', 'Tell me about', 'Can you elaborate on', and group items by canonical disease/label; then run the same 5-fold cross-validation with GroupKFold so all paraphrases of one question stay in the same fold, and retrain BERT Large Uncased with the reported settings. If its accuracy drops below 100%—or drops materially from 99.87% for RoBERTa-base—the published scores are leakage artifacts rather than evidence of generalization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that BERT Large Uncased achieves 100% accuracy, precision, recall, and F1 on medical question classification—depends entirely on the evaluation being a clean test of generalization. That condition is not met. Section 4 describes a 6,800-sample dataset scraped from Healthline and augmented with programmatically generated synthetic samples; Table 1 shows near-identical template paraphrases ('What is diabetes?', 'Tell me about diabetes?', 'What kind of disease is diabetes?', 'Can you elaborate on diabetes?') all mapped to the same label. No deduplication or grouping step is reported before 5-fold cross-validation. Random folds can therefore place paraphrases of the same original question in both training and validation splits, allowing a high-capacity model like BERT Large to memorize template-to-label mappings and appear perfect on validation. The paper itself acknowledges in Section 6 that the perfect scores 'could indicate potential overfitting, especially if the evaluation dataset lacks diversity.' The split arithmetic is also inconsistent: 70% of 6,800 is 4,760 and 30% is 2,040, but the text reports 1,020 validation samples, further undermining confidence that the reported metrics come from a valid held-out evaluation. If leakage is present, the 100% numbers say nothing about which model is preferred for real medical QA.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper fine-tunes four models (LoRA Roberta-large, Roberta-base, Bert Uncased, and Bert Large Uncased) on a 6,800-sample Healthline-derived medical question classification dataset and reports that Bert Large Uncased achieves 100% accuracy, precision, recall, and F1, concluding that it is the preferred model for high-accuracy medical QA. The evaluation uses 5-fold cross-validation with accuracy, precision, recall, and F1 metrics, and also compares training times.","tokens_in":12990,"tokens_out":2933,"duration_ms":26869,"significance":"If the reported performance numbers were trustworthy, the paper would provide a useful comparison of standard classification models for medical question classification. The use of a publicly available dataset and standard evaluation metrics is a modest strength, as is the authors' explicit acknowledgment in Section 6 that the perfect BERT Large score may reflect overfitting. However, the central empirical claim is not supported because of likely data leakage from templated, non-deduplicated questions, an inconsistent train/validation split arithmetic, and missing hyperparameter details. The contribution is therefore not currently established.","major_comments":[{"comment":"The dataset is composed largely of templated paraphrases (e.g., 'What is diabetes?', 'Tell me about diabetes?', 'What kind of disease is diabetes?', 'Can you elaborate on diabetes?'), and no deduplication or grouped splitting is reported before 5-fold cross-validation. Random folds can place near-duplicate questions in both the training and validation splits, allowing a high-capacity model like BERT Large to memorize template-to-label mappings. The paper itself cautions in Section 6 that the perfect scores 'could indicate potential overfitting, especially if the evaluation dataset lacks diversity.' The 100% result is therefore likely an artifact of leakage rather than genuine generalization.","section":"Section 4, Table 1"},{"comment":"The split arithmetic is internally inconsistent: the text states the dataset has 6,800 samples, with 70% (4,760) for training and the remaining 30% (1,020) for validation. Thirty percent of 6,800 is 2,040, not 1,020, and 4,760 + 1,020 = 5,780, which is less than 6,800. This discrepancy invalidates the reported evaluation setup and makes it impossible to know how many samples actually appeared in the validation folds.","section":"Section 4"},{"comment":"The paper claims that hyperparameters such as learning rate, batch size, and optimizer were 'carefully examined and tuned' but does not report any concrete values. No learning rate, batch size, optimizer, LoRA rank/alpha, or number of training epochs is specified. Without these details the experiments are not reproducible, and the comparative conclusions cannot be independently verified or interpreted.","section":"Section 5, Experimental Setup"},{"comment":"The LoRA Roberta-large model achieves only 78.47% accuracy while the non-LoRA Roberta-base achieves 99.87%, yet the manuscript provides no analysis of this large gap. Given that Section 3 emphasizes LoRA as an 'Innovative Use' intended to maintain accuracy while reducing computational cost, the unexplained underperformance undermines a key part of the paper's claimed contribution.","section":"Section 6, LoRA Roberta-large results"}],"minor_comments":[{"comment":"The phrase 'data cleaning to remove unnecessary or redundant information' is stated but no concrete cleaning or deduplication steps are described, which is directly relevant to the leakage concern.","section":"Section 4"},{"comment":"The text states that 'all the models eventually reach high accuracy,' but the LoRA Roberta-large model peaks around 78%, so this claim is inaccurate and should be rephrased.","section":"Section 6"},{"comment":"The sentence 'All the 5 fold cross validation on the two models were done on 10 epochs each' appears to refer to 'two models' but four models are used; this is likely a typo and should be corrected.","section":"Section 3"},{"comment":"There are numerous typographical and grammatical errors (e.g., 'developtment', 'IIn this study', 'accessing performance'), which should be corrected for clarity.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The paper's central claim of 100% accuracy for BERT Large Uncased is not credible given the highly templated, non-deduplicated dataset and the authors' own admission of possible overfitting. The split arithmetic error further indicates that the evaluation was not carefully conducted. The study is also not novel in methodology and lacks the experimental detail needed for reproducibility. I do not see a path to a sound paper without essentially redoing the experiments with a properly controlled evaluation, which is beyond a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a standard fine-tuning comparison of BERT/RoBERTa/LoRA on a small, template-heavy Healthline-derived QA dataset. The headline claim—BERT Large Uncased at 100% accuracy—should not be believed, and the paper does not supply enough for a reader to check it. This is not a takedown; the authors are transparent about the dataset's construction and explicitly warn that perfect scores may reflect overfitting. That honesty is the best part of the paper.\n\nWhat it does well: it reports 5-fold cross-validation metrics for four models, plots per-fold behavior, and includes a related-work table that is broadly relevant. The dataset description in Table 1 is actually useful because it shows exactly why the task is easy: the questions are paraphrases of the same disease queries, all mapped to the same label. The authors also note the predefined-answer limitation and call for more diverse data and interpretability work. Those are the right caveats.\n\nThe soft spots are substantial. First, the 100% number is not a clean generalization result. With near-duplicate templated questions and no deduplication or grouping before 5-fold CV, random splits put paraphrases in both training and validation; a high-capacity model can memorize the template-to-label mapping. The authors' own overfitting caveat in Section 6 is the right worry, but it stops short of acknowledging the leakage mechanism. Second, the split arithmetic is wrong: 30% of 6,800 is 2,040, not 1,020. That kind of inconsistency makes it hard to trust the reported evaluation setup. Third, hyperparameters (learning rate, batch size, optimizer, LoRA rank/alpha) are not reported, so the comparison cannot be reproduced. The LoRA model's much lower score is left unexplained—it could be a bad configuration rather than a property of LoRA. Finally, the paper introduces no new method, theory, or benchmark; it is a parameter scan of existing models on an existing GitHub dataset.\n\nWho is this for? Someone collecting examples of how not to evaluate template-derived QA datasets might cite it as a cautionary case. A reader looking for reliable evidence about which fine-tuned model to use for real medical QA will not find it here. I would not send this to peer review in its current form; the evaluation flaws are load-bearing, not cosmetic. If the authors fixed the leakage, reported hyperparameters, and reframed the result as a negative lesson about template datasets, it could become a modest preprint, but not a paper that needs referee time.\n\nRecommendation: desk reject.","headline":"Routine fine-tuning scan whose headline 100% accuracy is undermined by templated-data leakage and an arithmetic slip; the authors' own overfitting caveat is the most honest part.","tokens_in":13520,"tokens_out":2863,"would_cite":false,"duration_ms":26064,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that fine-tuning BERT Large Uncased on a curated medical QA dataset yields perfect classification scores of 100% across accuracy, precision, recall, and F1.","keywords":["medical question answering","BERT","RoBERTa","LoRA","Healthline dataset","text classification","fine-tuning","5-fold cross-validation"],"falsifier":"Re-run the same 5-fold cross-validation after removing all near-duplicate or templated questions so that no paraphrase appears in more than one fold, or evaluate on a held-out set of real patient questions from another source; if BERT Large Uncased scores drop below 100%, the perfect-performance claim fails.","tokens_in":12558,"feed_emoji":"🩺","tokens_out":4703,"duration_ms":37752,"temperature":0.7,"pith_summary":"This paper builds a two-stage medical question answering system: first classify the question into a predefined label, then return a stored answer for that label. The authors fine-tune four models — LoRA-tuned RoBERTa-large, RoBERTa-base, BERT Uncased, and BERT Large Uncased — on a 6,800-sample dataset scraped from Healthline, and report that BERT Large Uncased reaches 100% accuracy, precision, recall, and F1 under 5-fold cross-validation. RoBERTa-base also scores near perfect at 99.87%, while LoRA RoBERTa-large trails at 78.47%. The paper itself cautions that the perfect scores may reflect overfitting on a dataset with limited diversity. The stated takeaway is that BERT Large Uncased is preferred for high-accuracy medical QA, with RoBERTa-base as the efficient alternative.","feed_headline":"BERT Large scores perfect 100% on medical QA test","feed_subtitle":"Two-stage classify-then-answer pipeline tested on 6,800 Healthline questions; RoBERTa-base hits 99.87%.","key_machinery":"The machinery is a two-stage classify-then-retrieve pipeline. A transformer encoder fine-tuned for sequence classification (BERT or RoBERTa) maps each medical question to one of a fixed set of disease-related labels, and a lookup table then returns the predefined answer associated with that label. The four variants compared are full fine-tunes of BERT Large Uncased, BERT Uncased, and RoBERTa-base, plus a Low-Rank Adaptation (LoRA) tuned RoBERTa-large; all are trained for 10 epochs per fold under 5-fold cross-validation.","core_discovery":"The central discovery, as the authors state it, is that a classification-based pipeline using BERT Large Uncased achieves perfect scores — accuracy, precision, recall, and F1 of 100% — on the task of labeling medical questions drawn from Healthline. The system does not generate answers; it selects from predefined answers by first predicting the question's label. The authors describe RoBERTa-base as demonstrating the most balanced and consistent performance across folds, and they explicitly flag the perfect BERT Large Uncased result as warranting cautious interpretation because it could indicate overfitting.","pith_inferences":["The perfect scores most plausibly reflect near-duplicate templated questions shared between training and validation folds; the paper reports no deduplication, and Table 1 shows many paraphrases of 'What is diabetes?'.","A deduplicated or out-of-distribution evaluation would likely lower all reported numbers, and the ordering of models could change once the easiest pattern-matching signal is removed.","A testable extension is to freeze the label set but generate answers dynamically with a generative model, turning the strong classifier into a real open-ended medical QA system.","The large gap between LoRA RoBERTa-large and the other models may be a tuning artifact (rank, epochs, learning rate) rather than a property of LoRA itself; a similar LoRA configuration for BERT Large would isolate that variable."],"forward_implications":["If the reported scores hold, a fine-tuned classifier can serve medical answers with near-perfect fidelity to a curated answer bank, with BERT Large Uncased as the top performer.","RoBERTa-base offers a nearly identical result (99.87% accuracy) at lower training cost, making it the pragmatic choice when compute is limited.","The LoRA-tuned RoBERTa-large performs markedly worse (78.47%), suggesting that in this low-resource setting low-rank adaptation trades away accuracy for efficiency.","Because answers are predefined, the approach can only handle questions that map onto the existing label set; novel or complex queries will fall outside its coverage.","The paper's own caution implies that the 100% figure should be re-tested on more diverse data before clinical deployment."],"supporting_citations":[{"why":"The HealifyAI GitHub repository is the source of the 6,800 Healthline-derived questions and the secondary answer dataset used for training and evaluation.","marker":"[5]"},{"why":"Hu et al.'s LoRA method is the low-rank adaptation technique applied to the RoBERTa-large model, explaining its reduced accuracy in the comparison.","marker":"[4]"},{"why":"Liu et al.'s RoBERTa pretraining work supplies the base model for the RoBERTa-base and LoRA RoBERTa-large variants compared in the study.","marker":"[11]"},{"why":"Lee et al.'s BioBERT establishes the transformer-based medical question classification benchmark this study positions itself against.","marker":"[14]"}],"fun_headline_variants":["BERT Large perfect 100% on medical QA, but overfit risk","Two-stage classify-then-answer: BERT Large scores perfect","Medical QA: BERT Large perfect, authors flag overfitting","BERT Large aces medical QA, but authors warn of overfit","Perfect medical QA classification by BERT Large, overfit noted"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Healthline-derived dataset, with its 5-fold splits, fairly represents the range of real medical questions; if near-duplicate templated questions appear in both training and validation, the reported accuracy measures memorization rather than generalization.","fun_headline_variants_meta":{"raw":{"variants":["BERT Large perfect 100% on medical QA, but overfit risk","Two-stage classify-then-answer: BERT Large scores perfect","Medical QA: BERT Large perfect, authors flag overfitting","BERT Large aces medical QA, but authors warn of overfit","Perfect medical QA classification by BERT Large, overfit noted"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000595,"raw_usage":{"total_tokens":2809,"prompt_tokens":994,"completion_tokens":1815,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":1725}},"tokens_in":610,"tokens_out":1815,"duration_ms":10841,"temperature":1.0,"reasoning_tokens":1725,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:57:27.310640+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same 5-fold cross-validation after removing all near-duplicate or templated questions so that no paraphrase appears in more than one fold, or evaluate on a held-out set of real patient questions from another source; if BERT Large Uncased scores drop below 100%, the perfect-performance claim fails.","supporting_citations":[{"cited_title":"HealifyAI - LLM based Healthcare System","cited_arxiv_id":null,"evidence_quote":"The HealifyAI GitHub repository is the source of the 6,800 Healthline-derived questions and the secondary answer dataset used for training and evaluation."},{"cited_title":"BioBERT: a pre-trained biomedical language representation model for biomedical text mining","cited_arxiv_id":null,"evidence_quote":"Lee et al.'s BioBERT establishes the transformer-based medical question classification benchmark this study positions itself against."}],"review_version":1}