Pith. sign in

REVIEW 4 major objections 4 minor 2 cited by

A Comprehensive Study on Fine-Tuning Large Language Models for Medical Question Answering Using Classification Models and Comparative Analysis

T0 review · 4 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that fine-tuning BERT Large Uncased on a curated medical QA dataset yields perfect classification scores of 100% across accuracy, precision, recall, and F1.

desk verdict Routine fine-tuning scan whose headline 100% accuracy is undermined by templated-data leakage and an arithmetic slip; the authors' own overfitting caveat is the most honest part. read the letter →

arxiv 2501.17190 v1 pith:YXVXCNYJ submitted 2025-01-27 cs.CL

Aysegul Ucar , Soumik Nayak , Anunak Roy , Burak Taşc{i} , Gülay Taşc{i} This is my paper · ORCID
classification cs.CL
keywords medicalquestionansweringBERTRoBERTaLoRAHealthlinedatasettextclassificationfine-tuning5-foldcross-validation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds a two-stage medical question answering system: first classify the question into a predefined label, then return a stored answer for that label. The authors fine-tune four models — LoRA-tuned RoBERTa-large, RoBERTa-base, BERT Uncased, and BERT Large Uncased — on a 6,800-sample dataset scraped from Healthline, and report that BERT Large Uncased reaches 100% accuracy, precision, recall, and F1 under 5-fold cross-validation. RoBERTa-base also scores near perfect at 99.87%, while LoRA RoBERTa-large trails at 78.47%. The paper itself cautions that the perfect scores may reflect overfitting on a dataset with limited diversity. The stated takeaway is that BERT Large Uncased is preferred for high-accuracy medical QA, with RoBERTa-base as the efficient alternative.

What carries the argument

The machinery is a two-stage classify-then-retrieve pipeline. A transformer encoder fine-tuned for sequence classification (BERT or RoBERTa) maps each medical question to one of a fixed set of disease-related labels, and a lookup table then returns the predefined answer associated with that label. The four variants compared are full fine-tunes of BERT Large Uncased, BERT Uncased, and RoBERTa-base, plus a Low-Rank Adaptation (LoRA) tuned RoBERTa-large; all are trained for 10 epochs per fold under 5-fold cross-validation.

What would settle it

Re-run the same 5-fold cross-validation after removing all near-duplicate or templated questions so that no paraphrase appears in more than one fold, or evaluate on a held-out set of real patient questions from another source; if BERT Large Uncased scores drop below 100%, the perfect-performance claim fails.

Watch

Extended reading notes

Core claim

The central discovery, as the authors state it, is that a classification-based pipeline using BERT Large Uncased achieves perfect scores — accuracy, precision, recall, and F1 of 100% — on the task of labeling medical questions drawn from Healthline. The system does not generate answers; it selects from predefined answers by first predicting the question's label. The authors describe RoBERTa-base as demonstrating the most balanced and consistent performance across folds, and they explicitly flag the perfect BERT Large Uncased result as warranting cautious interpretation because it could indicate overfitting.

Load-bearing premise

The load-bearing premise is that the Healthline-derived dataset, with its 5-fold splits, fairly represents the range of real medical questions; if near-duplicate templated questions appear in both training and validation, the reported accuracy measures memorization rather than generalization.

Editorial extensions

If this is right

  • If the reported scores hold, a fine-tuned classifier can serve medical answers with near-perfect fidelity to a curated answer bank, with BERT Large Uncased as the top performer.
  • RoBERTa-base offers a nearly identical result (99.87% accuracy) at lower training cost, making it the pragmatic choice when compute is limited.
  • The LoRA-tuned RoBERTa-large performs markedly worse (78.47%), suggesting that in this low-resource setting low-rank adaptation trades away accuracy for efficiency.
  • Because answers are predefined, the approach can only handle questions that map onto the existing label set; novel or complex queries will fall outside its coverage.
  • The paper's own caution implies that the 100% figure should be re-tested on more diverse data before clinical deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The perfect scores most plausibly reflect near-duplicate templated questions shared between training and validation folds; the paper reports no deduplication, and Table 1 shows many paraphrases of 'What is diabetes?'.
  • A deduplicated or out-of-distribution evaluation would likely lower all reported numbers, and the ordering of models could change once the easiest pattern-matching signal is removed.
  • A testable extension is to freeze the label set but generate answers dynamically with a generative model, turning the strong classifier into a real open-ended medical QA system.
  • The large gap between LoRA RoBERTa-large and the other models may be a tuning artifact (rank, epochs, learning rate) rather than a property of LoRA itself; a similar LoRA configuration for BERT Large would isolate that variable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper fine-tunes four models (LoRA Roberta-large, Roberta-base, Bert Uncased, and Bert Large Uncased) on a 6,800-sample Healthline-derived medical question classification dataset and reports that Bert Large Uncased achieves 100% accuracy, precision, recall, and F1, concluding that it is the preferred model for high-accuracy medical QA. The evaluation uses 5-fold cross-validation with accuracy, precision, recall, and F1 metrics, and also compares training times.

Significance. If the reported performance numbers were trustworthy, the paper would provide a useful comparison of standard classification models for medical question classification. The use of a publicly available dataset and standard evaluation metrics is a modest strength, as is the authors' explicit acknowledgment in Section 6 that the perfect BERT Large score may reflect overfitting. However, the central empirical claim is not supported because of likely data leakage from templated, non-deduplicated questions, an inconsistent train/validation split arithmetic, and missing hyperparameter details. The contribution is therefore not currently established.

major comments (4)
  1. [Section 4, Table 1] The dataset is composed largely of templated paraphrases (e.g., 'What is diabetes?', 'Tell me about diabetes?', 'What kind of disease is diabetes?', 'Can you elaborate on diabetes?'), and no deduplication or grouped splitting is reported before 5-fold cross-validation. Random folds can place near-duplicate questions in both the training and validation splits, allowing a high-capacity model like BERT Large to memorize template-to-label mappings. The paper itself cautions in Section 6 that the perfect scores 'could indicate potential overfitting, especially if the evaluation dataset lacks diversity.' The 100% result is therefore likely an artifact of leakage rather than genuine generalization.
  2. [Section 4] The split arithmetic is internally inconsistent: the text states the dataset has 6,800 samples, with 70% (4,760) for training and the remaining 30% (1,020) for validation. Thirty percent of 6,800 is 2,040, not 1,020, and 4,760 + 1,020 = 5,780, which is less than 6,800. This discrepancy invalidates the reported evaluation setup and makes it impossible to know how many samples actually appeared in the validation folds.
  3. [Section 5, Experimental Setup] The paper claims that hyperparameters such as learning rate, batch size, and optimizer were 'carefully examined and tuned' but does not report any concrete values. No learning rate, batch size, optimizer, LoRA rank/alpha, or number of training epochs is specified. Without these details the experiments are not reproducible, and the comparative conclusions cannot be independently verified or interpreted.
  4. [Section 6, LoRA Roberta-large results] The LoRA Roberta-large model achieves only 78.47% accuracy while the non-LoRA Roberta-base achieves 99.87%, yet the manuscript provides no analysis of this large gap. Given that Section 3 emphasizes LoRA as an 'Innovative Use' intended to maintain accuracy while reducing computational cost, the unexplained underperformance undermines a key part of the paper's claimed contribution.
minor comments (4)
  1. [Section 4] The phrase 'data cleaning to remove unnecessary or redundant information' is stated but no concrete cleaning or deduplication steps are described, which is directly relevant to the leakage concern.
  2. [Section 6] The text states that 'all the models eventually reach high accuracy,' but the LoRA Roberta-large model peaks around 78%, so this claim is inaccurate and should be rephrased.
  3. [Section 3] The sentence 'All the 5 fold cross validation on the two models were done on 10 epochs each' appears to refer to 'two models' but four models are used; this is likely a typo and should be corrected.
  4. [Throughout] There are numerous typographical and grammatical errors (e.g., 'developtment', 'IIn this study', 'accessing performance'), which should be corrected for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports a direct supervised fine-tuning and 5-fold cross-validation evaluation; the reported metrics are empirical measurements, not derivations from fitted inputs.

full rationale

The paper's central claim is an empirical benchmark: four pretrained LLMs are fine-tuned on a Healthline-derived classification dataset and evaluated by 5-fold cross-validation, with accuracy, precision, recall, and F1 reported on validation folds. No derivation chain exists in which a predicted quantity is defined in terms of the target quantity or in which a fitted parameter is renamed as a prediction. The 'predefined answer' stage is explicitly a lookup from a secondary dataset, not a learned answer-generation claim, and the classification labels come from the external HealifyAI dataset rather than from the models' outputs. All cited prior work, including the LoRA paper and the HealifyAI data source, is external to the authors; no load-bearing self-citation or imported uniqueness theorem is invoked. Potential issues such as templated near-duplicate questions (Section 4, Table 1), the inconsistent split arithmetic, and the paper's own acknowledgment that 100% scores 'could indicate potential overfitting' (Section 6) are validity and generalization concerns, not circularity: they do not make the evaluation equivalent to its inputs by construction. Therefore, the correct finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 2 assumptions · 0 invented entities

The central empirical claim rests on several unreported tuning choices and on the assumption that a small, template-heavy dataset is a fair generalization test. No new entities are introduced.

free parameters (5)
  • learning_rate
    Chosen by hand for each model; values are not reported, yet they directly determine the reported accuracy.
  • batch_size
    Set manually and never disclosed in the paper.
  • optimizer = not specified
    The text says hyperparameters such as optimizer were tuned, but the optimizer is never named.
  • LoRA rank and alpha
    Required for the LoRA Roberta-large variant; values are not given and likely explain its much lower score.
  • number of epochs = 10
    Chosen based on pilot experiments; affects convergence and the final metrics.
assumptions (2)
  • domain assumption The five cross-validation folds are independent and do not share near-duplicate templated questions.
    Section 4 and Table 1 show many questions are paraphrases of 'What is diabetes?', making leakage likely.
  • domain assumption Labels are mutually exclusive and each predefined answer is clinically correct.
    The paper uses labels and answers scraped from Healthline without independent medical validation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Comprehensive Study on Fine-Tuning Large Language Models for Medical Question Answering Using Classification Models and Comparative Analysis." pith.science (2026). https://pith.science/paper/YXVXCNYJ

@misc{pith2026250117190,
  author       = {Pith},
  title        = {Pith review of: A Comprehensive Study on Fine-Tuning Large Language Models for Medical Question Answering Using Classification Models and Comparative Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YXVXCNYJ}},
  note         = {Machine review of arXiv:2501.17190}
}
read the original abstract

This paper presents the overview of the development and fine-tuning of large language models (LLMs) designed specifically for answering medical questions. We are mainly improving the accuracy and efficiency of providing reliable answers to medical queries. In our approach, we have two stages, prediction of a specific label for the received medical question and then providing a predefined answer for this label. Various models such as RoBERTa and BERT were examined and evaluated based on their ability. The models are trained using the datasets derived from 6,800 samples that were scraped from Healthline. com with additional synthetic data. For evaluation, we conducted a comparative study using 5-fold cross-validation. For accessing performance we used metrics like, accuracy, precision, recall, and F1 score and also recorded the training time. The performance of the models was evaluated using 5-fold cross-validation. The LoRA Roberta-large model achieved an accuracy of 78.47%, precision of 72.91%, recall of 76.95%, and an F1 score of 73.56%. The Roberta-base model demonstrated high performance with an accuracy of 99.87%, precision of 99.81%, recall of 99.86%, and an F1 score of 99.82%. The Bert Uncased model showed strong results with an accuracy of 95.85%, precision of 94.42%, recall of 95.58%, and an F1 score of 94.72%. Lastly, the Bert Large Uncased model achieved the highest performance, with an accuracy, precision, recall, and F1 score of 100%. The results obtained have helped indicate the capability of the models in classifying the medical questions and generating accurate answers in the prescription of improved health-related AI solutions.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Multi-scale measures of time-varying epidemic spread on human mobility networks

    q-bio.QM 2026-07 conditional novelty 6.5 of 10

    A mobility-informed PDE-to-renewal framework defines inward, outward, type, pairwise, meeting-location, and network R(t) plus transience indicators for targeted epidemic control.

  2. Educational programs and crime: a compartmental model approach

    physics.soc-ph 2025-05 conditional novelty 5.0 of 10

    The authors propose an SIR-style model of crime with a prison-education compartment and show that the long-run outcome depends on a reproduction number and a second threshold set by education parameters.

Reference graph

Works this paper leans on

23 extracted references · 19 canonical work pages · cited by 2 Pith papers

  1. [1]

    Large language models in healthcare and medical domain: A review

    Nazi ZA, Peng W. Large language models in healthcare and medical domain: A review. Informatics. Vol 11: MDPI; 2024:57

    work page 2024

  2. [2]

    Large language models in medicine

    Thirunavukarasu AJ, Ting DSJ, Elangovan K, Gutierrez L, Tan TF, Ting DSW. Large language models in medicine. Nature medicine. 2023;29:1930-1940

    work page 2023

  3. [3]

    KIMedQA: towards building knowledge - enhanced medical QA models

    Zafar A, Sahoo SK, Varshney D, Das A, Ekbal A. KIMedQA: towards building knowledge - enhanced medical QA models. Journal of Intelligent Information Systems. 2024:1-26

    work page 2024

  4. [4]

    Lora: Low -rank adaptation of large language models

    Hu EJ, Shen Y, Wallis P, et al. Lora: Low -rank adaptation of large language models. arXiv preprint arXiv:2106.09685. 2021

    arXiv 2021

  5. [5]

    HealifyAI - LLM based Healthcare System

    "HealifyAI - LLM based Healthcare System" https://github.com/tanvir-ishraq/HealifyAI--LLM- based-Healthcare-System/tree/main. Vol 2024. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

    work page 2024

  6. [6]

    Enhancing health care communication with large language models —the role, challenges, and future directions

    Subramanian CR, Yang DA, Khanna R. Enhancing health care communication with large language models —the role, challenges, and future directions. JAMA Network Open. 2024;7:e240347-e240347

    work page 2024

  7. [7]

    Large language models could change the future of behavioral healthcare: a proposal for responsible development and ev aluation

    Stade EC, Stirman SW, Ungar LH, et al. Large language models could change the future of behavioral healthcare: a proposal for responsible development and ev aluation. NPJ Mental Health Research. 2024;3:12

    work page 2024

  8. [8]

    The opportunities and risks of large language models in mental health

    Lawrence HR, Schneider RA, Rubin SB, Matarić MJ, McDuff DJ, Bell MJ. The opportunities and risks of large language models in mental health. JMIR Mental Health. 2024;11:e59479

    work page 2024

Show all 23 references
  1. [9]

    Generative AI and large language models in health care: pathways to implementation

    Raza MM, Venkatesh KP, Kvedar JC. Generative AI and large language models in health care: pathways to implementation. npj Digital Medicine. 2024;7:62

    2024

  2. [10]

    Opportunities and challenges for ChatGPT and large language models in biomedicine and health

    Tian S, Jin Q, Yeganova L, et al. Opportunities and challenges for ChatGPT and large language models in biomedicine and health. Briefings in Bioinformatics. 2024;25:bbad493

    2024

  3. [11]

    Roberta: A robustly optimized bert pretraining approach

    Liu Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. 2019;364

    1907 arXiv

  4. [12]

    Albert: A lite bert for self-supervised learning of language representations

    Lan Z. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942. 2019

    1909 arXiv

  5. [13]

    Fine -tuning language models from human preferences

    Ziegler DM, Stiennon N, Wu J, et al. Fine -tuning language models from human preferences. arXiv preprint arXiv:1909.08593. 2019

    1909 arXiv

  6. [14]

    BioBERT: a pre-trained biomedical language representation model for biomedical text mining

    Lee J, Yoon W, Kim S, et al. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics. 2020;36:1234-1240

    2020

  7. [15]

    The promise of large language models in health care

    Arora A, Arora A. The promise of large language models in health care. The Lancet. 2023;401:641

    2023

  8. [16]

    The future landscape of large language mod els in medicine

    Clusmann J, Kolbinger FR, Muti HS, et al. The future landscape of large language mod els in medicine. Communications medicine. 2023;3:141

    2023

  9. [17]

    Artificial intelligence: revolutionizing cardiology with large language models

    Boonstra MJ, Weissenbacher D, Moore JH, Gonzalez -Hernandez G, Asselbergs FW. Artificial intelligence: revolutionizing cardiology with large language models. European Heart Journal. 2024;45:332-345

    2024

  10. [18]

    Implications of large language models such as ChatGPT for dental medicine

    Eggmann F, Weiger R, Zitzmann NU, Blatz MB. Implications of large language models such as ChatGPT for dental medicine. Journal of Esthetic and Restorative Dentistry. 2023;35:1098- 1102

    2023

  11. [19]

    Large language models in health care: Development, applications, and challenges

    Yang R, Tan TF, Lu W, Thirunavukarasu AJ, Ting DSW, Liu N. Large language models in health care: Development, applications, and challenges. Health Care Science. 2023;2:255-263

    2023

  12. [20]

    Updated primer on generative artificial intelligence and large language models in medical imaging for medical professionals

    Kim K, Cho K, Jang R, et al. Updated primer on generative artificial intelligence and large language models in medical imaging for medical professionals. Korean Journal of Radiology. 2024;25:224

    2024

  13. [21]

    Embracing large language models for medical applications: opportunities and challenges

    Karabacak M, Margetis K. Embracing large language models for medical applications: opportunities and challenges. Cureus. 2023;15

    2023

  14. [22]

    Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data

    Ding J-E, Thao PNM, Peng W-C, et al. Large Language Multimodal Models for 5-Year Chronic Disease Cohort Prediction Using EHR Data. arXiv preprint arXiv:2403.04785. 2024

    2024 arXiv

  15. [23]

    Process knowledge -infused learning for clinician -friendly explanations

    Roy K, Zi Y, Gaur M, et al. Process knowledge -infused learning for clinician -friendly explanations. Proceedings of the AAAI Symposium Series. Vol 12023:154-160. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 ...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.