Pith. sign in

REVIEW 4 major objections 4 minor 26 references

From Chat to Checkup: Can Large Language Models Assist in Diabetes Prediction?

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper reports that a 27-billion-parameter open-source language model, prompted with three labeled examples, predicts diabetes from Pima Indian patient records with 74.35% accuracy and an F1-score of 0.7250, edging out Random Forest’s…

desk verdict A small honest benchmark with a plausible but unverifiable headline result; the few-shot example provenance and test-set identity must be disclosed before the F1 comparison can be trusted. read the letter →

arxiv 2506.14949 v1 pith:T2XHZLAL submitted 2025-06-17 cs.CL

Shadman Sakib , Oishy Fatema Akhand , Ajwad Abrar This is my paper · ORCID
classification cs.CL
keywords LargelanguagemodelsDiabetespredictionPimaIndianDatabaseFew-shotpromptingIn-contextlearningStructurednumericaldataMedicalMachinebaselines
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether large language models can predict diabetes from structured numerical health records when prompted in natural language, without any task-specific training. It reports that Gemma-2-27B with three-shot prompting reaches 74.35% accuracy and an F1-score of 0.7250 on the Pima Indian Diabetes Database, and that both Gemma-2-27B and GPT-4o beat three traditional machine learning baselines in F1. The authors argue this shows LLMs can serve as flexible, zero-training tools for early medical screening, especially where labeled data or specialized models are scarce, while acknowledging that classic models still hold the accuracy edge. The study also finds that three-shot prompting generally improves performance over zero-shot, but gains are inconsistent across models.

What carries the argument

The load-bearing mechanism is few-shot in-context learning applied to a tabular binary classification task. Each patient record is rendered as a natural-language sentence (for example, 'Pregnancies: 3, Glucose: 120, ...') and the model is asked to return only 0 or 1. In the one-shot and three-shot settings, one or three labeled records from the dataset are inserted into the prompt as demonstrations, and the LLM’s existing language-understanding priors are asked to generalize the pattern to the new record. The paper attributes the measured performance differences across models and prompt counts to the strength of these in-context learning mechanisms, and it evaluates the results using accuracy, precision, recall, and F1-score averaged over three independent runs.

What would settle it

Inspect the code or prompts in the linked GitHub repository to determine whether the one-shot and three-shot examples were selected from the 80% training split or from the full dataset. Then re-run the three-shot Gemma-2-27B evaluation with examples guaranteed to be drawn only from the training partition and with the same 80/20 split used for Random Forest; if the F1 drops to near Random Forest’s level, the paper’s central comparison no longer supports the conclusion that an LLM beats classical ML in F1.

Watch

Extended reading notes

Core claim

The central claim is that a large language model, prompted with three labeled patient examples in plain text, can match or exceed conventional machine learning classifiers in balanced prediction quality for diabetes, despite never being trained on the dataset. Concretely, Gemma-2-27B (three-shot) achieves F1 = 0.7250 on PIDD, outperforming Random Forest (0.6726), Logistic Regression (0.6607), and SVM (0.6019), while GPT-4o (three-shot) reaches F1 = 0.7234. The best LLM accuracy, 74.35%, is slightly below Random Forest’s 75.97%, but the LLMs require no feature engineering, no model fitting, and no per-patient retraining, only a textual prompt with the patient’s eight numeric measurements. The authors interpret this as evidence that prompt-based LLM inference is a viable complement to classical ML in low-resource medical settings, even though it remains short of the 85–98% accuracies reported for specialized deep learning pipelines on the same dataset.

Load-bearing premise

The paper assumes that the one and three labeled examples used in the few-shot prompts come from the training portion of the dataset and never from the test set, but it does not state this; if any test patient appears as an in-context example, the reported few-shot gains, including Gemma-2-27B’s headline F1, would be inflated.

Editorial extensions

If this is right

  • If the result holds, an open-source 27B model prompted with three examples becomes a zero-training baseline that matches or beats classical ML on F1 for a standard tabular medical benchmark.
  • Clinicians and researchers in data-scarce settings could use prompt engineering rather than model training to obtain a first-pass diabetes risk classifier, provided the prompts are carefully designed per model.
  • The performance gap between Gemma-2-27B and GPT-4o is small (74.35% vs 74.22% accuracy), suggesting that open-weight models can be competitive with proprietary APIs for structured numeric classification.
  • The inconsistent benefit of three-shot prompting across models implies that prompt-count alone is not a reliable lever; model choice and prompt phrasing matter as much as the number of demonstrations.
  • Traditional ML retains an accuracy advantage, so the practical near-term use of LLMs is likely as an interpretable-by-language sanity check or triage tool, not as a replacement for trained classifiers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A concrete, testable extension the paper leaves open is to examine where the few-shot examples come from: if any demonstration patient also appears in the test set, the reported F1 gains would be inflated by label leakage. Re-running the experiment with examples drawn strictly from the training partition would settle this.
  • The paper’s comparison is asymmetric because LLM scores are averaged over three shuffled test sets while ML baselines come from a single, unseeded 80/20 split; without fixing the test set identity, the reported F1 gaps (0.7250 vs 0.6726) could shrink or grow by chance.
  • A further implication is that the high accuracy but low F1 of Gemini Flash 2.0 (73% accuracy, 0.47 F1) probably reflects majority-class overprediction, which means accuracy alone can mislead in imbalanced medical data; a calibration or decision-threshold analysis would clarify which models are actually usable in screening.
  • If the three-shot effect is real, then a natural next step is to test whether scaling to 5-, 10-, or dynamically retrieved examples further narrows the gap to Random Forest, and whether the same pattern holds on other tabular medical datasets.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript reports an empirical comparison of six large language models (Gemma-2-27B, Mistral-7B, Llama-3.1-8B, Llama-3.2-2B, GPT-4o, and Gemini Flash 2.0) with three traditional machine learning classifiers (Random Forest, Logistic Regression, and SVM) on the Pima Indians Diabetes Database. The authors use zero-shot, one-shot, and three-shot prompting with natural-language feature descriptions and evaluate accuracy, precision, recall, and F1-score. The headline result is that Gemma-2-27B with three-shot prompting achieves F1=0.7250, surpassing the traditional ML baselines, with GPT-4o close behind at F1=0.7234. The paper concludes that proprietary models perform better and that three-shot prompting consistently helps, although these conclusions conflict with the reported table.

Significance. If the result holds, the paper would provide a modest but useful data point: a zero-training open-source LLM can compete with conventional classifiers on a small tabular medical benchmark, which is relevant for low-resource or data-scarce settings. The breadth of the model comparison and the use of standard evaluation metrics are strengths, and the decision to average LLM runs is a positive step toward stability. However, the empirical claim is not yet load-bearing because the provenance of few-shot examples, the identity of test sets, and the internal consistency of Table III are not established. Resolving the leakage concern and correcting the table would make this a valuable short empirical study.

major comments (4)
  1. [Section III-C and III-B] The one-shot and three-shot prompts use labeled examples whose provenance is not specified. Section III-C says the three-shot prompt includes 'three labeled examples from the dataset' but does not state that the examples are drawn exclusively from the 80% training partition or that test patients are excluded. Section III-B says each LLM configuration was run on 'independently shuffled test sets' with no reported split seed or test-set identity. If a test patient appears as an in-context example, the model can copy its label, inflating accuracy and F1. This directly affects the headline comparison of Gemma-2-27B three-shot (F1=0.7250) with Random Forest (F1=0.6726). The authors must document how demonstrations were sampled, use a fixed held-out test set shared by all LLM runs and ML baselines, and report the split seed or otherwise demonstrate that no test patient appears among the demonstrations.
  2. [Table III] The Gemma-2-27B one-shot row reports F1=0.6549 alongside precision=0.7232 and recall=0.7220. Substituting these values into Eq. (4) gives F1 approximately 0.7226, so the table is internally inconsistent. The identical value in the accuracy column suggests a transcription or copying error. Because Table III is the sole evidence for the paper's main claims, the corrected metrics and a statement of how they were computed are required before the comparison can be accepted.
  3. [Section III-E, Abstract, and Conclusion] The statement 'Across nearly all LLMs, three-shot prompting consistently led to performance improvements' is contradicted by Table III: Gemini Flash 2.0's accuracy falls from 0.7331 (zero-shot) to 0.7305 (three-shot), and Llama-3.1-8B's three-shot accuracy (0.3568) is below its zero-shot accuracy (0.3646). The abstract's claim that 'proprietary LLMs perform better than open-source ones' conflicts with Gemma-2-27B, an open-source model, attaining the highest LLM accuracy and F1 in Table III. These claims should be revised to match the data or supported with additional evidence.
  4. [Section III-B and III-E] The LLM evaluation averages three runs over 'independently shuffled test sets,' while the ML baselines are evaluated on one fixed 80/20 split. No standard deviations, confidence intervals, or statistical tests are reported, and the split seed is not given. With only three runs and no shared test partition, the statement that LLMs 'outperformed traditional models in F1-score' cannot be distinguished from variation due to test-set composition. Report per-run results, variance, and a common held-out set across all models.
minor comments (4)
  1. [Abstract and Section IV] The abstract and conclusion state that three-shot prompting 'consistently enhanced model performance,' but Section III-F itself notes that 'improvements from three-shot prompting were not consistent across models'; this internal contradiction should be resolved.
  2. [Table I and Section II] The text reports that Mujumdar et al. found Logistic Regression achieved 96% accuracy, but Table I lists 77% for Gradient Boost, LDA, and AdaBoost as the best results from that paper; these values should be reconciled.
  3. [Footnote 1] The GitHub URL is broken across lines and contains a space; provide a clean, clickable link.
  4. [References] Reference [22] for Gemini Flash 2.0 cites only a general DeepMind page and reference [25] is a blog post; use primary technical reports or peer-reviewed sources where available.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the study is an empirical benchmark with no fitted constants, and the only self-citations are background context.

full rationale

The paper's central claim is empirical: six LLMs are prompted zero-/one-/three-shot on the PIDD and compared with three scikit-learn baselines. There is no derivation or fitted constant from which the reported accuracies or F1-scores follow by construction, and no quantity is defined in terms of the target metric. The self-citations [12] and [14] support only background statements about LLM performance and bias; they are not load-bearing for the headline comparison. The paper does not import a uniqueness theorem, smuggle in an ansatz via citation, or rename a known result. Concerns about the provenance of the few-shot examples and the internal inconsistency in Gemma-2-27B's one-shot F1 are correctness/reproducibility risks, not circularity. The central empirical claim is therefore self-contained and not circular.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The evaluation uses a standard public dataset and standard metrics; no physically fitted constants or invented entities are introduced. The main tacit premises are data quality, split validity, and prompt-example provenance, which are not independently verified.

free parameters (2)
  • LLM decoding temperature = not reported
    Output variability across three runs is raw evidence of stochastic decoding, yet temperature is not stated; the reported mean may depend on it.
  • 80/20 train-test split seed = not reported
    ML baselines are evaluated on one fixed split with no seed; a different split could change the comparison.
assumptions (3)
  • domain assumption PIDD is a valid and representative diabetes prediction benchmark.
    The paper relies on PIDD as ground truth without examining its known limitations (small sample, missing values, non-representative population) despite citing its provenance in Section III-A.
  • domain assumption One- and three-shot examples come from the training partition and exclude test patients.
    Section III-C says examples come 'from the dataset' but never specifies partition; if this fails, few-shot results are leaky.
  • ad hoc to paper Averaging three LLM runs and a single ML split is sufficient for comparison.
    The paper sets this evaluation protocol in Section III-B/E without statistical justification, which is a choice specific to this study.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Chat to Checkup: Can Large Language Models Assist in Diabetes Prediction?." pith.science (2026). https://pith.science/paper/T2XHZLAL

@misc{pith2026250614949,
  author       = {Pith},
  title        = {Pith review of: From Chat to Checkup: Can Large Language Models Assist in Diabetes Prediction?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/T2XHZLAL}},
  note         = {Machine review of arXiv:2506.14949}
}
read the original abstract

While Machine Learning (ML) and Deep Learning (DL) models have been widely used for diabetes prediction, the use of Large Language Models (LLMs) for structured numerical data is still not well explored. In this study, we test the effectiveness of LLMs in predicting diabetes using zero-shot, one-shot, and three-shot prompting methods. We conduct an empirical analysis using the Pima Indian Diabetes Database (PIDD). We evaluate six LLMs, including four open-source models: Gemma-2-27B, Mistral-7B, Llama-3.1-8B, and Llama-3.2-2B. We also test two proprietary models: GPT-4o and Gemini Flash 2.0. In addition, we compare their performance with three traditional machine learning models: Random Forest, Logistic Regression, and Support Vector Machine (SVM). We use accuracy, precision, recall, and F1-score as evaluation metrics. Our results show that proprietary LLMs perform better than open-source ones, with GPT-4o and Gemma-2-27B achieving the highest accuracy in few-shot settings. Notably, Gemma-2-27B also outperforms the traditional ML models in terms of F1-score. However, there are still issues such as performance variation across prompting strategies and the need for domain-specific fine-tuning. This study shows that LLMs can be useful for medical prediction tasks and encourages future work on prompt engineering and hybrid approaches to improve healthcare predictions.

Figures

Figures reproduced from arXiv: 2506.14949 by the authors.

Figure 1
Figure 1. Workflow of the experimental pipeline from data preparation to evaluation. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. One-shot and Three-shot prompts in a diabetes prediction. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Comparison of performance metrics between Gemma [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 25 canonical work pages

  1. [1]

    Type 2 Diabetes

    Rajeev Goyal, Mayank Singhal, and Ishwarlal Jialal. Type 2 Diabetes . StatPearls Publishing, 2023

    work page 2023

  2. [2]

    Diagnosis and classification of diabetes mellitus

    American Diabetes Association. Diagnosis and classification of diabetes mellitus. Diabetes care, 37(Supplement 1):S81–S90, 2014

    work page 2014

  3. [3]

    Diabetes mel- litus, the fastest growing global public health concern: Early detection should be focused

    Md Jamal Hossain, Md Al-Mamun, and Md Rabiul Islam. Diabetes mel- litus, the fastest growing global public health concern: Early detection should be focused. Health Science Reports , 7(3):e2004, 2024

    work page 2024

  4. [4]

    Rising trends of diabetes in south asia: A systematic review and meta-analysis

    Priyanga Ranasinghe, Nethmini Rathnayake, Sameera Wijayawardhana, Hajanthy Jeyapragasam, V Jithmal Meegoda, Ranil Jayawardena, and Anoop Misra. Rising trends of diabetes in south asia: A systematic review and meta-analysis. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, page 103160, 2024

    work page 2024

  5. [5]

    J. W. Smith, J. E. Everhart, W. C. Dickson, W. C. Knowler, and R. S. Johannes. Using the adap learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Symposium on Computer Applications and Medical Care, pages 261–265, Washington, DC, USA,

    work page

  6. [6]

    Deep learning approach for diabetes prediction using pima indian dataset

    Huma Naz and Sachin Ahuja. Deep learning approach for diabetes prediction using pima indian dataset. Journal of Diabetes & Metabolic Disorders, 19:391–403, 2020

    work page 2020

  7. [7]

    Diabetes prediction using machine learning algorithms

    Aishwarya Mujumdar and Vb Vaidehi. Diabetes prediction using machine learning algorithms. Procedia Computer Science , 165:292– 299, 2019

    work page 2019

  8. [8]

    Pima indians diabetes mellitus classification based on machine learning (ml) algorithms

    Victor Chang, Jozeene Bailey, Qianwen Ariel Xu, and Zhili Sun. Pima indians diabetes mellitus classification based on machine learning (ml) algorithms. Neural Computing and Applications , 35(22):16157–16173, 2023

    work page 2023

Show all 26 references
  1. [9]

    Basic concepts of artificial neural network (ann) modeling and its application in pharma- ceutical research

    S Agatonovic-Kustrin and Rosemary Beresford. Basic concepts of artificial neural network (ann) modeling and its application in pharma- ceutical research. Journal of pharmaceutical and biomedical analysis , 22(5):717–727, 2000

    2000

  2. [10]

    Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network

    Alex Sherstinsky. Fundamentals of recurrent neural network (rnn) and long short-term memory (lstm) network. Physica D: Nonlinear Phenomena, 404:132306, 2020

    2020

  3. [11]

    A novel rfe-gru model for diabetes classification using pima indian dataset

    Mahmoud Y Shams, Zahraa Tarek, and Ahmed M Elshewey. A novel rfe-gru model for diabetes classification using pima indian dataset. Scientific Reports, 15(1):982, 2025

    2025

  4. [12]

    Performance evaluation of large language models in bangla consumer health query summarization, 2025

    Ajwad Abrar, Farzana Tabassum, and Sabbir Ahmed. Performance evaluation of large language models in bangla consumer health query summarization, 2025

    2025

  5. [13]

    Au large, February 2024

    Mistral AI Team. Au large, February 2024. Accessed: 2025-02-16

    2024

  6. [14]

    Religious bias landscape in language and text-to-image models: Analysis, detection, and debiasing strategies, 2025

    Ajwad Abrar, Nafisa Tabassum Oeshy, Mohsinul Kabir, and Sophia Ananiadou. Religious bias landscape in language and text-to-image models: Analysis, detection, and debiasing strategies, 2025

    2025

  7. [15]

    A comparative study of diabetes detection using the pima indian diabetes database

    ABDULAZEEZ MOUSA, W ARAZ MUSTAFA, RIDW AN BOY A MARQAS, and SHIV AN HM MOHAMMED. A comparative study of diabetes detection using the pima indian diabetes database. Journal of Duhok University , 26(2):277–288, 2023

    2023

  8. [16]

    Analysis of pima indian diabetes using knn classifier and support vector machine technique

    VK Kalaiselvi, Hariharan Shanmugasundaram, E Aishwarya, M Ragavi, C Nandhini, and SJ Bhuvaneshwari. Analysis of pima indian diabetes using knn classifier and support vector machine technique. In 2022 Third International Conference on Intelligent Computing Instrumentation and C...

    2022

  9. [17]

    Gemma-2-27b: A state-of-the-art open-source language model

    Gemma Team et al. Gemma-2-27b: A state-of-the-art open-source language model. http://arxiv.org/pdf/2408.00118v3, 2024

    2024 arXiv

  10. [18]

    Mistral 7b: A next-generation open-source language model

    Mistral AI. Mistral 7b: A next-generation open-source language model. https://mistral.ai/, 2023

    2023

  11. [19]

    Llama 3.1 8b: Advancing open language models for scalable nlp

    Meta AI. Llama 3.1 8b: Advancing open language models for scalable nlp. https://ai.meta.com/llama, 2024

    2024

  12. [20]

    Llama 3.2 2b: A compact variant for efficient deployment

    Meta AI. Llama 3.2 2b: A compact variant for efficient deployment. https://ai.meta.com/llama, 2024

    2024

  13. [21]

    Gpt 4o: The next iteration in the gpt series

    OpenAI. Gpt 4o: The next iteration in the gpt series. https://openai. com/research/gpt-4, 2023

    2023

  14. [22]

    Gemini flash 2.0: Advancing multimodal language models with flash attention

    Google DeepMind. Gemini flash 2.0: Advancing multimodal language models with flash attention. https://deepmind.com/research, 2024

    2024

  15. [23]

    Can large language models reason over structured data? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , 2023

    Bailin Wang, Debajyoti Ghosh, Joon Sung Park, Wenhao Wu, Chen- guang Zhu, and Han Zhang. Can large language models reason over structured data? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL) , 2023

    2023

  16. [24]

    Evaluation metrics and statistical tests for machine learning

    Oona Rainio, Jarmo Teuho, and Riku Kl ´en. Evaluation metrics and statistical tests for machine learning. Scientific Reports , 14(1):6086, 2024

    2024

  17. [25]

    What is in-context learning? https://www.lakera.ai/blog/ what-is-in-context-learning, 2024

    Lakera. What is in-context learning? https://www.lakera.ai/blog/ what-is-in-context-learning, 2024. Accessed: 2025-05-09. 6

    2024

  18. [1988]

    IEEE Computer Society Press

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.