REVIEW 3 major objections 6 minor 29 references
Leveraging large language models and traditional machine learning ensembles for ADHD detection from narrative transcripts
T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read An ensemble of LLaMA3, RoBERTa, and SVM lifts ADHD recall to 0.91 on narrative transcripts.
desk verdict Well-described ensemble application, but the headline superiority claim is statistically unsupported and the manuscript contains inconsistent numbers. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the majority-voting rule $\hat{y}^{Ens} = 1$ if $\hat{y}^{LLM} + \hat{y}^{RoBERTa} + \hat{y}^{SVM} \ge 2$, applied to the binary predictions of three deliberately heterogeneous classifiers. LLaMA3 is prompted as a psychiatrist with a DSM-5-based symptom checklist iteratively refined on the development set; RoBERTa is fine-tuned on interviewee-only transcripts with a sliding window for long inputs; the SVM combines 1,000 TF-IDF n-gram features with engineered transcript metrics such as mean interviewee response length, total number of responses, and mean interviewer question length. The premise is that diversity in inductive bias—pretrained discourse knowledge, task-specific fine-tuning, and explicit lexical and structural features—makes the majority vote more robust than any single model.
What would settle it
Run a paired exact test on the 89 test-set predictions comparing the ensemble to the best individual model; if the difference in error counts is not significant at the 0.05 level, the ensemble's claimed superiority is indistinguishable from chance. Alternatively, re-estimate the $F_1$ gap over many random 60/20/20 splits of the 441 transcripts; if the ensemble does not win in a majority of splits, the central claim collapses.
Extended reading notes
Core claim
The paper's central empirical claim is that majority voting over the three classifiers is the best configuration on the test set: it achieves the highest $F_1$ (0.71) and recall (0.91), with precision 0.59, compared with LLaMA3 ($F_1$ 0.67, recall 0.87), RoBERTa ($F_1$ 0.69, recall 0.87), and SVM ($F_1$ 0.68, recall 0.75). The confusion matrix shows the ensemble produces only 4 false positives versus 11 for SVM while keeping true positives at 15, which the authors interpret as the ensemble filtering out idiosyncratic single-model errors while preserving sensitivity to ADHD-related linguistic cues.
Load-bearing premise
The claim that the ensemble outperforms individual models assumes that the observed 0.02–0.03 $F_1$ improvement on an 89-transcript test set is signal rather than noise, given that the bootstrap confidence intervals overlap substantially and no significance test is reported.
Editorial extensions
If this is right
- If the result holds, psychiatric text screening can be improved without retraining an LLM: the language model contributes its pretrained understanding through a prompt, while smaller models capture task-specific patterns.
- The ensemble's 0.91 recall suggests it could function as a sensitive first-pass screening tool that flags children for full clinical evaluation, accepting lower precision as the cost of not missing cases.
- The framework transfers to other narrative-based diagnostic tasks such as depression or autism, where labeled data are scarce, because the classical components train on modest data and the LLM component needs no fine-tuning.
- The result indicates that traditional feature-based classifiers still add value on top of LLMs in clinical text, making hybrid models a viable alternative to scaling up a single model.
Reading between the lines
- The reported ensemble advantage over RoBERTa and SVM is a 0.02–0.03 $F_1$ gap on only 89 test instances; a paired significance test or repeated train/test splits would likely show the gap is not statistically reliable, so the headline claim should be read as suggestive rather than established.
- The engineered features (response length, number of responses) are proxies for verbosity and narrative fragmentation; the same ensemble architecture could be tested on other conditions with narrative coherence deficits, where these features would plausibly carry signal.
- Because the LLM is used zero-shot, its contribution may be replicated by any capable instruction-following LLM; a natural experiment is to swap in a smaller model to see whether the ensemble's diversity, rather than LLaMA3's raw size, drives the gain.
- Weighted or confidence-based voting may outperform majority voting, since LLaMA3's overprediction of the positive class (low precision) suggests its vote should probably be down-weighted.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a hybrid ensemble for binary ADHD detection from narrative transcripts collected in the HBN post-scan interview setting. The ensemble combines three complementary classifiers—LLaMA3-70B used with a manually refined DSM-5-based prompt, a fine-tuned RoBERTa model using a sliding window, and an SVM over TF-IDF plus engineered features—aggregated by majority voting. On a held-out test set of 89 transcripts, the paper reports that the ensemble achieves the highest F1 score (0.71, 95% CI [0.60–0.80]) and recall (0.91), outperforming individual models (LLaMA3 0.67, RoBERTa 0.69, SVM 0.68). The authors argue this demonstrates the promise of combining LLMs with traditional ML for psychiatric text classification.
Significance. If the reported result were robust, the contribution would be useful: the task is clinically relevant, the chosen dataset is non-trivial, and the three model families are genuinely diverse, so a majority-vote ensemble is a reasonable architectural hypothesis. The paper also has some experimental hygiene—a fixed train/dev/test split, bootstrap confidence intervals for F1, and a documented prompt-refinement procedure. However, the central empirical claim is currently not supported with statistical rigor, and the manuscript contains internally contradictory numbers. The contribution is therefore more of a proof-of-concept with a substantial correctness concern than an established finding.
major comments (3)
- [§4, Table 2, Figure 4] The confusion-matrix prose in Section 4 is numerically incompatible with the metrics in Table 2. The text states that LLaMA3 correctly identified 11 of 44 ADHD instances, RoBERTa 15, and SVM 23 (with 21 false negatives), and that the ensemble had 15 true positives with only 4 false positives. Given that the test set contains 45 ADHD and 44 non-ADHD instances (Table 1), these counts imply recall values of roughly 0.24, 0.33, 0.51, and 0.33 for LLaMA3, RoBERTa, SVM, and the ensemble, respectively—contradicting the reported recalls of 0.87, 0.87, 0.75, and 0.91. The manuscript must either correct the prose or the table, because as written the paper contains two mutually inconsistent versions of the primary experimental result.
- [§4, Table 2] The paper's central claim that the ensemble outperforms individual models rests on F1 differences of 0.02 over RoBERTa (0.71 vs. 0.69) and 0.03 over SVM (0.71 vs. 0.68) on an 89-instance test set. The bootstrap 95% confidence intervals overlap substantially (ensemble [0.60–0.80], RoBERTa [0.58–0.78], SVM [0.57–0.77]), and no significance test is reported. Because all models are evaluated on the same 89 instances, a paired test is appropriate (e.g., McNemar's test for binary predictions or a bootstrap confidence interval for the F1 difference). Without such a test, the observed advantage is statistically indistinguishable from noise, and the claim that 'the ensemble outperforms individual models' is not supported by the evidence presented.
- [Abstract and Table 1] The abstract reports 352 training instances and 89 validation instances, but Table 1 reports 264 training, 88 development, and 89 test instances. The abstract's numbers appear to combine training and development (264 + 88 = 352) and call the test set 'validation.' This inconsistency affects reproducibility and the interpretation of how hyperparameters and the LLaMA3 prompt were selected. Please clarify the exact split used for training, prompt/development tuning, and final evaluation, and use consistent terminology throughout.
minor comments (6)
- [§2.2] The sentence 'The detailed data statistics are shown in Table 2' should refer to Table 1, which contains the data statistics; Table 2 contains the performance metrics.
- [§3] The bootstrap description is confusing: 'Resampling was performed with replacement (N=1000) over 1000 iterations' should be rephrased to state that 1000 bootstrap samples were drawn from the test set to compute the 95% confidence intervals.
- [§2.2.1] There is a grammatical error: 'The model operate through a mechanism' should be 'The model operates through a mechanism.'
- [§2.2.1, Figure 3] The prompt output format is described as 'YES/NO.My reasons are ...' with an odd period placement; this should be cleaned up to 'YES/NO. My reasons are ...' for clarity.
- [§1, reference [18]] The text refers to 'Malvika et al.' but the reference is by Pillai et al.; please align in-text author names with the reference list.
- [§5.1] The limitations subsection acknowledges the small dataset and the majority-voting design, which is good; however, it should also acknowledge the absence of a significance test for the ensemble-vs-individual comparisons, as this is directly related to the reliability of the headline claim.
Circularity Check
No significant circularity: the central ensemble claim is an empirical held-out evaluation, not a self-referential derivation.
full rationale
The paper makes no derivation whose conclusion is presupposed by its inputs. The ensemble prediction is defined as a majority vote over three independently produced per-transcript predictions (the equation for y^Ens_i with threshold >= 2), and it is then evaluated against held-out test labels. The reported F1 and recall values are therefore empirical measurements, not identities or fitted parameters renamed as predictions. Model selection and prompt refinement were performed on training/development partitions with the test set held out, which is standard supervised practice and does not create a fitted-input-called-prediction loop. The paper's self-citations ([11], [12], [21]) support background claims about TF-IDF, transformer benchmarking, and supervised-versus-LLM behavior; none of these citations carries the central ensemble-superiority result, so no load-bearing self-citation chain exists. The internal inconsistency between the Section 4 confusion-matrix prose and Table 2 is a correctness/reproducibility flaw, not a circularity: it does not make the claimed comparison equivalent to the input. Likewise, overlapping confidence intervals and the absence of a paired significance test are statistical-support concerns, not circularity. No circular step can be quoted and exhibited, so the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (7)
- SVM regularization C =
1024
- SVM kernel =
radial basis function
- TF-IDF vocabulary size =
1000
- Engineered features =
mean interviewee response length, total response count, mean interviewer question length
- RoBERTa learning rate and number of epochs =
one of {1e-5, 2e-5, 4e-5} x {10, 15, 20}
- Sliding window size =
512 tokens
- Majority voting threshold =
at least 2 of 3 models
assumptions (4)
- domain assumption Final HBN clinician diagnosis is a valid gold standard for ADHD.
- domain assumption Post-scan narrative transcripts contain reliable linguistic markers of ADHD.
- domain assumption The test set was not used for prompt or hyperparameter selection.
- standard math Standard statistical tools (TF-IDF, SVM, bootstrap, majority voting) are appropriate for this task.
Cite this review
Pith. "Pith review of Leveraging large language models and traditional machine learning ensembles for ADHD detection from narrative transcripts." pith.science (2026). https://pith.science/paper/JAYLD4RE
@misc{pith2026250521324,
author = {Pith},
title = {Pith review of: Leveraging large language models and traditional machine learning ensembles for ADHD detection from narrative transcripts},
year = {2026},
howpublished = {\url{https://pith.science/paper/JAYLD4RE}},
note = {Machine review of arXiv:2505.21324}
}
abstract
Despite rapid advances in large language models (LLMs), their integration with traditional supervised machine learning (ML) techniques that have proven applicability to medical data remains underexplored. This is particularly true for psychiatric applications, where narrative data often exhibit nuanced linguistic and contextual complexity, and can benefit from the combination of multiple models with differing characteristics. In this study, we introduce an ensemble framework for automatically classifying Attention-Deficit/Hyperactivity Disorder (ADHD) diagnosis (binary) using narrative transcripts. Our approach integrates three complementary models: LLaMA3, an open-source LLM that captures long-range semantic structure; RoBERTa, a pre-trained transformer model fine-tuned on labeled clinical narratives; and a Support Vector Machine (SVM) classifier trained using TF-IDF-based lexical features. These models are aggregated through a majority voting mechanism to enhance predictive robustness. The dataset includes 441 instances, including 352 for training and 89 for validation. Empirical results show that the ensemble outperforms individual models, achieving an F$_1$ score of 0.71 (95\% CI: [0.60-0.80]). Compared to the best-performing individual model (SVM), the ensemble improved recall while maintaining competitive precision. This indicates the strong sensitivity of the ensemble in identifying ADHD-related linguistic cues. These findings demonstrate the promise of hybrid architectures that leverage the semantic richness of LLMs alongside the interpretability and pattern recognition capabilities of traditional supervised ML, offering a new direction for robust and generalizable psychiatric text classification.
Figures
Reference graph
Works this paper leans on
-
[1]
AI@Meta. 2024. Llama 3 Model Card. https://github.com/meta-llama/llama3/ blob/main/MODEL_CARD.md
work page 2024
-
[2]
Nizar Alsharif, Mosleh Hmoud Al-Adhaileh, Saleh Nagi Alsubari, and Mohammed Al-Yaari. 2024. ADHD Diagnosis Using Text Features and Predictive Machine Learning and Deep Learning Algorithms.Journal of Disability Research3, 7 (2024). doi:10.57197/JDR-2024-0082
-
[3]
Santiago Berrezueta-Guzman, Mohanad Kandil, María-Luisa Martín-Ruiz, Iván Pau de la Cruz, and Stephan Krusche. 2024. Future of ADHD Care: Evaluat- ing the Efficacy of ChatGPT in Therapy Enhancement.Healthcare (Basel)12, 6 (2024), 683. doi:10.3390/healthcare12060683
-
[4]
ELLEN BIALYSTOK, KORNELIA HAWRYLEWICZ, MELODY WISEHEART, and MAGGIE TOPLAK. 2017. Interaction of bilingualism and Attention- Deficit/Hyperactivity Disorder in young adults.Bilingualism: Language and Cognition20, 3 (2017), 588–601. doi:10.1017/S1366728915000887
-
[5]
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...
arXiv 2020
-
[6]
Florian Raphaël Cafiero, Juan Barrios Rudloff, and Simon Gabay. 2024. Harnessing Linguistic Analysis for ADHD Diagnosis Support: A Stylometric Approach to Self-Defining Memories. InProceedings of the Fifth Workshop on Resources and ProcessIng of linguistic, para-linguistic and extra-linguistic Data from people with various forms of cognitive/psychiatric/d...
work page 2024
-
[7]
Tianhua Chen, Ilias Tachmazidis, Sotiris Batsakis, Marios Adamou, Emmanuel Papadakis, and Grigoris Antoniou. 2023. Diagnosing attention-deficit hyperac- tivity disorder (ADHD) using artificial intelligence: a clinical study in the UK. Frontiers in PsychiatryVolume 14 - 2023 (2023). doi:10.3389/fpsyt.2023.1164433
-
[8]
Corinna Cortes and Vladimir Vapnik. 1995. Support-vector networks.Machine learning20 (1995), 273–297
work page 1995
Show all 29 references
-
[9]
Bradley Efron. 1979. Another Look at the Jackknife.The Annals of Statistics7 (1979), 1–26. Issue 1
1979
-
[10]
C Thomas Gualtieri and Lynda G Johnson. 2005. ADHD: Is Objective Diagnosis Possible?Psychiatry (Edgmont)2, 11 (November 2005), 44–53
2005
-
[11]
Yuting Guo, Mohammed A Al-Garadi, Wendy M Book, Lindsey C Ivey, Fred H Rodriguez III, Cheryl L Raskind-Hood, Chad Robichaux, and Abeed Sarker. 2023. Supervised text classification system detects Fontan patients in electronic records with higher accuracy than ICD codes.Journal ...
2023
-
[12]
Yuting Guo, Xiangjue Dong, Mohammed Ali Al-Garadi, Abeed Sarker, Cecile Paris, and Diego Mollá Aliod. 2020. Benchmarking of Transformer-Based Pre- Trained Models on Social Media Text Classification Datasets. InProceedings of the 18th Annual Workshop of the Australasian Languag...
2020
-
[13]
Ida Bonnerup Jepsen, Esben Hougaard, Susan Tomczak Matthiesen, and Rikke Lambek. 2022. A Systematic Review and Meta-analysis of Narrative Language Abilities in Children with Attention-Deficit/Hyperactivity Disorder.Res Child Adolesc Psychopathol50, 6 (June 2022), 737–751. doi:...
2022 doi
-
[14]
Tong Min Kim, Young-Hoon Kim, Sung-Hee Song, In-Young Choi, Dai-Jin Kim, and Taehoon Ko. 2024. Explainability Enhanced Machine Learning Model for Classifying Intellectual Disability and Attention-Deficit/Hyperactivity Disorder With Psychological Test Reports.J Korean Med Sci40...
2024 doi
-
[15]
Lee, Noelle Lim, and Michael Guerzhoy
Claire S. Lee, Noelle Lim, and Michael Guerzhoy. 2024. Detecting a Proxy for Potential Comorbid ADHD in People Reporting Anxiety Symptoms from Social Media Data. arXiv:2403.05561 [cs.CY] https://arxiv.org/abs/2403.05561
2024 arXiv
-
[16]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. InarXiv(Online). arXiv. Issue
2019
- [17]
-
[18]
Nguyen, Yuta Y
Valeria Parlatini, Takashi Itahashi, Yeji Lee, Siwei Liu, Thuan T. Nguyen, Yuta Y. Aoki, Stephanie J. Forkel, Marco Catani, Katya Rubia, Juan H. Zhou, Declan G. Murphy, and Samuele Cortese. 2023. White matter alterations in Attention- Deficit/Hyperactivity Disorder (ADHD): a s...
2023
-
[19]
Malvika Pillai, Jose Posada, Rebecca M Gardner, Tina Hernandez-Boussard, and Yair Bannett. 2024. Measuring quality-of-care in treatment of young children with attention-deficit/hyperactivity disorder using pre-trained language models. Journal of the American Medical Informatic...
2024
-
[20]
Laura Reale, Beatrice Bartoli, Massimo Cartabia, Michele Zanetti, Maria An- tonella Costantino, Maria Paola Canevini, Cristiano Termine, Maurizio Bonati, and Lombardy ADHD Group. 2017. Comorbidity prevalence and treatment out- come in children and adolescents with ADHD.Eur Chi...
2017 doi
-
[21]
Pulini, Wesley T
Alfredo A. Pulini, Wesley T. Kerr, Sandra K. Loo, and Agatha Lenartowicz
-
[22]
Dey, and Dakuo Wang
Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghassemi, Anind K. Dey, and Dakuo Wang. 2024. Mental-LLM: Lever- aging Large Language Models for Mental Health Prediction via Online Text Data.Proceedings of the ACM on Interactive, Mobile, ...
2024 doi
-
[23]
Han Yang, Mingchen Li, Huixue Zhou, Yongkang Xiao, Qian Fang, and Rui Zhang
-
[24]
Miaomiao Yu, Xinyu Gao, Xiaoyu Niu, Mengzhe Zhang, Zhengui Yang, Shaoqiang Han, Jingliang Cheng, and Yong Zhang. 2023. Meta-analysis of structural and functional alterations of brain in patients with attention-deficit/hyperactivity disorder.Frontiers in psychiatry13 (2023), 1070142
2023
-
[25]
Abeed Sarker, Rui Zhang, Yanshan Wang, Yunyu Xiao, Sudeshna Das, Dalton Schutte, David Oniani, Qianqian Xie, and Hua Xu. 2024. Natural Language Processing for Digital Health in the Era of Large Language Models.Yearbook of Medical Informatics33, 01 (2024), 229–240. doi:10.1055/...
2024 doi
-
[120]
doi:10.1016/j.bpsc.2018.06.003
2018 doi
-
[957]
doi:10.1093/jamia/ocae001 arXiv:https://academic.oup.com/jamia/article- pdf/31/4/949/57148541/ocae001.pdf
-
[2019]
Biological Psychiatry: Cognitive Neuroscience and Neuroimaging4, 2 (2019), 108–
Classification Accuracy of Neuroimaging Biomarkers in Attention- Deficit/Hyperactivity Disorder: Effects of Sample Size and Circular Analysis. Biological Psychiatry: Cognitive Neuroscience and Neuroimaging4, 2 (2019), 108–
2019
-
[2023]
doi:10.1101/2023.12.21.23300380 arXiv:https://www.medrxiv.org/content/early/2023/12/24/2023.12.21.23300380.full.pdf
One LLM is not Enough: Harnessing the Power of Ensemble Learning for Medical Question Answering.medRxiv(2023). doi:10.1101/2023.12.21.23300380 arXiv:https://www.medrxiv.org/content/early/2023/12/24/2023.12.21.23300380.full.pdf
2023 doi
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.