{"id":"4904140e-7081-40e1-8ff7-85d6109b5caa","arxiv_id":"2505.21324","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A three-model majority-vote ensemble (LLaMA3, RoBERTa, SVM) classifies ADHD from post-scan narrative transcripts with F1 0.71, a modest and not statistically robust gain over single models.","lead":"The authors combine LLaMA3, RoBERTa, and a TF-IDF support vector machine to classify ADHD from narrative transcripts of children recalling a short film. The majority-vote ensemble reaches F1 0.71 on 89 test transcripts, but the gain over individual models is within overlapping confidence intervals.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ensemble-vs-single-model F1 gains of 0.02–0.03 on 89 test cases are within the noise floor; no paired significance test is reported, so the paper's central superiority claim is not currently supported.","rationale":"In good faith, the paper's contribution is an empirical demonstration that an LLM + transformer + SVM majority-vote ensemble beats each component on ADHD transcript classification. That demonstration is only as strong as the reliability of the observed performance differences. With 89 test instances, an F1 gap of 0.02–0.03 is small relative to sampling variability, and the overlapping CIs plus the absence of any paired test leave 'outperforms' unsupported. This is a correctness-risk issue rather than a methodological or theoretical objection: majority voting is a sound and clearly described aggregation strategy, and the pipeline is reproducible in principle. The limitations section candidly acknowledges the small dataset, which reinforces rather than resolves the concern. The confusion-matrix prose inconsistency in Section 4 is an additional reporting problem that makes the underlying counts ambiguous, but even resolving it in favor of Table 2 does not remove the need for a significance test. The verdict remains CONDITIONAL: the method is reasonable and the omissions are fixable, so ACCEPT is not supportable, while REJECT would be excessive because the reported numbers, if verified, would provide some evidence for the ensemble approach.","tokens_in":12328,"tokens_out":8411,"duration_ms":93482,"concrete_test":"Reconstruct or obtain the per-instance binary predictions for all four models on the 89 test transcripts, then compute (1) a bootstrap 95% CI for the F1 difference between the ensemble and RoBERTa and between the ensemble and SVM over 10,000 resamples, and (2) McNemar's exact test for the paired discordant predictions on the same 89 instances. If the F1-difference CI includes 0 or the McNemar p-value exceeds 0.05, the empirical claim that the ensemble outperforms the individual models should be withdrawn or downgraded to 'statistically indistinguishable'. Also verify which of the two contradictory result descriptions (Section 4 text vs Table 2) matches the released predictions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that majority voting over LLaMA3, RoBERTa, and SVM outperforms every individual model. On the 89-instance test set this rests on an F1 gap of 0.02 over RoBERTa (0.71 vs 0.69) and 0.03 over SVM (0.71 vs 0.68), with overlapping bootstrap 95% CIs (Ensemble [0.60–0.80], RoBERTa [0.58–0.78], SVM [0.57–0.77]). No significance test is reported, and because the same 89 transcripts are scored by all models, the appropriate test is paired (e.g., McNemar or a bootstrap CI for the F1 difference). A compounding issue: Section 4's confusion-matrix prose is numerically incompatible with Table 2. If taken literally, the text implies LLaMA3 recall of about 11/44 = 0.25 and ensemble recall of 15/44 = 0.34, not the reported 0.87 and 0.91. Thus the manuscript currently contains two mutually contradictory sets of numbers, and the empirical basis for the superiority claim is not internally consistent. The load-bearing issue is that, even if Table 2 is taken at face value, the observed advantage is statistically indistinguishable from noise on this sample size.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a hybrid ensemble for binary ADHD detection from narrative transcripts collected in the HBN post-scan interview setting. The ensemble combines three complementary classifiers—LLaMA3-70B used with a manually refined DSM-5-based prompt, a fine-tuned RoBERTa model using a sliding window, and an SVM over TF-IDF plus engineered features—aggregated by majority voting. On a held-out test set of 89 transcripts, the paper reports that the ensemble achieves the highest F1 score (0.71, 95% CI [0.60–0.80]) and recall (0.91), outperforming individual models (LLaMA3 0.67, RoBERTa 0.69, SVM 0.68). The authors argue this demonstrates the promise of combining LLMs with traditional ML for psychiatric text classification.","tokens_in":12601,"tokens_out":3227,"duration_ms":32973,"significance":"If the reported result were robust, the contribution would be useful: the task is clinically relevant, the chosen dataset is non-trivial, and the three model families are genuinely diverse, so a majority-vote ensemble is a reasonable architectural hypothesis. The paper also has some experimental hygiene—a fixed train/dev/test split, bootstrap confidence intervals for F1, and a documented prompt-refinement procedure. However, the central empirical claim is currently not supported with statistical rigor, and the manuscript contains internally contradictory numbers. The contribution is therefore more of a proof-of-concept with a substantial correctness concern than an established finding.","major_comments":[{"comment":"The confusion-matrix prose in Section 4 is numerically incompatible with the metrics in Table 2. The text states that LLaMA3 correctly identified 11 of 44 ADHD instances, RoBERTa 15, and SVM 23 (with 21 false negatives), and that the ensemble had 15 true positives with only 4 false positives. Given that the test set contains 45 ADHD and 44 non-ADHD instances (Table 1), these counts imply recall values of roughly 0.24, 0.33, 0.51, and 0.33 for LLaMA3, RoBERTa, SVM, and the ensemble, respectively—contradicting the reported recalls of 0.87, 0.87, 0.75, and 0.91. The manuscript must either correct the prose or the table, because as written the paper contains two mutually inconsistent versions of the primary experimental result.","section":"§4, Table 2, Figure 4"},{"comment":"The paper's central claim that the ensemble outperforms individual models rests on F1 differences of 0.02 over RoBERTa (0.71 vs. 0.69) and 0.03 over SVM (0.71 vs. 0.68) on an 89-instance test set. The bootstrap 95% confidence intervals overlap substantially (ensemble [0.60–0.80], RoBERTa [0.58–0.78], SVM [0.57–0.77]), and no significance test is reported. Because all models are evaluated on the same 89 instances, a paired test is appropriate (e.g., McNemar's test for binary predictions or a bootstrap confidence interval for the F1 difference). Without such a test, the observed advantage is statistically indistinguishable from noise, and the claim that 'the ensemble outperforms individual models' is not supported by the evidence presented.","section":"§4, Table 2"},{"comment":"The abstract reports 352 training instances and 89 validation instances, but Table 1 reports 264 training, 88 development, and 89 test instances. The abstract's numbers appear to combine training and development (264 + 88 = 352) and call the test set 'validation.' This inconsistency affects reproducibility and the interpretation of how hyperparameters and the LLaMA3 prompt were selected. Please clarify the exact split used for training, prompt/development tuning, and final evaluation, and use consistent terminology throughout.","section":"Abstract and Table 1"}],"minor_comments":[{"comment":"The sentence 'The detailed data statistics are shown in Table 2' should refer to Table 1, which contains the data statistics; Table 2 contains the performance metrics.","section":"§2.2"},{"comment":"The bootstrap description is confusing: 'Resampling was performed with replacement (N=1000) over 1000 iterations' should be rephrased to state that 1000 bootstrap samples were drawn from the test set to compute the 95% confidence intervals.","section":"§3"},{"comment":"There is a grammatical error: 'The model operate through a mechanism' should be 'The model operates through a mechanism.'","section":"§2.2.1"},{"comment":"The prompt output format is described as 'YES/NO.My reasons are ...' with an odd period placement; this should be cleaned up to 'YES/NO. My reasons are ...' for clarity.","section":"§2.2.1, Figure 3"},{"comment":"The text refers to 'Malvika et al.' but the reference is by Pillai et al.; please align in-text author names with the reference list.","section":"§1, reference [18]"},{"comment":"The limitations subsection acknowledges the small dataset and the majority-voting design, which is good; however, it should also acknowledge the absence of a significance test for the ensemble-vs-individual comparisons, as this is directly related to the reliability of the headline claim.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an early-stage workshop submission (ACM 2018 template, arXiv May 2025) with a promising empirical setup but several material inconsistencies. The most serious issue is that the confusion-matrix prose in Section 4 directly contradicts Table 2; this is not a minor typo but a load-bearing inconsistency in the reported results. The second issue is the missing significance testing for the ensemble's claimed superiority over RoBERTa and SVM, which is essential for a paper whose title and abstract emphasize the ensemble's advantage. I recommend major revision rather than rejection, because the fix is within scope: correct the numbers, add a paired test or soften the superiority claim, and align the abstract with Table 1. If the authors cannot conduct a proper significance test, they should explicitly frame the result as preliminary evidence rather than a demonstrated improvement."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take on arXiv:2505.21324. The paper applies a textbook majority-voting ensemble—LLaMA3 via prompt, fine-tuned RoBERTa, and an SVM with TF-IDF plus engineered features—to ADHD classification from post-scan narrative transcripts of the HBN dataset. The application is new as far as I can tell, and the setup is described honestly: data source, prompt refinement cycles, hyperparameter ranges, and a candid limitations section. That's all to the good.\n\nThe problem is the evidence for the central claim. The ensemble beats the best single model (RoBERTa) by 0.02 in F1 (0.71 vs 0.69) on an 89-instance test set. The bootstrap CIs overlap heavily, and no paired significance test (McNemar or bootstrap on the F1 difference) is reported. On this sample size that gap is indistinguishable from noise. The discussion even claims the ensemble CI is 'tighter' than the others, but the interval widths are essentially identical (0.20 for ensemble vs 0.20 for RoBERTa/SVM). So the 'superiority' claim is not currently supported.\n\nThere are also internal inconsistencies. The abstract says 352 training instances; Table 1 says 264. The confusion-matrix prose in Section 4 says LLaMA3 correctly identified 11 of 44 ADHD cases, which corresponds to recall ~0.25, not the 0.87 in Table 2; the ensemble prose says 15 true positives but Table 2 reports recall 0.91. These are not minor typos; they make it impossible to know which numbers to trust.\n\nThe method itself is standard and the direction is reasonable. The limitations section is more honest than most. But as submitted, the reporting errors and the lack of significance testing mean the central claim is not defensible. The paper deserves a serious referee, though, because the underlying question—whether an LLM+transformer+SVM ensemble adds anything over single models for psychiatric text classification—is worth answering. With corrected numbers, a paired test, and ideally released predictions/artifacts, this could become a useful modest contribution. I'd send it to review, but my own verdict would be major revision.\n\nRegards.","headline":"Well-described ensemble application, but the headline superiority claim is statistically unsupported and the manuscript contains inconsistent numbers.","tokens_in":13157,"tokens_out":2748,"would_cite":false,"duration_ms":27214,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An ensemble of LLaMA3, RoBERTa, and SVM lifts ADHD recall to 0.91 on narrative transcripts.","keywords":["large language models","ensemble methods","ADHD detection","narrative transcripts","text classification","RoBERTa","support vector machine","majority voting"],"falsifier":"Run a paired exact test on the 89 test-set predictions comparing the ensemble to the best individual model; if the difference in error counts is not significant at the 0.05 level, the ensemble's claimed superiority is indistinguishable from chance. Alternatively, re-estimate the $F_1$ gap over many random 60/20/20 splits of the 441 transcripts; if the ensemble does not win in a majority of splits, the central claim collapses.","tokens_in":12106,"feed_emoji":"🧠","tokens_out":7804,"duration_ms":70393,"temperature":0.7,"pith_summary":"This paper claims that a simple majority-vote ensemble of three very different classifiers—LLaMA3, a large language model prompted with DSM-5 criteria and used without fine-tuning; RoBERTa, a transformer fine-tuned on clinical narratives; and a support vector machine trained on TF-IDF lexical features plus engineered response metrics—detects ADHD from narrative transcripts more reliably than any of the three alone. On a held-out test set of 89 transcripts from a 441-transcript pediatric sample, the ensemble reaches an $F_1$ score of 0.71 (95% CI 0.60–0.80) and a recall of 0.91, while cutting false positives well below the individual classifiers. The authors present this as evidence that hybrid LLM-plus-traditional-ML architectures are a promising direction for psychiatric text classification.","feed_headline":"Majority vote lifts ADHD detection recall to 0.91","feed_subtitle":"Majority vote over three model types cuts false positives and lifts recall above 0.9","key_machinery":"The load-bearing mechanism is the majority-voting rule $\\hat{y}^{Ens} = 1$ if $\\hat{y}^{LLM} + \\hat{y}^{RoBERTa} + \\hat{y}^{SVM} \\ge 2$, applied to the binary predictions of three deliberately heterogeneous classifiers. LLaMA3 is prompted as a psychiatrist with a DSM-5-based symptom checklist iteratively refined on the development set; RoBERTa is fine-tuned on interviewee-only transcripts with a sliding window for long inputs; the SVM combines 1,000 TF-IDF n-gram features with engineered transcript metrics such as mean interviewee response length, total number of responses, and mean interviewer question length. The premise is that diversity in inductive bias—pretrained discourse knowledge, task-specific fine-tuning, and explicit lexical and structural features—makes the majority vote more robust than any single model.","core_discovery":"The paper's central empirical claim is that majority voting over the three classifiers is the best configuration on the test set: it achieves the highest $F_1$ (0.71) and recall (0.91), with precision 0.59, compared with LLaMA3 ($F_1$ 0.67, recall 0.87), RoBERTa ($F_1$ 0.69, recall 0.87), and SVM ($F_1$ 0.68, recall 0.75). The confusion matrix shows the ensemble produces only 4 false positives versus 11 for SVM while keeping true positives at 15, which the authors interpret as the ensemble filtering out idiosyncratic single-model errors while preserving sensitivity to ADHD-related linguistic cues.","pith_inferences":["The reported ensemble advantage over RoBERTa and SVM is a 0.02–0.03 $F_1$ gap on only 89 test instances; a paired significance test or repeated train/test splits would likely show the gap is not statistically reliable, so the headline claim should be read as suggestive rather than established.","The engineered features (response length, number of responses) are proxies for verbosity and narrative fragmentation; the same ensemble architecture could be tested on other conditions with narrative coherence deficits, where these features would plausibly carry signal.","Because the LLM is used zero-shot, its contribution may be replicated by any capable instruction-following LLM; a natural experiment is to swap in a smaller model to see whether the ensemble's diversity, rather than LLaMA3's raw size, drives the gain.","Weighted or confidence-based voting may outperform majority voting, since LLaMA3's overprediction of the positive class (low precision) suggests its vote should probably be down-weighted."],"forward_implications":["If the result holds, psychiatric text screening can be improved without retraining an LLM: the language model contributes its pretrained understanding through a prompt, while smaller models capture task-specific patterns.","The ensemble's 0.91 recall suggests it could function as a sensitive first-pass screening tool that flags children for full clinical evaluation, accepting lower precision as the cost of not missing cases.","The framework transfers to other narrative-based diagnostic tasks such as depression or autism, where labeled data are scarce, because the classical components train on modest data and the LLM component needs no fine-tuning.","The result indicates that traditional feature-based classifiers still add value on top of LLMs in clinical text, making hybrid models a viable alternative to scaling up a single model."],"supporting_citations":[{"why":"Supplies the LLaMA3 model used without fine-tuning.","marker":"[1]"},{"why":"Supplies the support vector machine classifier and its suitability for high-dimensional feature spaces.","marker":"[8]"},{"why":"Supplies the bootstrap resampling method used to compute 95% confidence intervals for the $F_1$ scores.","marker":"[9]"},{"why":"Supplies the TF-IDF text classification approach adapted for the SVM baseline.","marker":"[11]"},{"why":"Supplies the RoBERTa transformer model that is fine-tuned on the clinical narratives.","marker":"[16]"},{"why":"Supplies prior evidence that majority voting over models improves medical question-answering accuracy, motivating the ensemble strategy.","marker":"[23]"}],"fun_headline_variants":["Majority voting of three models boosts ADHD recall to 0.91","Ensemble of LLM and SVM improves ADHD detection recall","Hybrid LLM-SVM vote raises ADHD recall, cuts false positives","Three-model majority vote lifts ADHD recall to 0.91"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the ensemble outperforms individual models assumes that the observed 0.02–0.03 $F_1$ improvement on an 89-transcript test set is signal rather than noise, given that the bootstrap confidence intervals overlap substantially and no significance test is reported.","fun_headline_variants_meta":{"raw":{"variants":["Majority voting of three models boosts ADHD recall to 0.91","Ensemble of LLM and SVM improves ADHD detection recall","Hybrid LLM-SVM vote raises ADHD recall, cuts false positives","Three-model majority vote lifts ADHD recall to 0.91"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2089,"prompt_tokens":976,"completion_tokens":1113,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":1039}},"tokens_in":592,"tokens_out":1113,"duration_ms":9474,"temperature":1.0,"reasoning_tokens":1039,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:29:19.719917+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a paired exact test on the 89 test-set predictions comparing the ensemble to the best individual model; if the difference in error counts is not significant at the 0.05 level, the ensemble's claimed superiority is indistinguishable from chance. Alternatively, re-estimate the $F_1$ gap over many random 60/20/20 splits of the 441 transcripts; if the ensemble does not win in a majority of splits, the central claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLaMA3 model used without fine-tuning."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the support vector machine classifier and its suitability for high-dimensional feature spaces."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the bootstrap resampling method used to compute 95% confidence intervals for the $F_1$ scores."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the TF-IDF text classification approach adapted for the SVM baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the RoBERTa transformer model that is fine-tuned on the clinical narratives."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies prior evidence that majority voting over models improves medical question-answering accuracy, motivating the ensemble strategy."}],"review_version":1}