{"id":"9fd93e2d-68f2-4e26-87ab-def8da42cad7","arxiv_id":"2411.09413","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A zero-shot LLM pipeline that converts audio-visual behavior into textual scripts and emotion descriptions detects ASD in two-year-olds with 95.24% F1, but the estimate is weakened by threshold tuning on the evaluation data.","lead":"A team built a pipeline that turns videos of young children's social interactions into text scripts, then asks large language models to decide whether the child has autism spectrum disorder. The best version reports 95.24% F1, with the model giving reasons for its decisions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 95.24% F1 is not trustworthy because the emotion threshold (0.175) was selected using the evaluation set; the paper does not describe a nested/validation protocol, so the zero-shot result is likely optimistically biased.","rationale":"I choose evaluation leakage over script fidelity because the central claim is a specific high number. Even if the script transcription preserves all discriminative information (the reader's weakest assumption), a threshold selected on the evaluation set can inflate F1 and make the comparison with [15] invalid. The paper's own Section IV-E is direct evidence: Fig. 11 shows metrics versus threshold, and the chosen value 0.175 is then used for all experiments, without any inner validation. This is not a matter of external consensus; it is an internal protocol flaw. The reader's rationale already mentions threshold tuning, but their weakest_assumption focuses on the transcription mapping. I partially agree with the reader; I consider the evaluation protocol the more decisive issue. My recommended verdict remains CONDITIONAL because the flaw is fixable: if a nested LOOCV or fixed 0.2 threshold reproduces the 95.24% F1, the central claim would be substantially supported. If not, the paper should be rejected or substantially revised. The check proposed above would settle this directly.","tokens_in":19832,"tokens_out":7022,"duration_ms":70345,"concrete_test":"Recompute the SCBU-Agents (w/ emotion) result under a strict nested leave-one-out protocol: for each held-out subject, select the emotion threshold (and any prompt/script variants) using only the remaining 94 subjects, or fix the threshold at 0.2 as specified in Algorithm 1, and report the aggregated F1. If the resulting F1 drops below the supervised baseline (92.20%) or moves by more than 2 points, the headline claim is not supported.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The single most load-bearing concern is that the reported zero-shot F1 (95.24%) is computed under an evaluation protocol that allows test labels to influence the pipeline. Section IV-E ('Emotion thresholds') plots evaluation metrics for different thresholds in Fig. 11 and then states: 'the emotion threshold for all experiments in this study is set to 0.175.' Under the leave-one-out protocol used for Table III, this is a global selection based on the held-out subjects' labels. Algorithm 1 had specified a threshold of 0.2; the switch to 0.175 is justified only by test-set performance. The same pattern appears in the choice to add background/gender descriptions ('we found that adding background and gender descriptions ... can improve diagnostic accuracy') and in the ablation-driven selection of prompt components in Table VIII. For a genuine zero-shot claim, every hyperparameter and prompt choice must be fixed before seeing test labels, or selected by an inner validation loop on training folds. As written, the 95.24% F1 of SCBU-Agents (w/ emotion) is an optimistic, possibly overfit estimate and cannot be compared fairly with the supervised baseline. This is a correctness risk in the central quantitative claim, not a mere generalization caveat.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a zero-shot ASD detection framework, SCBU, that converts audio-visual behavioral recordings into structured text scripts through a behavioral transcription module and a script transcription module, then augments those scripts with emotion descriptions derived from valence dynamics, and finally queries LLMs with domain-specific prompts to obtain an ASD/TD classification and an interpretable rationale. The method is evaluated on a clinical database of 95 children (71 ASD, 24 TD) under leave-one-out cross-validation. The authors report that SCBU-Agents with emotion descriptions reaches an F1-score of 95.24%, surpassing the supervised baseline of 92.20% from Cheng et al. [15], and they provide additional few-shot results and ablations of domain knowledge, human experience, and emotional descriptions.","tokens_in":20083,"tokens_out":2811,"duration_ms":30126,"significance":"If the reported results are valid, the work would be a meaningful contribution to automated, interpretable ASD screening in low-data clinical settings, and the combination of hand-crafted behavior scripts with LLM reasoning is a plausible direction for affective computing. The paper's strengths include the use of a clinically collected multimodal dataset, explicit comparisons against a supervised baseline, a multi-LLM fusion mechanism, and a public release of behavior scripts, which supports reproducibility. However, the central quantitative claim of a 95.24% zero-shot F1-score is not supported by the current evaluation protocol: the emotion threshold and several prompt components are selected using the same leave-one-out test labels, and no confidence intervals, significance tests, or external validation are provided. Because the headline result is an optimistic point estimate with selection on the evaluation set, the paper needs substantial revision before the claimed performance can be credited.","major_comments":[{"comment":"The emotion threshold is selected using the evaluation set itself. Algorithm 1 specifies a threshold of 0.2, but Section IV-E reports scanning Fig. 11 over thresholds and states that \"the emotion threshold for all experiments in this study is set to 0.175.\" Under the leave-one-out protocol used for Table III, this is a global choice informed by the held-out subjects' labels, so the reported improvements from emotion descriptions, including the headline F1 of 95.24% for SCBU-Agents (w/ emotion), are optimistically biased estimates of the true zero-shot performance. To repair this, the authors must either fix all hyperparameters and prompt choices before seeing test labels, or perform model selection in an inner loop on training folds only and report the resulting unbiased estimate.","section":"§IV-E, Fig. 11 and Algorithm 1"},{"comment":"The same selection-on-test-data pattern applies to the prompt components and the background/gender descriptions. Table VIII ablates domain knowledge, human experience, and emotion description based on leave-one-out accuracy, and the text in Section III-B2 states that \"adding background and gender descriptions before the instruction description can improve diagnostic accuracy.\" These are not described as pre-registered choices or as validated on held-out folds. For a genuine zero-shot claim, every component included in the final pipeline must be justified independently of the test labels, or selected with a nested validation split.","section":"§IV-E, Table VIII and Section III-B2"},{"comment":"No confidence intervals, variance estimates, or significance tests are reported, although the evaluation is over only 95 subjects (71 ASD, 24 TD) and the LLM temperature is set to 0.7, which makes the outputs stochastic across runs. The differences among the reported F1 values (e.g., 95.24% vs. 92.20% for the baseline, or 94.04% vs. 91.55% for single LLMs) could be within sampling or stochastic noise. The authors should report repeated-run statistics, exact binomial or bootstrap confidence intervals, and a significance test against the supervised baseline.","section":"Table III and Section IV-B"},{"comment":"The \"zero-shot\" claim is substantially narrower than the framework description suggests. The script transcription module uses hand-coded response descriptions (Table II) and predefined response events that were designed for the same paradigm and dataset as the supervised system [15], so the pipeline encodes substantial clinical prior knowledge before any LLM inference. The paper does not validate that these hand-coded scripts preserve the discriminative information present in the raw audio-visual data or in the features used by [15]; the ablation in Table V only shows that generic video-SALMONN descriptions are much worse, which is not the same as demonstrating that the script mapping is lossless. The authors should clarify what exactly is zero-shot (the LLM's role) and provide an analysis of information retention, for example by comparing the script-derived features against the raw behavioral features from [15].","section":"§III-B and §IV-A"},{"comment":"All experiments are conducted on a single dataset of 95 participants with an imbalanced class ratio (71 ASD, 24 TD), and no external validation is reported. Given the small sample size and the well-known instability of leave-one-out estimates on small imbalanced data, the authors should either validate on an independent ASD audio-visual dataset, report cross-validated confidence intervals, or explicitly state the generalizability limits of the current evidence. This is not a presentation detail; it is necessary to support the paper's claim that the method provides a reliable assisted-diagnosis tool.","section":"§IV-A and §IV-F"}],"minor_comments":[{"comment":"The condition \"dn > 0.2 or dn < 0.2\" appears to be a typo: it should presumably read \"dn > 0.2 or dn < -0.2\", matching the description in Section III-B3.","section":"Algorithm 1"},{"comment":"The caption contains typos: \"Script-Centeric\" should be \"Script-Centric\", and \"anwser\" should be \"answer\".","section":"Fig. 2 caption"},{"comment":"The text refers to \"SCBU-DeepSeekR1-4O (w/ emotion)\" while the corresponding table row is for DeepSeekR1-671B; also, one row reads \"× w\" instead of \"× w/o\". These inconsistencies should be corrected.","section":"Table III and its discussion in Section IV-B"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a clinically important problem and the framework is inventive, but the evaluation protocol currently permits test-set information to influence the headline result through threshold and prompt selection. The authors should be asked to rerun the experiments with a nested validation scheme or pre-registered hyperparameters, and to report uncertainty estimates. If the unbiased F1 remains competitive after this correction, the paper would be a useful contribution to the affective computing community; otherwise, the central claim cannot be accepted as stated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the SCBU paper. The core idea is good: convert audio-visual behavioral data into structured text scripts, enrich them with emotion descriptions, and let LLMs do zero-shot ASD detection with rationales. That is a real step beyond the supervised baseline and beyond LLaVA-ASD, and the multi-LLM voting/agent discussion is a sensible way to hedge against single-model hallucination. The hand-crafted response descriptions in Table II are thoughtful, the emotion-textualization via valence derivatives is a nice touch, and the few-shot results on the 20/75 split give some evidence that the approach can learn from examples. The interpretability examples are genuinely useful for clinicians.\n\nThe soft spot is the evaluation protocol. The stress-test note is right. The emotion threshold was set to 0.175 after looking at Fig. 11, which plots test metrics across thresholds under the same LOOCV used for the headline numbers. Algorithm 1 originally used 0.2; the change is justified only by test-set performance. Similarly, adding background/gender descriptions and assembling the prompt components in Table VIII were choices made after seeing test accuracy. So the 95.24% F1 for SCBU-Agents (w/ emotion) is an optimistic, selected estimate, not an honest zero-shot number. The comparison with the supervised baseline is therefore not fair as reported. There are also no confidence intervals or significance tests, and specificity swings from 37.5% to 91.7% across LLMs, which suggests the results are not stable. The \"first to introduce LLMs for detecting ASD from audio-visual data\" claim is contradicted by LLaVA-ASD [55], which they cite.\n\nThat said, the paper is honest about its limitations, shares the GitHub repo, and the few-shot protocol is cleaner. The fix is straightforward: pre-register the threshold, or select it by an inner validation loop on training folds, and report uncertainty. I would send this to review, but with a clear request to fix the evaluation before acceptance. For researchers working on LLM-based behavioral assessment, it is worth reading; the pipeline idea will likely be influential even if the exact numbers are not.","headline":"The pipeline is genuinely new and worth discussing, but the headline F1 is optimistic: the emotion threshold and prompt components were chosen on the evaluation set.","tokens_in":20603,"tokens_out":2779,"would_cite":false,"duration_ms":25354,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims an LLM pipeline can detect autism spectrum disorder from text-scripted behavior with an F1 of 95.24%, beating the supervised baseline without training on ASD samples.","keywords":["autism spectrum disorder","zero-shot detection","large language models","behavior textualization","emotion textualization","audio-visual behavior analysis","prompt engineering","multi-agent detection"],"falsifier":"Replace the fixed textual response descriptions with plain, faithful transcripts of the raw behavioral logs, keeping the same language model and prompt; if the detection score does not stay near the reported 95%, the hand-built script mapping, not the language model, carries the diagnosis.","tokens_in":19658,"feed_emoji":"🧩","tokens_out":5176,"duration_ms":50250,"temperature":0.7,"pith_summary":"The paper argues that the diagnostic signal for autism spectrum disorder in young children can be written down as a time-stamped behavioral script, and that a general-purpose large language model reading that script can classify the child without ever seeing labeled ASD data. The proposed pipeline converts audio-visual recordings of six structured assessment paradigms into text, adds textual descriptions of emotional dynamics, injects clinical knowledge through prompts, and lets one or several LLMs produce a diagnosis with an explanation. On a clinical dataset of 95 children with an average age of two years, the best multi-LLM version reaches an F1-score of 95.24%, surpassing the supervised behavior-signal baseline of 92.20%. If correct, this means the main barrier to automated ASD screening shifts from scarce labeled training samples to the quality of the script and the prompt.","feed_headline":"Zero-shot LLM autism screening hits 95.24% F1","feed_subtitle":"Videos become text scripts the model reads; emotion cues lift accuracy past a supervised baseline with no ASD training data.","key_machinery":"The central object is the Script-Centric Behavior Understanding (SCBU) pipeline, which converts audio-visual recordings into text scripts that LLMs read. It has four load-bearing parts: a behavior transcription module that extracts positions, gestures, head poses, gaze, speech, and facial expressions from video; a script transcription module with a hand-coded response parser and textualization table that maps observed events into fixed natural-language descriptions per paradigm; an emotion textualization module that detects moments of large valence change and adds a video-LLM description of the emotional segment; and a domain prompt module that injects DSM-V criteria and clinical experience. The multi-LLM variant, SCBU-Agents, adds a re-analysis, collaborative discussion, and final decision stage to reduce individual model hallucination.","core_discovery":"The central claim is that a zero-shot LLM framework can detect ASD from audio-visual behavioral data as accurately as, or better than, a supervised classifier that was trained on the same paradigms. The authors report that their Script-Centric Behavior Understanding (SCBU) method, which turns behavioral logs into structured text scripts and enriches them with emotion-related descriptions, reaches an F1-score of 95.24% when several LLMs discuss and jointly decide, and that even single LLMs approach or exceed the supervised baseline. They also report that the LLMs generate interpretable detection rationales that align with DSM-V diagnostic criteria, and that few-shot learning with a small number of script examples continues to improve performance beyond the supervised baseline.","pith_inferences":["If the central claim holds, a testable extension would run the same prompts on recordings from other clinics or other paradigm sets to see whether the scripts, not the site-specific recording setup, are what generalizes.","Because response latency, response duration, and valence dynamics differ significantly between groups in the raw signals, a direct head-to-head experiment stripping timestamps or emotional descriptions from the scripts would isolate how much diagnostic signal survives the transcription.","If the fixed textual response descriptions implicitly encode the clinicians' expectations, the method may be reusing human expertise at the script-construction stage; a further test would generate scripts automatically from raw logs without the fixed response table and compare detection scores."],"forward_implications":["A general-purpose LLM can match or exceed a supervised behavior-signal classifier on ASD screening using no ASD-labeled training data, so the diagnostic bottleneck shifts from labeled samples to script design and prompt engineering.","Encoding emotional dynamics as text improves detection for most tested LLMs and raises the best multi-LLM system's F1 from 91.78% to 95.24%, indicating that valence-change descriptions carry clinically relevant signal.","Few-shot runs on an open-source model keep improving up to 20 script examples and surpass the supervised baseline, suggesting the method degrades gracefully when labeled data are scarce.","The same framework produces free-text rationales that align with DSM-V criteria, giving clinicians a reason to trust or audit each prediction.","Scripts generated by a generic audio-visual video understanding model instead of the hand-crafted transcriptor collapse specificity to near zero, showing the textualization design, not just LLM size, carries the result."],"supporting_citations":[{"why":"Supplies the clinical audio-visual dataset, the six-paradigm assessment structure, the behavior transcription module, and the supervised baseline that the method must beat.","marker":"[15]"},{"why":"Provides the DSM-V diagnostic criteria used as domain knowledge in the prompt module.","marker":"[2]"},{"why":"The audio-visual LLM used to generate emotional descriptions and as the alternative script generator in the ablation that shows direct video descriptions fail.","marker":"[78]"},{"why":"The multi-agent medical reasoning framework that SCBU-Agents builds on for collaborative detection and final decision.","marker":"[80]"},{"why":"Evidence that children with ASD show narrower valence and arousal ranges, motivating the emotion dynamic point design.","marker":"[82]"},{"why":"Prior work on assessing social skills of children with ASD through language-image pre-training, used as a source of domain experience in the prompts.","marker":"[79]"},{"why":"Instance segmentation model used in the behavior transcription module to locate human body regions.","marker":"[59]"},{"why":"Face recognition model used to distinguish identities of characters in the multi-person localization stage.","marker":"[61]"},{"why":"Body keypoint model used to recognize hand-raising movements in the single-person behavior perception stage.","marker":"[65]"}],"fun_headline_variants":["LLM reads video scripts, detects autism zero-shot at 95% F1","Emotion-enhanced scripts drive zero-shot LLM autism detection to 95.24 F1","Zero-shot LLM autism diagnosis beats supervised, hits 95.24% F1","Videos become text: LLM autism screening hits 95.24% F1 zero-shot","Joint LLM votes on behavior scripts, autism detected at 95.24% F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The hand-coded rules that turn raw behavioral logs into fixed textual responses keep all the clinically relevant information; if those rules drop the subtle timing or emotional cues that separate ASD from typical children, the reported accuracy cannot be reproduced no matter how capable the language model is.","fun_headline_variants_meta":{"raw":{"variants":["LLM reads video scripts, detects autism zero-shot at 95% F1","Emotion-enhanced scripts drive zero-shot LLM autism detection to 95.24 F1","Zero-shot LLM autism diagnosis beats supervised, hits 95.24% F1","Videos become text: LLM autism screening hits 95.24% F1 zero-shot","Joint LLM votes on behavior scripts, autism detected at 95.24% F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00035,"raw_usage":{"total_tokens":1899,"prompt_tokens":919,"completion_tokens":980,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":866}},"tokens_in":535,"tokens_out":980,"duration_ms":8932,"temperature":1.0,"reasoning_tokens":866,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:39:38.015279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Replace the fixed textual response descriptions with plain, faithful transcripts of the raw behavioral logs, keeping the same language model and prompt; if the detection score does not stay near the reported 95%, the hand-built script mapping, not the language model, carries the diagnosis.","supporting_citations":[{"cited_title":"Computer-aided autism spectrum disorder diagnosis with behavior signal processing,","cited_arxiv_id":null,"evidence_quote":"Supplies the clinical audio-visual dataset, the six-paradigm assessment structure, the behavior transcription module, and the supervised baseline that the method must beat."},{"cited_title":"The dsm-5: Classification and criteria changes,","cited_arxiv_id":null,"evidence_quote":"Provides the DSM-V diagnostic criteria used as domain knowledge in the prompt module."},{"cited_title":"Medagents: Large language models as collaborators for zero-shot medical reasoning,","cited_arxiv_id":null,"evidence_quote":"The multi-agent medical reasoning framework that SCBU-Agents builds on for collaborative detection and final decision."},{"cited_title":"Using the circumplex model of affect to study valence and arousal ratings of emotional faces by children and adults with autism spectrum disorders,","cited_arxiv_id":null,"evidence_quote":"Evidence that children with ASD show narrower valence and arousal ranges, motivating the emotion dynamic point design."},{"cited_title":"Assessing the social skills of children with autism spectrum disorder via language-image pre-training models,","cited_arxiv_id":null,"evidence_quote":"Prior work on assessing social skills of children with ASD through language-image pre-training, used as a source of domain experience in the prompts."},{"cited_title":"Solov2: Dynamic and fast instance segmentation,","cited_arxiv_id":null,"evidence_quote":"Instance segmentation model used in the behavior transcription module to locate human body regions."},{"cited_title":"Arcface: Additive angular margin loss for deep face recognition,","cited_arxiv_id":null,"evidence_quote":"Face recognition model used to distinguish identities of characters in the multi-person localization stage."},{"cited_title":"Deep high-resolution representa- tion learning for human pose estimation,","cited_arxiv_id":null,"evidence_quote":"Body keypoint model used to recognize hand-raising movements in the single-person behavior perception stage."}],"review_version":1}