{"id":"9f4f199d-ae99-4f63-81ae-8e29dd95b659","arxiv_id":"2411.13797","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new FACS-annotated dataset of children's facial action units with and without autism, including atypicality ratings and baseline AU detection results.","lead":"Researchers created a new dataset of facial action unit annotations from children with and without autism, labeling about 130,000 frames by FACS experts. The resource is meant to help explain why autistic children's facial expressions look atypical and to support future screening tools.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The static ASD/TD AU comparison in §IV.B is circular because emotion labels are filtered using the same AUs that are then tested as outcomes, so the headline group differences need independent emotion labels to verify.","rationale":"The reader's weakest assumption correctly identifies the circularity in §IV.B: emotion labels are filtered by AU rules that overlap with the AUs compared in Table IV. This is the most load-bearing concern because the static group-difference analysis is the paper's second listed contribution and a central piece of evidence that the dataset reveals meaningful ASD/TD differences. If the emotion labels are biased—and the paper's own evaluation of OpenFace and Py-feat indicates that adult-trained models have poor performance on children with ASD—then the filtering step can create or distort group differences in the same AUs that are subsequently tested. My concrete test asks for the minimal reanalysis that would settle this: use emotion labels that do not depend on the outcome AUs, and repeat the comparison at the subject level rather than the frame level. The subject-level component is important because the current Mann-Whitney tests treat every frame as independent, which inflates significance even if the label construction were unbiased. I do not think this requires rejecting the paper: the dataset, the FACS-expert annotations, the atypicality ratings, and the baseline AU detection results are independent contributions that retain value. However, the static analysis must be redone or carefully qualified before the headline claim about group differences can be accepted. The reader's CONDITIONAL verdict is therefore appropriate, and I do not propose changing it.","tokens_in":29730,"tokens_out":6855,"duration_ms":70150,"concrete_test":"Reproduce §IV.B twice: first with the AU-filtered pseudo-labels as in the paper, then with emotion labels that are independent of the tested AUs—for example, taken directly from the soft-vote argmax without the FACS filter, or from independent FACS-expert emotion judgments on a random subset of frames. Recompute the Mann-Whitney tests using per-subject AU rates, or a mixed-effects logistic model with subject random effects, and report which AUs survive. If the Table IV differences, especially AU12 for happy, AU1/AU2 for surprise, and AU1/AU4/AU17 for sad, disappear or change sign, the static result is an artifact of label construction; if they persist under both independent labels and subject-level analysis, the circularity concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result—significant AU/AD differences between ASD and TD children when displaying the same emotional expressions—rests entirely on the frame-selection procedure in §IV.B. Emotion pseudo-labels come from three adult-trained FER models (POSTER++, EAC, DDAMFN++) with a 0.6 soft-vote threshold, and are then filtered by FACS rules that use the same AUs that later appear as dependent variables: happy requires AU12; surprise requires AU1, AU2, AU5, AU25, or AU26/27; sad requires AU1, AU4, AU6, AU15, or AU17. The Mann-Whitney tests in Table IV then report group differences on, for example, AU12 for happy and AU1/AU2 for surprise/sad. Because the selection condition and the outcome share variables, the two groups are not formed independently of the response: any bias in the pseudo-labeling is propagated directly into the comparison. The paper itself shows in §III.D that adult-trained AU tools perform poorly on these children, which makes such bias a concrete risk rather than a hypothetical one. In the extreme, if every \"happy\" frame has AU12 by construction, a significant AU12 effect cannot have the usual interpretation. Furthermore, the analysis treats roughly 131,000 frames as independent observations from 98 children, ignoring within-subject correlation, so the extreme Z-values do not reflect subject-level evidence. These issues do not invalidate the dataset or its annotations, but they do undermine contribution (2), the static group-difference claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Hugging Rain Man (HRM), a dataset of facial action units (AUs) and action descriptors (ADs) from 66 children with ASD and 32 typically developing (TD) children, totaling about 130,000 frames annotated frame-by-frame by FACS experts, plus atypicality ratings for expression segments. The authors report three main results: (1) a static analysis claiming significant AU/AD differences between ASD and TD groups when displaying the same emotional expressions (happiness, surprise, sadness); (2) a temporal regression baseline that predicts perceived atypicality from AU/AD sequences; and (3) AU detection baselines for several supervised and self-supervised models on the HRM dataset. The paper also documents that existing adult-trained AU tools (OpenFace, Py-feat) have low agreement with expert annotations on these children, motivating the need for a child-specific dataset.","tokens_in":30215,"tokens_out":3527,"duration_ms":101773,"significance":"If the dataset is released and validated, HRM would be a valuable new resource for studying atypical facial expressions in ASD, as it is, to my knowledge, the first dedicated FACS-expert-annotated AU dataset for children with ASD that also includes a TD control group. The careful expert-annotation protocol (inter-rater ICC 0.66–0.97, average 0.859) and the quantitative demonstration that adult-trained tools perform poorly on this population (average F1 around 0.3) are concrete strengths. The benchmark results for AU detection provide a useful reference for future work. The central claim about static ASD/TD differences, however, rests on a frame-selection procedure whose validity is not established, and the statistical testing ignores within-subject correlation; these issues are load-bearing for contribution (2) and require reanalysis.","major_comments":[{"comment":"The static ASD/TD comparison is circular. Emotion pseudo-labels are obtained from three adult-trained FER models with a 0.6 confidence threshold and then refined by AU activation constraints: happy requires AU12; surprise requires AU1, AU2, AU5, AU25, or AU26/27; sad requires AU1, AU4, AU6, AU15, or AU17. Table IV then reports group differences on those very AUs, e.g., AU12 for happy (Z=-2.236, P=0.025), AU1/AU2 for surprise (Z=-4.248 and -3.789), and AU1/AU4/AU17 for sad. Because the selection condition shares variables with the outcome, a significant group difference on a selection AU cannot receive the usual interpretation; in the extreme, if every 'happy' frame has AU12 by construction, a difference in AU12 frequency is partly a description of the selection rule rather than of the children's expressions. The paper should re-run the analysis with emotion labels independent of the outcome AUs (e.g., human emotion labels, task condition, or a held-out subset) or clearly reframe the analysis as a comparison of AU-constrained expression categories rather than 'same emotional conditions.'","section":"Section IV.B, Table IV"},{"comment":"The statistical tests treat each of the roughly 131,000 frames as independent observations, although frames are nested within expression segments, which are nested within 98 participants. The Mann-Whitney U tests therefore vastly overstate the effective sample size, and the extreme Z-values (e.g., Z=-47.485) reflect the number of frames per participant more than a subject-level effect. The authors should use subject-level summaries (e.g., per-participant AU occurrence rates per emotion) or mixed-effects models with participant and segment as random effects, and report effect sizes and confidence intervals rather than only P-values.","section":"Section IV.B, Table IV"},{"comment":"The pseudo-labeling step inherits a domain-shift risk that the authors themselves document for AU tools: Section III.D shows that adult-trained AU detectors (OpenFace and Py-feat) agree poorly with expert annotators on children with ASD (average F1 around 0.3). The three FER models used for emotion pseudo-labels are likewise adult-trained, and no validation of their emotion outputs on children is provided. If the FER models are differentially biased by group (e.g., they misclassify ASD expressions as sad or neutral more often), the 'same emotional expression' condition is violated, and both the selection and the outcome are confounded. At minimum, the paper should report agreement on a subset of the HRM frames between the FER pseudo-labels and human emotion labels, and show that the 0.6 threshold is not group-dependent.","section":"Section III.D and Section IV.B"},{"comment":"The two groups differ in age (ASD 5.29±2.30 vs. TD 4.37±1.62 years) and sex distribution (80% male in ASD vs. 56% male in TD). Since AU occurrence is known to vary with age and sex, the unadjusted Mann-Whitney comparisons in Table IV may be confounded. The authors should adjust for age and sex (e.g., regression with these covariates or a matched/subsampled analysis) before claiming that the observed AU/AD differences are attributable to ASD status.","section":"Table II and Table IV"}],"minor_comments":[{"comment":"The abstract says 'approximately 130,000 frames' while Table I reports 131,758 frames; please use one consistent number throughout.","section":"Abstract and Table I"},{"comment":"The abstract mentions '22 AUs and 10 ADs,' but Table III lists 33 AU/AD codes (including AU2X and AD19, AD32). Please clarify the relationship among 22 AUs, 10 ADs, and the 33 annotated categories.","section":"Abstract and Table III"},{"comment":"The claim that the atypicality-regression results are 'acceptable' would be more convincing with a trivial baseline (e.g., predicting the mean rating for every segment) or a chance-level reference, so the reader can gauge the added value of the AU/AD features.","section":"Section V.A.4"},{"comment":"The discussion reports a Fleiss' Kappa of 0.565, but Section III.E reports Kappa = 0.503; please correct the inconsistency.","section":"Section III.E and Section VI"},{"comment":"The temporal window size of 15 frames is described as empirically chosen; a small sensitivity analysis (e.g., windows of 9, 15, 21 frames) would strengthen the atypicality-regression results.","section":"Section V.A.3"},{"comment":"The claim that HRM is the 'first dedicated dataset of facial action units for children with ASD' should explicitly acknowledge prior expert-annotated AU studies in ASD (e.g., Weiss et al. [45], which the paper cites) to make the novelty claim precise and verifiable.","section":"Section II.B and Section III.E"}],"recommendation":"major_revision","confidential_remarks":"The dataset itself is a solid contribution, and the annotation quality metrics are good. The main barrier to acceptance is the static-difference claim (contribution 2), which suffers from a real circularity: the frame-selection filter uses the same AUs that are later tested as outcomes. This is fixable in principle by using independent emotion labels and accounting for subject-level clustering, but it requires re-analysis, not just rewording. If the authors cannot obtain independent emotion labels, they should either drop the 'same emotional expression' framing or present the analysis as descriptive, with the circularity explicitly stated as a limitation. The temporal atypicality regression and AU detection benchmarks are less affected and could be reported in the same paper after the re-analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good to see this paper. The Hugging Rain Man dataset is the real contribution here, and it's a good one: first FACS-expert-annotated AU dataset for children with ASD, with TD controls, 22 AUs, 10 ADs, atypicality ratings, ~130k frames from 98 kids. They also show convincingly that OpenFace and Py-feat get average F1 around 0.3 on these children, which justifies the expensive manual annotation. The AU detection baselines and the atypicality regression are useful starting points for the community.\n\nThe soft spot is the static group-difference analysis in Section IV.B. The emotion pseudo-labels come from adult-trained FER models and are then filtered by AU activation rules that use the same AUs the paper later tests for group differences. Happy requires AU12; surprise requires AU1/2/5/25/26/27; sad requires AU1/4/6/15/17. That's a textbook circularity: the selection condition and the outcome share variables. The paper itself shows these adult-trained tools perform poorly on this population, so this isn't hypothetical bias; it's probable bias. On top of that, the analysis treats roughly 131,000 frames as independent, ignoring that they come from only 98 children. The Z-scores in Table IV are wildly inflated by frame-level pseudoreplication. This doesn't sink the dataset, but it does sink contribution (2) as currently presented.\n\nThe atypicality regression (contribution 3) is not affected by this issue; it uses the manually annotated AU sequences, not the pseudo-labels. The atypicality ratings themselves show moderate agreement (Kappa 0.503, ICC 0.761), which they disclose and discuss.\n\nThe newness claim is fair: I don't know of another manual FACS AU dataset for ASD. The authors cite the relevant prior work (Weiss et al., DASD, etc.) and are appropriately careful about limitations, including the temporary unavailability of the facial images.\n\nWho should read it: anyone building AU detectors for children, and researchers studying atypical facial expressions in ASD. The dataset deserves to exist in the literature even if the static analysis needs rework. For the journal process, I'd send it to referees but ask for a major revision: re-do the static comparison with independent emotion labels (or manually verified ones), and use subject-level or mixed-effects analysis rather than frame-level tests. If the images stay private, at least the annotations and features are out there, which is still valuable.\n\nVerdict: engage with it, but referee with a clear request to fix the circularity before the claims about AU differences are taken seriously.","headline":"The dataset is the real contribution; the static ASD/TD AU comparison is circular and needs reanalysis before its claims are trusted.","tokens_in":30618,"tokens_out":2307,"would_cite":true,"duration_ms":22039,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Hugging Rain Man provides about 130,000 expert-annotated frames of 22 facial action units and 10 action descriptors in children with ASD and TD, and reports significant group differences in AU patterns for happy, surprised, and sad…","keywords":["autism spectrum disorder","facial action units","FACS annotation","atypical facial expressions","children","AU detection","temporal regression","dataset"],"falsifier":"Recompute the happy, surprise, and sadness AU comparisons using emotion labels provided by human raters or by a model that does not filter on AUs; if the significant AU/AD differences between ASD and TD disappear or reverse, the paper's central group-difference claim is an artifact of the emotion filtering rules.","tokens_in":29547,"feed_emoji":"🎭","tokens_out":5792,"duration_ms":54687,"temperature":0.7,"pith_summary":"This paper builds and analyzes a new dataset, Hugging Rain Man (HRM), of roughly 130,000 frames of facial expressions from 66 children with autism spectrum disorder (ASD) and 32 typically developing (TD) children, with every frame annotated by FACS experts for 22 action units (AUs) and 10 action descriptors (ADs), plus segment-level atypicality ratings. The paper's central claim is that these expert labels reveal measurable, objective differences in how children with ASD move their faces: for happy, surprised, and sad expressions, the ASD group differs significantly from the TD group on multiple AUs and ADs, and uses more complex and more diverse AU combinations. It then claims that a temporal regression model can predict human judges' atypicality ratings from AU sequences, bridging subjective perception and objective facial features. This matters because such objective markers could support earlier, more consistent ASD screening and because child-specific AU data may improve automatic facial expression analysis where adult-trained tools currently perform poorly. The paper also provides AU detection baselines and releases labels, features, and pretrained model weights for the community.","feed_headline":"130,000 annotated frames map atypical faces in autism","feed_subtitle":"FACS-expert labels show children with ASD use more complex, varied muscle patterns for the same emotions.","key_machinery":"The load-bearing object is the frame-level FACS annotation itself: each frame carries presence labels for 22 AUs and 10 ADs, with direction codes for left, right, upper, and lower regions, so that asymmetry and rare descriptors are recorded. The annotation was done by one FACS expert and checked by a second on a subset, with inter-expert ICC values ranging from 0.66 to 0.97. On top of these labels the paper builds emotion pseudo-labels by soft-voting three facial expression recognition models and then filtering with AU-emotion rules (e.g., happy requires AU12, surprise requires at least one of AU1, AU2, AU5, AU25, or AU26/27), and then applies Mann-Whitney $U$ tests per AU and a count of distinct AU combination types at each complexity level for the group comparisons; for dynamics, a sliding window of 15 frames converts AU sequences into input for GRU and BiLSTM regressors that predict segment-level atypicality ratings.","core_discovery":"The paper introduces HRM as, to the authors' knowledge, the first dedicated facial action unit dataset for children with ASD, and it claims that the dataset supports three findings: (1) in static frames labeled as the same emotion, children with ASD and TD differ significantly in several AUs/ADs, with the ASD group showing greater AU combination complexity and a wider variety of AU combinations; (2) these group differences align with prior reports of reduced AU6/AU12 activity in happy expressions and atypical AU activation in ASD; and (3) temporal AU/AD sequences fed to GRU or BiLSTM regression models predict averaged human atypicality ratings with MAE around 0.30 on a 1–5 scale, demonstrating that perceived atypicality correlates with objective facial movement features. The discovery is a new application: bringing fine-grained, expert FACS annotation plus subjective atypicality ratings to ASD expression research.","pith_inferences":["The emotion pseudo-labels are filtered by rules involving the same AUs later tested for group differences, so a cleaner test would use human emotion labels or unfiltered model labels to confirm the reported AU differences are not artifacts of the selection rule.","Because the atypicality judges were partly blind but the FACS experts were not, part of the rated atypicality could reflect group identity or appearance rather than expression dynamics; showing the model predicts atypicality within the ASD group alone would isolate expression-specific signal.","The AU combination complexity and diversity measures could be turned into a single per-child score (for instance, entropy over observed AU sets) and tested prospectively as a screening index against independent diagnostic labels.","If the temporal regression generalizes across the 10 random cross-validation splits, a minimal clinically useful next step is to test it on a new cohort with standardized emotion-elicitation tasks to separate posed from spontaneous expression effects."],"forward_implications":["If the group differences replicate, AU and AD activation patterns from expert FACS coding become candidate objective biomarkers for early ASD screening from facial behavior.","Child-specific AU detection models trained or fine-tuned on HRM should outperform adult-trained tools, whose average F1 agreement with experts is around 0.3 on this population.","Temporal AU sequences can serve as input for automated atypicality scoring, offering a quantitative complement to subjective ratings.","Future datasets can use the HRM annotation scheme (direction-coded AUs, ADs, atypicality ratings) as a template for other clinical populations with atypical facial expressions.","The benchmark results suggest self-supervised pretraining is a promising route for AU detection when expert labels are scarce."],"supporting_citations":[{"why":"Supplies the FACS coding standard and the AU definitions that the expert annotations and the emotion-filtering rules rely on.","marker":"[10]"},{"why":"Existing child TD dataset that offers only basic emotion labels, contrasted with HRM's AU/AD labels in the dataset gap argument.","marker":"[26]"},{"why":"The main existing ASD facial dataset, lacking raw video and expert AU annotations, which HRM is designed to fill.","marker":"[33]"},{"why":"Adult spontaneous AU dataset used to train the open-source AU tools whose low agreement with experts motivates the new dataset.","marker":"[22]"},{"why":"Adult-trained AU detection toolkit used in the reliability comparison showing poor agreement with expert annotations on children with ASD.","marker":"[19]"},{"why":"Another adult-trained AU detection tool used in the same reliability comparison and found to have low F1 scores on this population.","marker":"[20]"},{"why":"Prior study finding reduced AU6 and AU12 activity in happy expressions of ASD, which the static analysis extends to a larger dataset.","marker":"[25]"},{"why":"Prior computational study of expressive facial dynamics in children with ASD that HRM extends with expert AU annotations and atypicality ratings.","marker":"[34]"}],"fun_headline_variants":["New facial action unit dataset maps atypical expressions in ASD","130K expert-annotated frames reveal varied facial actions in autism","FACS data links diverse facial muscle patterns to autism atypicality","Temporal patterns of facial actions predict atypicality in autism","First dedicated dataset of facial action units in autism children"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The static group comparison assumes the emotion labels assigned to each frame are accurate, but those labels come from adult-trained models and are then filtered by rules using the very AUs later compared between groups, so a bias in the filtering could produce the reported differences.","fun_headline_variants_meta":{"raw":{"variants":["New facial action unit dataset maps atypical expressions in ASD","130K expert-annotated frames reveal varied facial actions in autism","FACS data links diverse facial muscle patterns to autism atypicality","Temporal patterns of facial actions predict atypicality in autism","First dedicated dataset of facial action units in autism children"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000901,"raw_usage":{"total_tokens":3893,"prompt_tokens":973,"completion_tokens":2920,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":589,"completion_tokens_details":{"reasoning_tokens":2838}},"tokens_in":589,"tokens_out":2920,"duration_ms":17244,"temperature":1.0,"reasoning_tokens":2838,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:51:37.347683+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the happy, surprise, and sadness AU comparisons using emotion labels provided by human raters or by a model that does not filter on AUs; if the significant AU/AD differences between ASD and TD disappear or reverse, the paper's central group-difference claim is an artifact of the emotion filtering rules.","supporting_citations":[{"cited_title":"An advanced deep learning framework for video-based diagnosis of asd,","cited_arxiv_id":null,"evidence_quote":"The main existing ASD facial dataset, lacking raw video and expert AU annotations, which HRM is designed to fill."},{"cited_title":"Pilot Study to Discover Candidate Biomarkers for Autism based on Perception and Production of Facial Expressions","cited_arxiv_id":"2404.16040","evidence_quote":"Prior study finding reduced AU6 and AU12 activity in happy expressions of ASD, which the static analysis extends to a larger dataset."},{"cited_title":"Computer vision in autism spectrum disorder research: a systematic review of published studies from 2009 to 2019,","cited_arxiv_id":null,"evidence_quote":"Prior computational study of expressive facial dynamics in children with ASD that HRM extends with expert AU annotations and atypicality ratings."}],"review_version":1}