{"id":"87aa5508-30f6-4bbd-8e2b-1a51b020b2d9","arxiv_id":"2502.10145","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"AGCM classifies facial expressions and conversational engagement through learnable, spatially localized human-readable concepts, reporting accuracy at or above black-box baselines on RAF-DB, AffectNet, Aff-Wild2, and NOXI.","lead":"The paper introduces AGCM, an interpretable model for facial expression and engagement recognition that predicts through human-readable concepts such as facial action units, gaze, and vocal pitch, and that also shows where each concept appears on the input.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Explainability claim depends on unvalidated OpenFace AU labels; CAS measures agreement with OpenFace, not with human-coded concepts, so the 'what' and 'where' explanations are not yet shown to be faithful.","rationale":"The reader's weakest assumption identifies exactly this issue: the framework's explainability relies on automatically generated concept annotations that are never validated against human annotation. My independent reading of Sections III.A, IV.C, IV.F, and the loss formulation in Eq. (2) confirms that this is the most load-bearing weakness. The CAS metric cannot rescue the claim because it uses the same OpenFace-derived labels as ground truth, making it a measure of self-consistency rather than semantic fidelity. A concrete test on Aff-Wild2 is feasible because manual AU annotations already exist, and it would directly settle whether the concept supervision is trustworthy. The other concerns raised by the reader, such as missing variance and comparing against published black-box numbers, are important but secondary: they affect the strength of the performance claim, whereas the unvalidated concept labels undermine the interpretability claim itself. The paper may well be correct, and the architecture is coherent, so the appropriate verdict remains CONDITIONAL pending this validation. I am not moving the reader's verdict because the concern, while real, is a missing-evidence problem rather than an observed internal contradiction; it should be resolved by additional experiments rather than by rejection.","tokens_in":24936,"tokens_out":2801,"duration_ms":32774,"concrete_test":"On Aff-Wild2, which provides manual 12-AU annotations, compute per-AU F1 agreement between the OpenFace AU labels used to supervise AGCM and the manual AU labels. Then recompute the Concept Alignment Score (Table V) using the manual AU labels as ground truth instead of the OpenFace-derived labels. If OpenFace per-AU F1 is low (e.g., below 0.6 on key AUs) or if CAS against manual labels is substantially lower than the reported values, the concept explanations cannot be considered faithful human-understandable concepts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that AGCM provides human-interpretable conceptual explanations rests on the premise that the automatically generated concept annotations are accurate. Section IV.C states that OpenFace 2.0 was used to extract 18 AUs as intermediary concepts for RAF-DB and AffectNet, and Section III.A says that spatial supervision maps are generated from an open-source landmark detector, without describing how each AU's ROI is constructed or applying any confidence threshold. The paper never checks these labels against human AU annotation. The Concept Alignment Score in Table V is computed against the same OpenFace-derived labels, so it primarily demonstrates that AGCM can reproduce OpenFace outputs, not that the concepts are semantically correct. Because Eq. (2) trains concept probabilities with BCE and concept attention maps with cosine similarity against these noisy targets, any systematic OpenFace error is inherited by both the 'what' concept scores and the 'where' attention maps. Task accuracy can remain high even when concept explanations are wrong, since the task loss does not distinguish between correct and incorrect concept semantics. Thus the explainability half of the headline claim is not yet supported by the reported evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes the Attention-Guided Concept Model (AGCM), a concept-based neural framework for interpretable affective computing. The visual branch uses a Vision Transformer backbone and per-concept attention maps and activation probabilities, supervised by automatically extracted Action Units, gaze, and head-pose concepts; a second stage freezes the visual branch, adds an acoustic concept generator, and performs sequence-to-sequence fusion for multimodal prediction. The authors evaluate AGCM on RAF-DB, AffectNet-7/8, Aff-Wild2, and the NOXI engagement dataset, reporting accuracy, F1, and CCC scores that match or exceed several black-box baselines while providing 'what' and 'where' explanations as concept probabilities and attention maps.","tokens_in":24968,"tokens_out":5591,"duration_ms":54794,"significance":"AGCM addresses a genuine need in affective computing for inherently interpretable models: it couples concept supervision with spatial attention and demonstrates the combination across multiple benchmarks, with an ablation (Table VI) showing that each component contributes to task accuracy. If the concept explanations are faithful, this would be a practically useful framework for high-stakes applications. However, the core interpretability claim is not yet established because the concept supervision is generated automatically and is never validated against human annotation, and the quantitative concept-alignment metric (CAS, Table V) is computed against those same automatic labels. The reported performance gains over black-box baselines are also presented without variance or significance testing. The architectural contribution is moderate relative to prior concept-embedding models, but the empirical scope is substantial.","major_comments":[{"comment":"The 'what' and 'where' explanations are supervised entirely by automatically generated concept annotations: Section IV.C states that OpenFace 2.0 was used to extract 18 AUs for RAF-DB and AffectNet, and Section III.A says ROI maps are generated from an open-source landmark detector, without specifying how each AU's ROI is constructed or applying any confidence threshold. Since Eq. (2) trains the concept probabilities with BCE and the concept attention maps with cosine similarity against these targets, any systematic OpenFace error is inherited by both the concept scores and the attention maps, while task accuracy can remain high because the task loss does not reward correct concept semantics. The paper never validates the automatic AU labels against human-coded AUs, so the central claim that AGCM provides human-interpretable conceptual explanations is not yet supported. I recommend reporting AU detection accuracy (e.g., against BP4D or DISFA annotations) for the same 18 AUs, describing the ROI construction, and/or adding a user study on explanation plausibility.","section":"§III.A, §IV.C, Eq. (2)"},{"comment":"The Concept Alignment Score is computed against the same OpenFace-derived labels used to train the model, so Table V primarily demonstrates that AGCM can reproduce OpenFace outputs rather than that the predicted concepts align with human semantic judgments. The 'No Concept' baseline already attains 63.6–67.5% CAS, suggesting a high automatic floor for the metric. The assertions in Section IV.F that AGCM has a 'superior ability to learn meaningful and aligned concepts' and in Section V.A that it delivers 'accurate conceptual explanations' are therefore not supported by the reported evidence. I recommend evaluating CAS against human-annotated concept labels, or at least reporting OpenFace's own accuracy on the concept labels used.","section":"§IV.F, Table V"},{"comment":"All reported task results are means over four random seeds with no standard deviations, confidence intervals, or significance tests, and the black-box baselines are drawn from published numbers with different backbones and training protocols. For example, the RAF-DB margin over S2D is only 1.83%, and the Table VI ablation steps from 93.31% to 94.40% with the addition of Concept Map Loss; without variance information it is impossible to judge whether these differences are real. I recommend reporting per-seed results with standard deviations and appropriate statistical tests, and clearly flagging which baseline numbers are copied from other papers rather than reproduced.","section":"§IV.D, Tables II and VI"},{"comment":"The paper claims that the learned probability scores 'quantify the contribution of individual concepts to the predicted label' and that the weighted attention map in Eq. (3) summarizes the explanation. However, the concept probabilities are trained to match the external AU labels, not to equal a measured contribution of each concept to the final prediction, and no causal or perturbation analysis is provided to verify that changing a concept's probability changes the output in the expected direction. The occlusion experiments in Section IV.H demonstrate prediction robustness, but they do not establish that the reported concept scores faithfully reflect the model's decision pathway. A simple intervention test (e.g., zeroing one concept's probability and measuring the change in prediction) would substantially strengthen this claim.","section":"§III.B, §IV.G.1, Eq. (3)"}],"minor_comments":[{"comment":"The acoustic concept construction is underspecified: the text says pitch/loudness variations are 'calculated using their first-order derivatives' and Jitter 'is inherently a percentage,' but no window length, sampling rate, or normalization formula is given, making the audio concepts difficult to reproduce.","section":"§IV.B"},{"comment":"The early-stopping criterion, the exact Vision Transformer variant, and the patch size are not specified, which limits reproducibility of the reported numbers.","section":"§IV.C"},{"comment":"The row labeled 'Feature FC FC V 25.27' appears to have a formatting error: the architecture column contains 'FC FC' and the entry is not aligned with the other rows.","section":"Table IV"},{"comment":"Reference [57] (Zarlenga et al.) has a garbled author list, with given names such as 'B. Pietro' and 'C. Gabriele' treated as family names; the citation should be corrected.","section":"References"},{"comment":"The assumption that 'acoustic concepts are shared across all frames' of a clip is stated without discussion; this is a strong simplification for frame-level explanations, since vocal cues can change within a one-second clip.","section":"§III.C"},{"comment":"The caption uses both 'Noxi' and 'NOXI' for the same dataset; the dataset name should be 'NOXI' consistently throughout.","section":"Figure 11"}],"recommendation":"major_revision","confidential_remarks":"This paper is a solid empirical contribution to interpretable affective computing, but the interpretability claim needs stronger validation before publication. The most important risk is that the concept-evaluation metric (CAS) is computed against the same automatic labels used to supervise the concepts, so the paper's central claim may appear stronger than the evidence supports. I recommend requiring human-labeled concept validation and statistical reporting on the task metrics. The novelty relative to the authors' prior CEM work is incremental, but the multimodal extension and spatial attention are sufficiently distinct for a specialized affective-computing venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a real methods contribution, the what-where multimodal combination is new in the cited literature, and the ablation shows every component earns its place. The weak joint is the explainability claim, which rests on OpenFace auto-labels that are never validated against human coding; the concept alignment numbers only show agreement with OpenFace, not semantic ground truth.\n\nWhat's actually new: AGCM extends the concept-embedding line with multi-scale spatial attention, a landmark-supervised concept map loss, and a late audio-visual fusion stage with the visual branch frozen. That specific package isn't in the cited prior work. The losses are standard, the architecture reads coherently, and the ablation in Table VI (90.47 to 94.40 on RAF-DB) gives real credit to the added components. There is solid implementation detail — backbone, seeds, embedding-size sweeps in the appendix — and the NOXI engagement experiments are a sensible generalization test. The qualitative figures do show what the authors intend: concept-specific heatmaps and probability scores for happy, anger, distraction, and so on.\n\nThe soft spots, in order of softness. First, the 'what' concepts are OpenFace AU extractions and the 'where' maps are landmark-derived ROI maps, with no validation against human AU annotation, no confidence threshold, and no description of how each ROI is constructed. The Concept Alignment Score in Table V is computed against those same OpenFace labels, so it demonstrates that AGCM reproduces OpenFace, not that the concepts are semantically correct. Since Eq. (2) trains concept probabilities with BCE and maps with cosine similarity against these targets, any systematic OpenFace error propagates into both what and where explanations while the task loss can remain high. This is the load-bearing gap in the headline claim. Second, all results are single means over four seeds, with no variance or significance tests, and the black-box SOTA comparisons use published numbers from different pipelines; the stripped baseline already beats several SOTA models, so a large share of the accuracy likely comes from the VGGFace2-pretrained ViT and recipe. Third, occlusion robustness is shown only on selected examples, with no quantitative table. Minor: the text labels the 2024 EU AI Act as 'GDPR in 2024'; that is a factual error and should be fixed.\n\nWho this is for: researchers working on interpretable affective computing, and anyone tracking concept bottleneck methods for faces and audio. It deserves a serious referee, not a desk rejection. I'd send it to review with a strong request: validate the OpenFace/ROI labels on a small human-coded subset, report per-seed spreads, release code and data, and correct the GDPR/AI Act mix-up.","headline":"A genuinely additive concept-based XAI architecture, but the 'human-interpretable' claim needs validation of the OpenFace-supervised concepts before it holds up.","tokens_in":25713,"tokens_out":3479,"would_cite":true,"duration_ms":32592,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Concept-based emotion modeling can match opaque black-box accuracy while producing human-readable explanations.","keywords":["explainable AI","concept-based learning","facial expression recognition","multimodal learning","affective computing","attention mechanism","action units","engagement estimation"],"falsifier":"Compare AGCM's concept probabilities and attention maps against human AU annotations and human-marked facial regions on a held-out set; if confident predictions often cite concepts that human annotators judge absent, the explanation is unfaithful even if task accuracy stays high.","tokens_in":24538,"feed_emoji":"🎯","tokens_out":7808,"duration_ms":69925,"temperature":0.7,"pith_summary":"This paper argues that an emotion-recognition system can be fully interpretable and still match or beat opaque black-box models. It introduces the Attention-Guided Concept Model (AGCM), which predicts emotion labels through intermediate concepts—facial action units (AUs), gaze, head pose, and acoustic cues—and reports both the score of each concept and the facial region where it was observed. On RAF-DB, AffectNet-7, AffectNet-8, Aff-Wild2, and NOXI, the reported numbers are competitive with or above several black-box baselines while the model emits human-readable explanations. If the claim is right, regulated high-stakes affective computing can satisfy transparency requirements without sacrificing accuracy.","feed_headline":"Concept-based emotion AI equals black-box accuracy","feed_subtitle":"AGCM reports 94.4% on RAF-DB, 69.5% on AffectNet-7, and 0.80 CCC on NOXI while naming the cues behind each call.","key_machinery":"The load-bearing object is the Attention-Guided Concept Generator (ACG), a transformer-based module that produces one probability score and one spatial attention map per concept. It combines Multi-scale Spatial Attention (MSA) heads, whose weighted sum forms the concept attention map, and Channel Attended Concept Mapping (CACM), which selects informative feature channels. A three-part loss—cross-entropy task loss, binary cross-entropy concept-probability loss, and cosine-similarity concept-map loss—trains the concept representations and the task prediction together, which the ablation attributes with most of the accuracy gain. In the multimodal stage, frozen visual concept predictions are fused with acoustic concept embeddings through a sequential bottleneck and a transformer label predictor.","core_discovery":"AGCM learns a mapping from input to label through trainable concept embeddings: for each concept, an activated and an inactivated neural embedding are weighted by a learned probability score, then concatenated and passed to a task predictor. The spatial side of the model is supervised by landmark-derived region-of-interest maps, so attention is trained to land where the concept actually appears. The paper's central claim is that this joint design delivers both faithfulness and performance, reporting 94.40% on RAF-DB, 69.45% on AffectNet-7, 65.62% on AffectNet-8, 47.52 F1 on audio-visual Aff-Wild2, and 0.80 CCC on NOXI engagement estimation. The authors interpret these numbers as showing that AGCM matches or exceeds black-box model performance while maintaining conceptual explainability.","pith_inferences":["One testable extension the paper does not run: corrupt the automatic AU labels by flipping a fraction of them and re-measure task accuracy and Concept Alignment Score; if accuracy holds while concept scores drift away from ground truth, the explanations are not faithful even though the framework still appears to work.","A consequence the authors leave implicit: because the concept supervision comes from an external AU detector, the interpretability guarantee is bounded by that detector's accuracy on faces outside its training distribution, so deployment to new demographics should include an audit of concept scores against human judgment.","The what-where explanation design could transfer to medical imaging: replacing AUs with clinician-supplied lesion ROIs would give the same kind of explainable classification in a high-stakes domain where transparency regulations also apply."],"forward_implications":["If AGCM's claim is right, an emotion-recognition system can give regulators and clinicians the what and where behind each prediction without an accuracy penalty.","AGCM's concept supervision can extend to any spatial or temporal signal with concept annotations, so the same architecture transfers from facial expressions to engagement estimation, and in principle to ECG, EEG, or EDA.","The authors' end-to-end joint training is necessary for their result: by-step training, where concept learning and task learning are separated, drops accuracy by roughly 4-12 points on every benchmark they report.","Multimodal fusion consistently beats the visual-only branch (Aff-Wild2 F1 47.52 versus 44.95; NOXI CCC 0.80 versus 0.59), so co-learning across modalities is part of the claimed performance."],"supporting_citations":[{"why":"Supplies the automatically extracted Action Unit scores used as visual concept supervision.","marker":"[8]"},{"why":"Provides the landmark detector whose outputs generate the ROI maps that supervise spatial attention.","marker":"[49]"},{"why":"The concept-embedding model AGCM extends; its CAS and accuracy numbers are the direct comparison baseline.","marker":"[35]"},{"why":"Defines the Concept Alignment Score used to measure whether learned concepts match ground-truth labels.","marker":"[57]"},{"why":"The vision-transformer backbone that converts the face image into patch-level features.","marker":"[19]"},{"why":"The speech encoder used to extract acoustic features for audio concepts in multimodal fusion.","marker":"[27]"},{"why":"The NOXI dataset used to evaluate engagement-estimation generalization on real interactions.","marker":"[15]"},{"why":"RAF-DB benchmark dataset used for visual facial-expression recognition evaluation.","marker":"[34]"},{"why":"AffectNet benchmark dataset used for visual facial-expression recognition evaluation.","marker":"[40]"},{"why":"Aff-Wild2 audio-visual benchmark used for multimodal facial-expression recognition evaluation.","marker":"[30]"}],"fun_headline_variants":["Explaining AI's emotion calls without sacrificing accuracy","Concept-based framework reveals what drives AI emotion predictions","Transparent emotion AI that matches black-box performance","AGCM: multimodal AI that explains its emotion decisions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The explainability claim rests on the premise that the automatically extracted facial action units and landmark-derived attention maps used as supervision are accurate enough to be trusted as the concepts they claim to explain.","fun_headline_variants_meta":{"raw":{"variants":["Explaining AI's emotion calls without sacrificing accuracy","Concept-based framework reveals what drives AI emotion predictions","Transparent emotion AI that matches black-box performance","AGCM: multimodal AI that explains its emotion decisions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000605,"raw_usage":{"total_tokens":2830,"prompt_tokens":965,"completion_tokens":1865,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":1805}},"tokens_in":581,"tokens_out":1865,"duration_ms":13961,"temperature":1.0,"reasoning_tokens":1805,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T19:15:12.805455+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare AGCM's concept probabilities and attention maps against human AU annotations and human-marked facial regions on a held-out set; if confident predictions often cite concepts that human annotators judge absent, the explanation is unfaithful even if task accuracy stays high.","supporting_citations":[{"cited_title":"Baltrusaitis, A","cited_arxiv_id":null,"evidence_quote":"Supplies the automatically extracted Action Unit scores used as visual concept supervision."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the landmark detector whose outputs generate the ROI maps that supervise spatial attention."},{"cited_title":"Li and M","cited_arxiv_id":null,"evidence_quote":"The concept-embedding model AGCM extends; its CAS and accuracy numbers are the direct comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Concept Alignment Score used to measure whether learned concepts match ground-truth labels."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The speech encoder used to extract acoustic features for audio concepts in multimodal fusion."},{"cited_title":"Cafaro, J","cited_arxiv_id":null,"evidence_quote":"The NOXI dataset used to evaluate engagement-estimation generalization on real interactions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"RAF-DB benchmark dataset used for visual facial-expression recognition evaluation."},{"cited_title":"Mollahosseini, B","cited_arxiv_id":null,"evidence_quote":"AffectNet benchmark dataset used for visual facial-expression recognition evaluation."}],"review_version":1}