{"id":"ecf53d17-f8ba-40fc-b86f-12aadd1285ca","arxiv_id":"2501.15864","paper_version":2,"verdict":"REJECT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"FAU-based textual and multimodal explanations improved user understanding and appropriate trust in a facial expression recognition model, but the explanation generator's use of the model's emotion vector as input creates a label leakage confound.","lead":"This paper proposes a new explanation method for facial expression recognition that uses facial action units, and runs a user study comparing it to LIME, SHAP, and saliency maps. The study finds that textual and combined visual-textual FAU explanations improve users' ability to predict the model's output and to calibrate trust, but the explanation system uses the model's emotion prediction as an input, which may leak the answer to users.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The FAU predictor receives the FER model's emotion vector as input (Sec. 3.1), so textual FAU explanations can encode the target label; the Hmp/appropriate-trust advantages may reflect label decoding, not increased understanding.","rationale":"The reader's weakest_assumption is precisely the load-bearing point. The paper's stated replication of Kim and Kim (2019) makes the leak explicit: the FAU head consumes the emotion prediction vector as an input. That architectural choice means the FAU explanation can be a near-paraphrase of the model's predicted label. I do not dispute the reported statistics, the pre-registration, or the balanced 50/50 correct/incorrect test design; the problem is interpretive. Hmp accuracy was meant to measure whether users understand the model, but if FAU text encodes the model's output, the task collapses to a code-breaking exercise. The training phase's explicit pairing of FAU explanations with model predictions only makes the code easier to learn. This also explains why FAU-T and FAU-VT, but not FAU-V, improved Hmp accuracy: text carries the semantic label channel, while the visual contours do not. The appropriate-trust result is inflated for the same reason, since it is defined through Hgtp/Hmp agreement. A plausible counterargument is that the FAU predictor also uses convolutional features, so the FAUs are not fully redundant with the emotion label. But the textual advantage in the data is consistent with the leak, and even partial redundancy undermines the causal interpretation. The proposed ablation of the emotion-vector input would settle the matter. Given that the central claim is an empirical comparison of explanation methods, an internal-validity confound of this kind warrants rejection until the study is rerun with a non-leaky FAU predictor. I therefore keep the reader's REJECT verdict.","tokens_in":12841,"tokens_out":6425,"duration_ms":61621,"concrete_test":"Ablate the emotion-vector input: replace the DNN input with only the final convolution layer, retrain the FAU head on CK+, regenerate textual and visual FAU explanations for the same 28 Aff-Wild2 test images, and rerun the between-subject modality comparison (control, FAU-T, FAU-V, FAU-VT) with the same procedure and pre-registered analyses. If the Hmp and appropriate-trust advantages over control shrink to non-significance, the original effect was driven by label leakage through the 1x8 emotion vector; if they persist, the FAU features carry independent explanatory value. A cheaper computational first step is to measure how accurately the 1x15 FAU vector alone predicts the model's emotion on the test set via cross-validated logistic regression; near-ceiling accuracy would corroborate the leak.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central causal claim—that FAU-based textual and visual-textual explanations improve users' Hmp accuracy and appropriate trust—depends on the FAU explanation being an independent window into the FER model's behavior. Section 3.1 breaks this assumption: the DNN FAU predictor receives a concatenation of the CNN's final convolution layer and its 1x8 emotion prediction vector as input. The FAU outputs are therefore not causally independent of the emotion label; a textual FAU list such as 'Lips Parted, Nose Wrinkled' can carry information about which of the seven emotions the model will output. The training phase reinforced this mapping by explicitly pairing each FAU explanation with the model's prediction for one correct and one incorrect example per emotion. In the test phase, FAU-T and FAU-VT participants could answer the Hmp question by decoding the implied emotion from the FAU text, rather than by understanding the model's decision boundary. Control, LIME, SHAP, and saliency groups had no comparable code. Because Hmp accuracy is the basis for RQ1, and appropriate trust is defined through agreement between Hgtp and Hmp, both headline effects are confounded; the significant Hmp advantage of FAU-T/FAU-VT over control and FAU-VT over FAU-V is exactly what a label-leak account predicts.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents DEFAULTS, an explanation method for facial expression recognition (FER) that generates textual and visual explanations from predicted Facial Action Units (FAUs), and reports a between-subject user study (N=280) over seven conditions: control, LIME, SHAP, saliency maps, FAU-textual (FAU-T), FAU-visual (FAU-V), and FAU-visual-plus-textual (FAU-VT). The headline claims are that FAU-T and FAU-VT explanations significantly improve participants' ability to predict the FER model's outputs (Hmp accuracy; F(3,156)=15.41, p<0.001), with FAU-VT also beating FAU-V, and that all FAU-based modalities yield significantly higher 'appropriate trust' than control (p<0.0001), while LIME, SHAP, and saliency maps show no significant benefit. The paper interprets these results as evidence that FAU-grounded multimodal explanations improve user understanding and trust calibration. The central problem is that the FAU predictor (Section 3.1) receives the FER model's 1x8 emotion prediction vector as a concatenated input, so the textual FAU list can encode the model's predicted emotion; because the training phase explicitly paired each FAU explanation with the model's prediction, participants in the textual conditions may be decoding the model's output rather than demonstrating understanding.","tokens_in":13105,"tokens_out":15453,"duration_ms":130333,"significance":"The contribution is potentially valuable: DEFAULTS grounds explanations in FACS-based facial action units, compares an explanation method across three XAI families in a pre-registered, a-priori powered between-subject design, and releases a participant dataset. The paper is also well organized around two explicit research questions and includes a balance check showing no group differences in human emotion recognition. If the reported effects were causally attributable to genuine understanding of the FER model, the finding that textual, and combined textual-visual, FAU explanations outperform visual-only and standard XAI methods would be an important result for the affective-computing and XAI communities. However, the empirical claims rest on Hmp accuracy and on an 'appropriate trust' metric defined entirely through Hmp; both are directly contaminated by the label-leakage confound described in the summary. The observed pattern—textual conditions improve Hmp while visual-only FAU-V does not—is precisely what a label-decoding account would predict.","major_comments":[{"comment":"The load-bearing flaw is the construction of the FAU explanation pipeline. In Section 3.1, the DNN used to predict FAUs is described as taking 'a concatenation of the final convolution layer of the CNN model (a 1x4032 Fully Connected node) and the 1x8 emotion prediction vector from the CNN' as input. The textual FAU list shown to participants is therefore a function of the model's predicted emotion, not an independent description of the face. In the training phase (Section 3.4), FAU-cohort participants were shown explanations together with the model's prediction for one correct and one incorrect example per emotion, which teaches the mapping from FAU text to model output. In the test phase, Question 2 (Hmp) asked participants to guess the model's prediction, and FAU-T and FAU-VT participants could answer by decoding the emotion implied by the FAU list (e.g., 'Nose Wrinkled' mapped to disgust). The reported Hmp results (F(3,156)=15.41, p<0.001; FAU-T vs. control p=0.0002; FAU-VT vs. control p<0.0001; FAU-VT vs. FAU-V p=0.0001) are exactly the pattern a label-decoding account predicts: the textual conditions improve Hmp accuracy while the visual-only FAU-V condition does not. Since Appropriate Trust (Section 3.5) is defined through agreement or disagreement between Hgtp and Hmp, the appropriate-trust effects (F(3,156)=17.26, p<0.0001 and F(4,195)=11.65, p<0.0001) are confounded in the same way. The central claim that FAU-based explanations improve user understanding and trust calibration therefore does not follow from the data; supporting it would require generating FAU explanations from a predictor that does not condition on the emotion output, or demonstrating empirically that the FAU output carries no information about the model's prediction beyond the image, and then re-running the user study.","section":"Sections 3.1, 3.4, and 4.1"},{"comment":"No means, standard deviations, confidence intervals, or effect sizes are reported for any dependent variable. Sections 4.1 and 4.2 report only F statistics, p-values, and Tukey HSD p-values (e.g., 'F(3,156)=15.41, p<0.001'; 'F(4,195)=11.65, p<0.0001'), and Section 4.1 describes a result as showing 'large significance,' which is not a meaningful property of a p-value. Without group means and effect sizes (e.g., partial eta-squared), readers cannot assess the practical magnitude of the FAU advantages, and the observed effects cannot be compared with the medium-effect-size assumption used in the G*Power sample-size calculation (Section 3.6). The boxplots are helpful but do not substitute for numeric reporting of these headline quantities.","section":"Section 4"},{"comment":"The 'Appropriate Trust' metric is defined purely by whether Hgtp and Hmp coincide on correct (GT=MP) images and differ on incorrect (GT≠MP) images; it does not measure whether users' reliance on the model tracks the model's actual competence. A participant who systematically answers Hmp differently from Hgtp receives credit on all incorrect images, and one who answers Hmp=Hgtp receives credit on all correct images, so the metric rewards these response biases even in the absence of any information leak. In addition, because Hmp is the single confounded quantity identified in Major Comment 1, the appropriate-trust comparisons do not constitute a valid test of trust calibration. The authors should justify this metric against standard calibration measures and, at minimum, report the Hgtp-Hmp agreement rates separately for correct and incorrect trials, along with any analysis that does not depend on Hmp.","section":"Section 3.5"}],"minor_comments":[{"comment":"The hypothesis numbering is inconsistent: Section 3.3 defines H1.1, H1.2, H2.1, and H2.2, but Section 3.5 refers to 'RQ1 H2.1 and H2.2' and 'RQ2 H3.1 and H3.2,' and Section 5.1 says 'This results partially support H2.2' where the relevant hypothesis is H1.2.","section":"Sections 3.3, 3.5, and 5.1"},{"comment":"There is a typo in the introduction: 'as well assystem understanding' should read 'as well as system understanding.'","section":"Section 1"},{"comment":"The exclusion criterion is written as a conjunction: participants were removed only if they completed the survey in less than half of the median time and failed both attention checks. As written, slow participants who failed both attention checks are retained; typically, failing both attention checks should trigger exclusion regardless of completion time, and the authors should clarify and justify the stated conjunction.","section":"Section 3.4"},{"comment":"The FAU predictor was trained on CK+ (posed expressions) but applied to Aff-Wild2 test images (spontaneous, in-the-wild expressions); the paper does not discuss this domain shift or report FAU prediction quality on the test distribution, which matters for judging whether the explanations themselves are veridical.","section":"Section 3.1"},{"comment":"The FAU predictor's accuracy is reported as a 'minimum accuracy of 96.33%' across cross-validation folds; please report the mean and standard deviation across folds instead of only the minimum.","section":"Section 3.1"},{"comment":"The model outputs eight emotion classes including contempt, but participants choose among only seven emotions; the paper does not state what happens on test images for which the model predicts contempt, nor how many such images exist, so the reader cannot assess the resulting noise in Hmp accuracy.","section":"Section 3.4"},{"comment":"The pre-registration links (osf.io/2ez64 and osf.io/35upf) appear as bare URLs in the text; they should be cited with a formal reference including the registration date, and the manuscript should state which reported analyses are confirmatory versus exploratory relative to the pre-registration.","section":"Section 3.3"}],"recommendation":"reject","confidential_remarks":"The manuscript is formatted as an AAMAS 2025 conference paper (ACM Reference Format line in the header) and is being submitted to a journal; the editor may wish to verify the relationship to the conference version to rule out dual-publication issues. On substance: the label-leakage problem is fundamental and cannot be repaired by re-analysis, because the explanation-generation architecture (Section 3.1) and the training protocol (Section 3.4) jointly guarantee that textual FAU explanations carry information about the model's predicted emotion, and the collected user-study data cannot undo that. A new user study with a FAU predictor that does not condition on the emotion vector, ideally including a label-only control condition, would be required before the central claims could be evaluated fairly. The pre-registration, power analysis, and released dataset are genuine assets that would make such a re-run worthwhile."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper has real strengths. It introduces a new explanation artifact - textual and visual FAU-based explanations for FER - and evaluates it against LIME, SHAP, and saliency maps in a pre-registered, between-subject study with 280 participants. The dataset is released, the appropriate-trust metric is a thoughtful way to measure calibration, and the comparison to generic XAI methods is a useful contribution to affective computing. The authors should get credit for that.\n\nBut the central finding does not survive contact with the method section. In Section 3.1, the DNN that predicts FAUs takes a concatenation of the CNN's final convolutional layer and its 1x8 emotion prediction vector as input. That means the textual FAU list is a function of the model's predicted emotion. Participants in the FAU-T and FAU-VT conditions were trained on examples that paired each FAU explanation with the model's prediction, so they could learn to decode the emotion label directly from the FAU text. In the test phase, they could answer the Hmp question by recognizing the implied emotion, not by understanding the model's decision boundary. The observed pattern - FAU-T and FAU-VT improve Hmp accuracy, FAU-V does not - is exactly what label decoding predicts. The appropriate-trust effect is equally suspect, since it is defined via agreement between Hgtp and Hmp. This is a load-bearing flaw, not a minor limitation.\n\nThere are smaller issues: the paper reports only F and p values, not means or effect sizes, which makes the practical magnitude hard to assess. No code is released, only the dataset. The novelty claim relative to the authors' own prior work is a bit generous, but that is secondary.\n\nWho should read this? Researchers designing XAI user studies, especially for affective computing, could learn from the careful experimental structure - and from the confound, which is a good example of why explanation generators must be causally independent of the explained model. The paper deserves peer review because a serious referee could identify the problem and the authors could address it, for instance by retraining the FAU predictor without the emotion vector or using an existing FAU detector. As it stands, the headline conclusions are not supported. My recommendation: send it to reviewers with a clear request to focus on the independence of the explanation pipeline, and reject the current version.","headline":"The DEFAULTS study is a well-run empirical comparison undermined by label leakage: the FAU explanations are generated from the model's emotion output, so the headline gains likely reflect users decoding the label rather than understanding the model.","tokens_in":706,"tokens_out":706,"would_cite":false,"duration_ms":31782,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Explanations built from facial action units help end-users understand when a facial-expression-recognition model will be right or wrong, and calibrate their trust, while LIME, SHAP, and saliency maps do not.","keywords":["facial expression recognition","explainable AI","facial action units","user understanding","appropriate trust","explanation modalities","DEFAULTS","human-model prediction"],"falsifier":"Run the same study with a version of the FAU predictor trained without access to the emotion vector, or with FAU labels randomly shuffled relative to each image; if the gains in Hmp accuracy and appropriate trust disappear, the reported effects came from the leaked emotion label.","tokens_in":12637,"feed_emoji":"😀","tokens_out":12176,"duration_ms":105847,"temperature":0.7,"pith_summary":"The paper claims that explaining a facial expression recognition (FER) model through Facial Action Units (FAUs)—the visible muscle movements behind an expression—lets non-expert users predict the model's output better than no explanation, and calibrates their trust to the model's actual accuracy. The claim is supported by a between-subject user study with 280 participants comparing seven conditions: no explanation, LIME, SHAP, saliency maps, textual FAUs, visual FAUs, and combined textual-visual FAUs. Textual-only and combined FAU explanations significantly improved users' ability to guess the model's emotion prediction, and all FAU modalities produced significantly higher appropriate trust than no explanation. The generic post-hoc methods did not significantly improve understanding or trust. The paper concludes that explanation content grounded in facial physiology matters more than explanation modality alone, with text doing the main work of conveying the model's reasoning.","feed_headline":"Facial action-unit explanations help users read an emotion model","feed_subtitle":"Text and visual muscle-movement explanations raised users' ability to predict model output and calibrated trust; generic XAI did not.","key_machinery":"The central object is the Facial Action Unit (FAU), a muscle activation defined by the Facial Action Coding System, used here as the atomic unit of explanation. DEFAULTS converts the outputs of a deep network FAU predictor into a textual list of activated FAUs and a visual mask that highlights the facial landmarks associated with those FAUs. The architecture matters for interpreting the result: a CNN trained on CK+ produces eight emotion-class scores, and a separate DNN head receives the concatenation of the CNN's final convolutional features and the 1x8 emotion vector to predict 15 boolean FAUs. The evaluation machinery is the two-question survey protocol that yields Human Model Prediction (Hmp) accuracy and the appropriate-trust score.","core_discovery":"The paper's central claim is that an explanation built from Facial Action Units—named muscle movements such as 'inner brow raised' or 'lips parted'—gives end-users a usable handle on what a facial-expression-recognition model will predict. In the study, users who saw textual-only or combined textual-visual FAU explanations predicted the model's emotion label significantly more accurately than controls, and every FAU-based explanation modality produced significantly higher appropriate trust, defined as trusting the model's prediction when it is correct and withholding trust when it is incorrect. LIME, SHAP, and saliency-map explanations produced no significant improvement on either measure. The paper concludes that FAU-based explanations, particularly those with a textual component, are more effective than generic attribution methods at supporting end-user understanding and calibrated trust in FER systems.","pith_inferences":["Because the FAU predictor takes the model's 1x8 emotion vector as one of its inputs, the textual FAU list can encode the predicted emotion; participants may have decoded the label from the explanation, so the Hmp accuracy gain is not necessarily evidence of deeper visual understanding. A version of the FAU predictor trained without the emotion vector would settle this.","The appropriate-trust metric is symmetric under always agreeing and always disagreeing with one's own emotion guess on a 50/50 mix of correct and incorrect model predictions, so the score measures alignment with model correctness relative to that design, not absolute trust calibration; a participant who uniformly distrusts the model can score 50 percent without calibrating anything.","The paper notes that its static landmark contours omit the 'action' that the textual labels convey; animated FAU visualizations are a natural extension that might close the gap between visual and textual modalities."],"forward_implications":["Text is the load-bearing modality: textual-only FAU explanations significantly improved users' ability to predict the model's emotion label, and adding visuals to text improved that ability further.","Visual-only explanations, including FAU-based visual explanations, did not significantly improve understanding of the model, although FAU visual explanations did raise appropriate trust relative to all other visual-only methods and controls.","General-purpose post-hoc explanation methods (LIME, SHAP, saliency maps) did not significantly help users understand or calibrate trust toward a FER model in this setup.","If the finding holds, FAU-based explanations could give non-expert users a way to know when an emotion-recognition system is likely to be wrong, which matters for applications where over-reliance on emotion judgments carries consequences."],"supporting_citations":[{"why":"Supplies the CNN-plus-DNN architecture that predicts both the emotion classes and the FAUs used to build explanations.","marker":"[21]"},{"why":"Defines the Facial Action Coding System whose action units give the explanations their formal grounding.","marker":"[13]"},{"why":"Baseline perturbation-based explanation method included as a comparison condition in the user study.","marker":"[39]"},{"why":"Baseline perturbation-based explanation method included as a comparison condition.","marker":"[28]"},{"why":"Baseline gradient-based explanation method included as a comparison condition.","marker":"[43]"},{"why":"Provides the CK+ dataset on which the emotion CNN and the FAU predictor were trained and their accuracies reported.","marker":"[27]"},{"why":"Provides the Aff-Wild2 in-the-wild images used in the testing phase.","marker":"[23]"},{"why":"Supplies the state-of-the-art FER accuracy estimate used to set the 50 percent correct / 50 percent incorrect balance of test images.","marker":"[40]"},{"why":"Provides the facial-landmark mapping used to draw the visual FAU explanations.","marker":"[36]"},{"why":"Prior study that motivates comparing explanation types and modalities for FER systems.","marker":"[33]"}],"fun_headline_variants":["Text-based action units improve understanding of emotion AI","Action-unit explanations calibrate trust in emotion recognition","FAU text and visuals improve user trust in emotion models","Muscle-movement cues outperform generic XAI for emotion recognition","Text-based action units help users read emotion AI output"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the FAU explanation is an independent account of the face, but the FAU predictor receives the model's emotion-prediction vector as an extra input, so the explanation may leak which emotion the model predicted.","fun_headline_variants_meta":{"raw":{"variants":["Text-based action units improve understanding of emotion AI","Action-unit explanations calibrate trust in emotion recognition","FAU text and visuals improve user trust in emotion models","Muscle-movement cues outperform generic XAI for emotion recognition","Text-based action units help users read emotion AI output"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001387,"raw_usage":{"total_tokens":5557,"prompt_tokens":833,"completion_tokens":4724,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":449,"completion_tokens_details":{"reasoning_tokens":4647}},"tokens_in":449,"tokens_out":4724,"duration_ms":32895,"temperature":1.0,"reasoning_tokens":4647,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:50:05.435906+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same study with a version of the FAU predictor trained without access to the emotion vector, or with FAU labels randomly shuffled relative to each image; if the gains in Hmp accuracy and appropriate trust disappear, the reported effects came from the leaked emotion label.","supporting_citations":[{"cited_title":"I think you need help! Here’s why","cited_arxiv_id":null,"evidence_quote":"Prior study that motivates comparing explanation types and modalities for FER systems."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the CNN-plus-DNN architecture that predicts both the emotion classes and the FAUs used to build explanations."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Facial Action Coding System whose action units give the explanations their formal grounding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CK+ dataset on which the emotion CNN and the FAU predictor were trained and their accuracies reported."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the state-of-the-art FER accuracy estimate used to set the 50 percent correct / 50 percent incorrect balance of test images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the facial-landmark mapping used to draw the visual FAU explanations."}],"review_version":1}