{"id":"91509477-a7f7-4c30-a94a-6c39628f2946","arxiv_id":"2509.09593","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On EXPRESS, best LLMs reach only ~31-36% lexical accuracy and ~39-44% basic-emotion-vector accuracy in predicting self-disclosed emotions, with chain-of-thought prompting hurting.","lead":"A new benchmark, EXPRESS, masks emotion words that Reddit users wrote about their own experiences in 33,679 long posts, and tests 14 language models at predicting those words. Even the best models, such as GPT-4o, rarely reproduce the exact emotion or even the right emotion family, and chain-of-thought prompting makes predictions worse.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Self-disclosed labels are treated as uniquely correct, but the paper's own expert eval finds LLM predictions preferred as often as self-disclosed ones (43.7% vs 40.0%, kappa=0.21); low alignment with self-disclosed words is not established as an emotional blind spot.","rationale":"The reader's weakest assumption identifies exactly the load-bearing issue: the evaluation treats self-disclosed emotion words as ground truth, while the paper's own expert evaluation shows that this ground truth is not uniquely correct and that LLM predictions are preferred at similar rates. I agree with the CONDITIONAL verdict because the benchmark itself and the low alignment numbers are real contributions, but the interpretive claim of 'emotional blind spots' needs to be conditioned on the validity of self-disclosed labels as uniquely correct. The concrete test would settle whether human annotators can actually predict the self-disclosed labels better than the LLM; if they cannot, low AccV is not evidence of a specifically emotional deficit. This does not change the reader's verdict, so the recommended outcome is UNCHANGED.","tokens_in":26820,"tokens_out":5077,"duration_ms":63255,"concrete_test":"Run a pre-registered human prediction study on a new stratified sample of ~300 EXPRESS posts. Mask the same emotion words, and ask both naive readers and the expert panel to (a) predict the masked emotion word(s) directly and (b) choose blind between the author's self-disclosed emotion and GPT-4o's few-shot-nearest prediction, with a 'both/neither' option. Compute human AccV against the self-disclosed labels and compare with GPT-4o's AccV; also compute expert preference rates with 95% confidence intervals and Fleiss kappa. If human AccV is no higher than GPT-4o's, or if the self-disclosed label is not preferred over the LLM prediction by a significant margin, the benchmark's ground truth is not a uniquely valid target for 'emotional accuracy' and the central framing should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that LLMs have emotional blind spots because they fail to align with human self-disclosed emotions (e.g., zero-shot GPT-4o AccV=0.388). This interpretation holds only if the self-disclosed emotion is the uniquely correct ground truth for what the writer was feeling. EXPRESS deliberately uses self-disclosed labels for ecological validity, but the paper's own human study directly tests that assumption and does not confirm it. For 213 posts where GPT-4o's prediction differed at the vector level, three emotion experts (majority vote) preferred the LLM emotion 97 times (43.7%) and the self-disclosed emotion 89 times (40.0%), with both/neither/no majority for the remainder and Fleiss kappa = 0.21. The 3.7-point gap is untested and within sampling error, and the low inter-rater agreement indicates that the text does not determine a unique correct emotion. The Limitations section itself concedes that there may be cases where a term cannot be predicted by a human without more context. Consequently, 'alignment with self-disclosed emotions' may measure idiosyncratic word choice rather than emotional accuracy. The abstract's 'challenging' claim is true by construction, but the 'emotional blind spots' framing is not supported by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EXPRESS, a benchmark of 33k+ Reddit posts in which authors' self-disclosed emotion words are masked, along with a fine-grained evaluation framework that decomposes predicted and actual emotion terms into 10-dimensional EmoLex-based vectors. Fourteen language models are evaluated in zero-shot, few-shot (random and nearest-example), and chain-of-thought settings. The main empirical finding is low lexical and vector-level alignment with self-disclosed emotions (e.g., GPT-4o reaches AccL=0.313 and AccV=0.388 in zero-shot), with few-shot prompting helping and CoT hurting. A human expert study on 213 posts finds that experts slightly prefer the LLM's predicted emotion over the self-disclosed one (43.7% vs. 40.0%, Fleiss' kappa=0.21).","tokens_in":27138,"tokens_out":5325,"duration_ms":64918,"significance":"EXPRESS is a potentially valuable resource: it is large, uses naturally occurring long-form self-disclosures, covers 251 fine-grained emotion labels, and the paper provides a systematic 14-model comparison with several prompting protocols and releases code and data. The few-shot vs. CoT results are informative for practical emotion-recognition use. However, the title and abstract overstate the conclusion: the quantitative results measure divergence from self-disclosed wording, and the paper's own human evaluation shows that the self-disclosed label is not a uniquely correct ground truth. If reframed as an alignment benchmark rather than evidence of 'emotional blind spots,' the contribution is solid; as it stands, the central interpretive claim is not supported by the evidence in the manuscript.","major_comments":[{"comment":"The paper's central claim—that LMs have 'emotional blind spots' because they fail to align with self-disclosed emotions—depends on treating the self-disclosed emotion word as the uniquely correct label. The paper's own expert evaluation undercuts this assumption: on the 213 posts where the LLM vector differed, experts chose the LLM emotion in 97 cases (43.7%) versus the self-disclosed emotion in 89 cases (40.0%), with Fleiss' kappa=0.21. The Limitations section further concedes that 'there may be cases where a term cannot be predicted by a human (at least without more context provided).' The 3.7-point gap is not tested for significance, and the low inter-rater agreement indicates that the text does not determine a unique emotion label. Consequently, low AccL/AccV is evidence of divergence from self-disclosed labels, not necessarily of a deficit in emotion recognition. The framing should","section":"Human Evaluation of Predicted and Actual Emotions (Results; Limitations)"},{"comment":"No human baseline is reported for the exact mask-prediction task. The random baselines (AccL=0.001, AccV=0.322) show the task is hard relative to chance, but the abstract's claim that 'accurately predicting emotions that align with human self-disclosed emotions remains challenging' requires a human upper bound. The expert-preference study is a different task: experts choose between two given alternatives, rather than generating the masked emotion word, and the result is a near-tie. A human-participant experiment on the same masked posts is needed to determine whether the low lexical/vector accuracy reflects a distinctive limitation of LLMs or a property of the task. Alternatively, the paper should explicitly state that the benchmark measures divergence from self-disclosed wording, not human-level emotional accuracy.","section":"Measuring Accuracy of Emotion Recognition; Human Evaluation"},{"comment":"The gold labels are produced by a regex protocol followed by a manual review, but no reliability or filtering statistics are reported ('we performed a manual review to filter out posts that did not satisfy the conditions but were not detected by the protocol'). Since all metrics compare model predictions to these extracted labels, masking errors or ambiguous cases—such as the multiple-label examples in Table A.3—directly depress AccL and AccV. The paper should report the number of posts removed or corrected by manual review, the specific exclusion criteria, and, if feasible, inter-annotator agreement on a sample of the masking decisions. Without this, the precision of EXPRESS as a gold-standard resource is unquantified.","section":"Emotion Masking"}],"minor_comments":[{"comment":"The introduction states EXPRESS 'consists of 33,679 human experiences,' while the dataset construction section says 'comprises 33,697 posts' (and Table 1 lists 33,679). Please reconcile the numbers.","section":"Introduction vs. Dataset Statistics"},{"comment":"The random baseline AccV=0.322 appears without a derivation. Specifying the random vector generation process (e.g., random independent Bernoulli draws with EmoLex marginals, or random permutations) would make the baseline interpretable.","section":"Evaluation Metrics (Appendix)"},{"comment":"The claim that GPT-4o has '1.75T parameters' is attributed without a citable source. If this is an estimate, please phrase it as such or cite a specific reference.","section":"Model Details"},{"comment":"The example 'grateful' vs. 'thankful' is presented as a misprediction, but these terms likely share the same EmoLex vector and therefore would not count as an error under AccV. Clarify which metric this error analysis refers to and whether the examples are lexical or vector-level errors.","section":"Error Analysis"},{"comment":"The log-scale x-axis in Figure 2 makes it difficult to compare model families; adding labels or separate panels for architecture types would improve readability.","section":"Table 3 / Figure 2"}],"recommendation":"major_revision","confidential_remarks":"This is a well-executed empirical study with a useful new benchmark, and the quantitative results are likely reproducible. The main problem is interpretive: the expert-preference result (LLM favored 43.7% vs. 40.0%) directly contradicts the 'emotional blind spots' framing, and the paper's own limitations acknowledge the ground-truth ambiguity. I would not reject the paper; with a reframed title/abstract and an added human baseline or explicit limitation, it could become a solid benchmark contribution. The masking reliability should also be reported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The dataset is the real contribution; the emotional-blind-spots framing is the load-bearing weakness. If you work on emotion recognition, EXPRESS is worth knowing: 33,679 Reddit posts, 251 self-disclosed labels, 259 words average context, code and data released, and 14 models tested under four prompting settings. That fills a genuine gap, since most benchmarks cap out around 27 labels and rely on short texts.\n\nThe measurement effort is solid. Lexical match is low, vector-level match is low even for GPT-4o (AccV 0.388), and the pattern holds across model families and sizes. The CoT degradation is a useful replication of prior work on subjective tasks, and the nearest-example few-shot gain is a concrete, non-obvious finding. The human evaluation is the most interesting section, and it is also where the paper undercuts itself. Three emotion experts preferred the LLM's predicted emotion over the self-disclosed one in 43.7% of the 213 cases versus 40.0% for the self-disclosed label, with Fleiss kappa of 0.21. The paper's own Limitations section concedes there may be cases where a term cannot be predicted without more context. So \"alignment with self-disclosed emotions\" is a real behavior, but it is not a clean measure of emotional accuracy; it is closer to a measure of whether the model can guess the writer's word choice. That is a legitimate benchmark target, but it does not support the title claim about emotional blind spots.\n\nSoft spots, in order of importance: (1) the interpretive frame overreaches the evidence; (2) the prompt template was tuned on a sample without a clear held-out split, which can inflate absolute accuracy numbers; (3) the main accuracy results are reported without error bars or confidence intervals, though the Wilcoxon tests provide some support; (4) the regex masking protocol plus manual review has no reported reliability. None of these kill the dataset. They do mean the paper should be read as a resource paper plus a cautionary evaluation, not as a demonstration that LLMs lack emotional understanding.\n\nWho is this for? People building or evaluating affective systems, mental-health NLP researchers, and anyone who needs a long-context, fine-grained emotion benchmark. It deserves a serious referee. I would ask the authors to reframe the headline claims, report uncertainty, and clarify the prompt-tuning procedure, but the underlying resource is valuable.","headline":"EXPRESS is a genuinely useful benchmark resource, but the paper's 'emotional blind spots' framing overreaches: the authors' own expert evaluation shows self-disclosed labels are not uniquely correct.","tokens_in":27636,"tokens_out":2678,"would_cite":true,"duration_ms":31648,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper builds a benchmark of 33,679 Reddit posts with self-disclosed emotion labels and shows that even the best language models, such as GPT-4o, align with those labels only about a third of the time at the lexical level and under 40% a","keywords":["emotion recognition","self-disclosed emotions","LLM evaluation","fine-grained emotion labels","benchmark dataset","few-shot learning","chain-of-thought prompting","mental health"],"falsifier":"A larger, pre-registered human evaluation using all 33,679 posts and a broader panel of emotion experts would settle the central claim: if expert majorities consistently prefer LLM predictions over self-disclosed emotions substantially more than 43.7% of the time, then 'alignment with self-disclosed emotions' is not a clean measure of emotional accuracy. Conversely, if agreement and preference for self-disclosures grow with more raters, the paper's low-alignment conclusion would be reinforced.","tokens_in":26735,"feed_emoji":"💭","tokens_out":4019,"duration_ms":46780,"temperature":0.7,"pith_summary":"The paper sets out to test whether language models can name the emotion a person actually says they feel, not just classify text into a fixed list of categories. It builds EXPRESS, a dataset of 33,679 Reddit posts in which the writer's own emotion word is masked, and asks 14 models to predict it. Across all models, exact lexical prediction stays low, with the best model reaching 31.3% lexical accuracy and 38.8% accuracy after decomposing emotions into eight basic-emotion and two sentiment dimensions. The paper concludes that LLMs still largely fail to align with human self-disclosed emotions, though few-shot examples help and chain-of-thought prompting hurts. A human expert panel preferred the LLM's predicted emotion over the writer's own in 43.7% of sampled disagreements, showing that the target itself is partly ambiguous.","feed_headline":"LLMs match self-disclosed emotions under 40 percent of the time","feed_subtitle":"New 34k-post benchmark finds GPT-4o scores 31% lexical, 39% emotion-vector accuracy; experts themselves split on which emotion fits.","key_machinery":"The evaluation framework decomposes both predicted and self-disclosed emotion words into 10-dimensional vectors using a widely used crowdsourced word-emotion lexicon aligned with Plutchik's Wheel of Emotion: eight basic emotions (joy, sadness, anger, fear, trust, disgust, anticipation, surprise) plus positive and negative sentiment. This decomposition lets the benchmark measure whether a model captures the emotional components of a self-disclosed feeling even when the exact word differs. EXPRESS itself is built by regex-based masking of phrases such as 'I feel + emotion' or 'I am + emotion' in 33,679 Reddit posts, which are segmented into 512-token contexts; this yields 251 distinct self-dis","core_discovery":"The paper introduces EXPRESS, a fine-grained emotion recognition benchmark built from naturally occurring self-disclosures on Reddit, where authors explicitly name their own emotions. The evaluation masks those self-disclosed emotion words and asks language models to predict them, then compares predictions against the original words at two levels: exact lexical match and a 10-dimensional decomposition into eight basic emotions plus positive and negative sentiment. The central empirical finding is that accurate alignment with human self-disclosed emotions remains challenging: GPT-4o, the best model, achieves only 0.313 lexical accuracy and 0.388 vector accuracy in zero-shot evaluation, and mo","pith_inferences":["Because expert judges preferred the LLM's prediction over the writer's own self-disclosed emotion in 43.7% of disputed cases, the benchmark's central metric may be better read as agreement between two plausible human judgments rather than a measure of objective emotional accuracy; system builders should consider modeling a distribution over plausible emotions instead of a single target.","The few-shot-nearest improvement suggests a concrete extension: retrieval-augmented emotion prediction, where a model is shown similar past self-disclosures before answering, may substantially narrow the gap to human alignment in mental-health support tools.","Chain-of-thought's consistent harm raises a testable hypothesis that reasoning prompts nudge models toward generic emotion priors (such as 'anxious' or 'frustrated'); an experiment requiring models to quote specific contextual cues before predicting could reveal whether forcing evidence-grounded reasoning recovers accuracy.","Fine-tuning on EXPRESS may inflate apparent gains if models exploit lexicon overlap rather than context; a stress test that paraphrases posts and swaps the self-disclosed emotion word would measure whether models truly track contextual emotion cues."],"forward_implications":["Zero-shot emotion recognition does not reach 40% alignment at the lexical or vector level even for the best model, so current LLMs cannot be relied on to detect fine-grained self-disclosed emotions in social media text without further adaptation.","Adding four nearest-example few-shot demonstrations improves every tested LLM, suggesting that in-context retrieval of similar emotional experiences is a practical route to better emotion awareness without task-specific training.","Chain-of-thought prompting reliably worsens emotion prediction across model families, indicating that step-by-step reasoning is counterproductive in this subjective, context-sensitive task.","Masked language models with around 100 million parameters match or beat causal LLMs many times their size on masked emotion prediction, offering a cost-effective alternative for emotion-focused applications.","Providing GPT-4o with full posts instead of 512-token segments does not change performance, so long-context models gain little from additional context on this task."],"fun_headline_variants":["LLMs can't feel: match self-disclosed emotions under 40%","Even GPT-4o fails emotion match: 31% lexical, 39% vector","Fluent but unfeeling: LLMs miss nuanced emotions on Reddit","Emotion blind spot: Best LLM hits <40% on human self-disclosures","Language models flunk fine-grained emotion test from Reddit"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The paper's central claim depends on treating the regex-extracted self-disclosed emotion word as the correct answer, yet the study's own experts preferred the self-disclosed word only 40.0% of the time and showed only fair agreement with each other.","fun_headline_variants_meta":{"raw":{"variants":["LLMs can't feel: match self-disclosed emotions under 40%","Even GPT-4o fails emotion match: 31% lexical, 39% vector","Fluent but unfeeling: LLMs miss nuanced emotions on Reddit","Emotion blind spot: Best LLM hits <40% on human self-disclosures","Language models flunk fine-grained emotion test from Reddit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000944,"raw_usage":{"total_tokens":3855,"prompt_tokens":716,"completion_tokens":3139,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":460,"completion_tokens_details":{"reasoning_tokens":3047}},"tokens_in":460,"tokens_out":3139,"duration_ms":25976,"temperature":1.0,"reasoning_tokens":3047,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T18:49:49.793702+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A larger, pre-registered human evaluation using all 33,679 posts and a broader panel of emotion experts would settle the central claim: if expert majorities consistently prefer LLM predictions over self-disclosed emotions substantially more than 43.7% of the time, then 'alignment with self-disclosed emotions' is not a clean measure of emotional accuracy. Conversely, if agreement and preference for self-disclosures grow with more raters, the paper's low-alignment conclusion would be reinforced.","supporting_citations":[],"review_version":1}