Pith. sign in

REVIEW 2 major objections 2 minor 27 references

Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies

T0 review · 2 major / 2 minor · reviewed 2026-07-02 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Three leading LLMs reach similar accuracies around 39 percent on zero-shot 13-class emotion classification with no significant differences.

desk verdict Three frontier LLMs top out near 39% accuracy on a 13-class emotion task with one zero-shot prompt, showing no significant pairwise differences but clear macro-F1 gaps. read the letter →

arxiv 2607.00968 v1 pith:KQCGEEF3 submitted 2026-07-01 cs.CL cs.HC

Lawrence Obiuwevwi , Krzysztof J. Rechowicz , Jessica M. Johnson , Vikas Ashok , Sachin Shetty , Sampath Jayarathna This is my paper · ORCID
classification cs.CLcs.HC
keywords zero-shotevaluationemotionclassificationlargelanguagemodelsaffectivecomputingfine-grainedtaxonomyLLMperformanceceiling13-classemotions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests Claude, GPT-5.4, and Gemini on assigning sentences to one of 13 specific emotion categories using only a single prompt and no examples. It reports that the three models post nearly identical results, with Gemini at 39.9 percent accuracy, GPT-5.4 at 38.8 percent, and Claude at 38.0 percent, and statistical tests confirm the differences are not meaningful. The models succeed on sarcasm and desire yet fail on love, confusion, and shame, while Claude shows extra bias toward majority classes. A reader would care because emotion classification supports mental-health tools and conversational systems that need reliable fine-grained understanding. The work therefore frames current frontier models as having hit a common zero-shot limit on this taxonomy.

What carries the argument

Uniform zero-shot prompt applied identically across models to the stratified sample, evaluated by accuracy and macro-F1 on the 13-class taxonomy.

What would settle it

Repeating the evaluation on a fresh stratified sample or with an altered prompt template and finding that one model pulls ahead by a statistically significant margin.

Watch

Extended reading notes

Core claim

On a stratified 1,000-sentence sample drawn from the 13-class boltuix/emotions dataset, a single uniform zero-shot prompt produces accuracies of 39.9 percent for Gemini, 38.8 percent for GPT-5.4, and 38.0 percent for Claude, with macro-F1 scores of 0.363, 0.291, and 0.159 respectively; McNemar tests find no pairwise differences at p greater than 0.10, indicating the models converge at a shared zero-shot ceiling while Claude additionally exhibits class-imbalance bias.

Load-bearing premise

The single uniform prompt and the 1,000-sentence stratified sample produce an unbiased estimate of each model's true zero-shot capability on the full 13-class taxonomy.

Editorial extensions

If this is right

  • Gemini, GPT-5.4, and Claude show no statistically significant accuracy differences on the task.
  • All three models perform well on sarcasm and desire but consistently fail on love, confusion, and shame.
  • Claude's lower macro-F1 reveals a stronger class-imbalance prediction bias than the other two models.
  • Current frontier models appear to share a performance ceiling in zero-shot fine-grained emotion classification.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The ceiling may indicate that zero-shot prompting alone is insufficient for reliable use in affective applications without further adaptation.
  • Testing whether few-shot examples or task-specific fine-tuning lift performance above this level would directly test the limit.
  • Mental-health and conversational systems relying on these models may need hybrid approaches that combine LLMs with other signals.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper conducts a zero-shot evaluation of three commercial LLMs—Claude (claude-sonnet-4-6), GPT-5.4, and Gemini (gemini-2.5-flash)—on a 13-class fine-grained emotion classification task. Using a single uniform prompt without exemplars on a stratified sample of 1,000 sentences from the boltuix/emotions dataset (131,306 sentences total), it reports Gemini achieving the highest accuracy of 39.9% and macro-F1 of 0.363, followed by GPT-5.4 (38.8%, 0.291) and Claude (38.0%, 0.159). McNemar tests show no significant pairwise differences (p > 0.10), leading to the conclusion of convergence at a shared zero-shot performance ceiling, with models performing well on sarcasm and desire but poorly on love, confusion, and shame. Claude shows class-imbalance bias.

Significance. If the empirical results are robust, the paper provides a timely benchmark demonstrating the limitations of current frontier LLMs in zero-shot fine-grained affective computing tasks. This has direct relevance for applications in conversational AI and mental health, quantifying an 'affective gap' that suggests further advances are needed beyond current model scales and training paradigms.

major comments (2)
  1. [Evaluation Setup] The central claim of a 'shared zero-shot ceiling' and lack of statistically significant differences depends on the representativeness of the single uniform prompt and the 1,000-sentence stratified sample. The manuscript does not include the exact prompt text, any prompt ablations, or results from multiple independent samples, which leaves open the possibility that prompt-specific or sampling artifacts drive the observed macro-F1 variations (0.363 vs 0.291 vs 0.159) and the McNemar outcomes.
  2. [Results] The reported McNemar tests (p > 0.10) are used to support no pairwise differences, but without accompanying confusion matrices, per-class precision/recall breakdowns, or error analysis on the 13 classes, it is difficult to assess whether the close accuracy scores reflect true convergence or masking effects from class imbalances and prediction biases noted for Claude.
minor comments (2)
  1. [Abstract] The dataset citation 'boltuix/emotions' should include a reference or link to the source for reproducibility.
  2. [Abstract] Model version names like 'GPT-5.4' and 'gemini-2.5-flash' appear non-standard; clarify if these are placeholders or specific API endpoints used in April 2026.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for these constructive comments, which help clarify the presentation of our evaluation. We respond to each major comment below, indicating planned revisions where appropriate.

read point-by-point responses

  1. Referee: [Evaluation Setup] The central claim of a 'shared zero-shot ceiling' and lack of statistically significant differences depends on the representativeness of the single uniform prompt and the 1,000-sentence stratified sample. The manuscript does not include the exact prompt text, any prompt ablations, or results from multiple independent samples, which leaves open the possibility that prompt-specific or sampling artifacts drive the observed macro-F1 variations (0.363 vs 0.291 vs 0.159) and the McNemar outcomes.

    Authors: We will include the exact prompt text verbatim in the revised Methods section for full reproducibility. The single uniform prompt was intentionally selected to enable a controlled, apples-to-apples zero-shot comparison across the three models without introducing variability from different prompting techniques or few-shot examples. The 1,000-sentence sample was drawn via stratified sampling to preserve the class distribution of the full 131,306-sentence dataset. While prompt ablations and repeated independent samples would provide additional robustness checks, they fall outside the scope of the current study, which focuses on convergence under a standardized zero-shot protocol. We will add explicit discussion of these design decisions and their limitations in the revised manuscript. revision: partial

  2. Referee: [Results] The reported McNemar tests (p > 0.10) are used to support no pairwise differences, but without accompanying confusion matrices, per-class precision/recall breakdowns, or error analysis on the 13 classes, it is difficult to assess whether the close accuracy scores reflect true convergence or masking effects from class imbalances and prediction biases noted for Claude.

    Authors: We agree that the addition of these materials will improve interpretability. In the revised version we will append full confusion matrices and per-class precision, recall, and F1 scores for all three models. We will also expand the Results and Discussion sections with a targeted error analysis covering the classes where models perform well (sarcasm, desire) and poorly (love, confusion, shame), along with further detail on Claude's observed class-imbalance bias. The McNemar tests remain appropriate for assessing paired accuracy differences on identical instances; the macro-F1 scores already surface distributional differences that we discuss as evidence of bias despite non-significant accuracy gaps. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: pure empirical benchmarking with direct API outputs and standard metrics

full rationale

The paper performs a zero-shot evaluation by sending a fixed prompt to three commercial LLM APIs on a stratified 1000-sentence sample from boltuix/emotions and reports accuracy, macro-F1, and McNemar p-values. No equations, fitted parameters, derivations, or predictions appear; all numbers are direct outputs of the API calls and off-the-shelf scikit-learn metrics. No self-citations are used to justify uniqueness or load-bearing premises, and the methodology contains no self-referential steps that reduce results to prior fitted values or ansatzes. The study is therefore self-contained against external benchmarks with no circularity.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

Empirical evaluation paper; no free parameters, axioms beyond standard statistical assumptions, or invented entities.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies." pith.science (2026). https://pith.science/paper/KQCGEEF3

@misc{pith2026260700968,
  author       = {Pith},
  title        = {Pith review of: Quantifying the Affective Gap: A Zero-Shot Evaluation of LLMs on Fine-Grained Emotion Taxonomies},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KQCGEEF3}},
  note         = {Machine review of arXiv:2607.00968}
}
read the original abstract

Emotion recognition in natural language is a foundational challenge in affective computing, with critical implications for human-computer interaction, mental health support, and conversational AI. This paper presents a rigorous, unified zero-shot evaluation of three leading commercial large language models: Claude (claude-sonnet-4-6), ChatGPT (GPT-5.4), and Gemini (gemini-2.5-flash). The models were queried through their respective production APIs as of April 2026 on a fine-grained 13-class emotion classification task. Using a stratified 1,000-sentence sample from the boltuix/emotions dataset, which comprises 131,306 sentences across 13 categories, a single uniform prompt with no exemplars was applied identically across all models. Gemini achieves the highest accuracy (39.9%) and macro-F1 score (0.363), followed by GPT-5.4 (38.8%, macro-F1 = 0.291) and Claude (38.0%, macro-F1 = 0.159). All models excel on sarcasm and desire while consistently failing on love, confusion, and shame. McNemar tests reveal no statistically significant pairwise differences (p > 0.10), suggesting convergence at a shared zero-shot ceiling. Claude's markedly lower macro-F1 score exposes a class-imbalance prediction bias. These findings highlight the current limitations of frontier AI systems in zero-shot fine-grained emotion classification.

Figures

Figures reproduced from arXiv: 2607.00968 by the authors.

Figure 1
Figure 1. Normalized confusion matrices for Claude, GPT-5.4, and Gemini. Off-diagonal mass concentrates on semantically proximal pairs: [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 3
Figure 3. Accuracy by sentence length. Medium sentences (5–15 words) yield [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages

  1. [1]

    Methods in predictive techniques for mental health status on social media: A critical review,

    S. Chancellor and M. De Choudhury, “Methods in predictive techniques for mental health status on social media: A critical review,”npj Digital Medicine, vol. 3, no. 43, 2020

    work page 2020

  2. [2]

    Towards em- pathetic open-domain conversation models: A new benchmark and dataset,

    H. Rashkin, E. M. Smith, M. Li, and Y .-L. Boureau, “Towards em- pathetic open-domain conversation models: A new benchmark and dataset,” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019, pp. 5370–5381

    work page 2019

  3. [3]

    An argument for basic emotions,

    P. Ekman, “An argument for basic emotions,”Cognition & Emotion, vol. 6, no. 3–4, pp. 169–200, 1992

    work page 1992

  4. [4]

    EmoBench: Evaluating the emotional in- telligence of large language models,

    S. Sabour, S. Liu, Z. Zhang, J. Liu, J. Zhou, A. Sunaryo, T. Lee, R. Mihalcea, and M. Huang, “EmoBench: Evaluating the emotional in- telligence of large language models,” inProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 5986–6004

    work page 2024

  5. [5]

    SentimentGPT: Exploiting GPT for advanced sentiment analysis and its departure from current machine learning,

    K. Kheiri and H. Karimi, “SentimentGPT: Exploiting GPT for advanced sentiment analysis and its departure from current machine learning,” arXiv preprint arXiv:2307.10234, 2023

    work page arXiv 2023

  6. [6]

    Crowdsourcing a word–emotion association lexicon,

    S. M. Mohammad and P. D. Turney, “Crowdsourcing a word–emotion association lexicon,”Computational Intelligence, vol. 29, no. 3, pp. 436– 465, 2013

    work page 2013

  7. [7]

    SemEval-2007 task 14: Affective text,

    C. Strapparava and R. Mihalcea, “SemEval-2007 task 14: Affective text,” inProceedings of the 4th International Workshop on Semantic Evaluations (SemEval-2007), 2007, pp. 70–74. [Online]. Available: https://aclanthology.org/S07-1013

    work page 2007

  8. [8]

    SemEval-2018 task 1: Affect in tweets,

    S. M. Mohammad, F. Bravo-Marquez, M. Salameh, and S. Kiritchenko, “SemEval-2018 task 1: Affect in tweets,” inProceedings of the 12th International Workshop on Semantic Evaluation (SemEval-2018), 2018, pp. 1–17

    work page 2018

Show all 27 references
  1. [9]

    SemEval- 2019 task 3: EmoContext contextual emotion detection in text,

    A. Chatterjee, K. N. Narahari, M. Joshi, and P. Agrawal, “SemEval- 2019 task 3: EmoContext contextual emotion detection in text,” in Proceedings of the 13th International Workshop on Semantic Evaluation (SemEval-2019), 2019, pp. 39–48

    2019

  2. [10]

    GoEmotions: A dataset of fine-grained emotions,

    D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “GoEmotions: A dataset of fine-grained emotions,” inProceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020, pp. 4040–4054

    2020

  3. [11]

    BERT: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre- training of deep bidirectional transformers for language understanding,” inProceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019, pp. 4171–4186

    2019

  4. [12]

    Using bert to understand tiktok users’ adhd discussion,

    K. Pineda, A. M. Perrotti, F. Poursardar, D. Graber, and S. Jayarathna, “Using bert to understand tiktok users’ adhd discussion,” in2023 IEEE 24th International Conference on Information Reuse and Integration for Data Science (IRI). IEEE, 2023, pp. 213–214

    2023

  5. [13]

    RoBERTa: A robustly optimized BERT pretraining approach,

    Y . Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V . Stoyanov, “RoBERTa: A robustly optimized BERT pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  6. [14]

    J. P. Tangney and R. L. Dearing,Shame and Guilt. New York, NY: Guilford Press, 2002

    2002

  7. [15]

    Automatic sarcasm detection: A survey,

    A. Joshi, P. Bhattacharyya, and M. J. Carman, “Automatic sarcasm detection: A survey,”ACM Computing Surveys, vol. 50, no. 5, pp. 73:1– 73:22, 2017

    2017

  8. [16]

    Cognitive prosthetic: An AI-enabled multimodal system for episodic recall in knowledge work,

    L. Obiuwevwi, K. J. Rechowicz, V . Ashok, S. Shetty, and S. Jayarathna, “Cognitive prosthetic: An AI-enabled multimodal system for episodic recall in knowledge work,” inProceedings of the Extended Abstracts of the 2026 CHI Conference on Human Factors in Computing Systems, ser....

    2026

  9. [17]

    Analyzing uncon- strained reading patterns of digital documents using eye tracking,

    B. Mahanama, G. Jayawardena, and S. Jayarathna, “Analyzing uncon- strained reading patterns of digital documents using eye tracking,” in 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL). IEEE, 2021, pp. 282–283

    2021

  10. [18]

    Multidisciplinary reading patterns of digital documents,

    B. Mahanama, G. Jayawardena, Y . Abeysinghe, V . Ashok, and S. Ja- yarathna, “Multidisciplinary reading patterns of digital documents,” in 2022 Symposium on Eye Tracking Research and Applications, 2022, pp. 1–2

    2022

  11. [19]

    How devices shape mental effort in digital document reading: An eye-tracking study,

    K. Thennakoon, Y . Abeysinghe, P. Thenahandi, L. Obiuwevwi, V . Ashok, and S. Jayarathna, “How devices shape mental effort in digital document reading: An eye-tracking study,” in2025 ACM/IEEE Joint Conference on Digital Libraries (JCDL), 2025, pp. 237–240

    2025

  12. [20]

    Language models are few-shot learners,

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language models are few-shot learners,”Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 1877–1901, 2020

    1901

  13. [21]

    Training language models to follow instructions with human feedback,

    L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Rayet al., “Training language models to follow instructions with human feedback,”Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 27 730–27 744, 2022

    2022

  14. [22]

    Finetuned language models are zero-shot learners,

    J. Wei, M. Bosma, V . Y . Zhao, K. Guu, A. W. Yu, B. Lester, N. Du, A. M. Dai, and Q. V . Le, “Finetuned language models are zero-shot learners,” inInternational Conference on Learning Representations (ICLR), 2022

    2022

  15. [23]

    emotions-dataset,

    boltuix, “emotions-dataset,” HuggingFace Datasets, 2023. [Online]. Available: https://huggingface.co/datasets/boltuix/emotions-dataset

    2023

  16. [24]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, and D. Zhou, “Chain-of-thought prompting elicits reasoning in large language models,”Advances in Neural Information Processing Systems (NeurIPS), vol. 35, pp. 24 824–24 837, 2022

    2022

  17. [25]

    A coefficient of agreement for nominal scales,

    J. Cohen, “A coefficient of agreement for nominal scales,”Educational and Psychological Measurement, vol. 20, no. 1, pp. 37–46, 1960

    1960

  18. [26]

    Note on the sampling error of the difference between correlated proportions or percentages,

    Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,”Psychometrika, vol. 12, no. 2, pp. 153–157, 1947

    1947

  19. [27]

    Zero-data learning of new tasks,

    H. Larochelle, D. Erhan, and Y . Bengio, “Zero-data learning of new tasks,” inProceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI), 2008, pp. 646–651. [Online]. Available: https: //aaai.org/papers/00646-aaai08-103-zero-data-learning-of-new-tasks/

    2008

Pith tools

Reviewed July 2, 2026 · model on record in the stance chip above.