Pith. sign in

REVIEW 3 major objections 5 minor 77 references

Rethinking Emotion Annotations in the Era of Large Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A crowdsourced study finds that human evaluators prefer GPT-4's zero-shot emotion labels over original human annotations in 62% of comparisons, suggesting that benchmarking LLMs against human labels as ground truth underestimates them.

desk verdict A useful, honest study of GPT-4 in emotion annotation, but the post-filtering evidence is weaker than claimed because the test set is built with the same rule as the training set. read the letter →

arxiv 2412.07906 v1 pith:MJSZGQI7 submitted 2024-12-10 cs.CL

Minxue Niu , Yara El-Tawil , Amrit Romana , Emily Mower Provost This is my paper · ORCID
classification cs.CL
keywords emotionannotationlargelanguagemodelsGPT-4humanevaluationcrowdsourcingmulti-labelclassificationpipelinesgroundtruth
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that human emotion labels should not be treated as the sole ground truth when evaluating large language models. In a crowdsourced evaluation, independent human evaluators preferred GPT-4's zero-shot emotion annotations over the original human labels in 62% of comparisons overall, with the preference strongest on the largest label space. The authors conclude that evaluating LLMs against human labels alone is prone to underestimation and can produce misleading results. They also propose two ways to integrate GPT-4 into human annotation pipelines: a pre-annotation filter that narrows label choices and a post-annotation filter that flags low-quality samples, reporting benefits in cognitive load, coverage, and downstream training efficiency.

What carries the argument

The central object is the pairwise human evaluation protocol: evaluators read a text and are shown two label sets, one from GPT-4 zero-shot prompting and one from the original human annotations, without knowing the source, and are asked for a forced preference, a 7-point perceived-accuracy rating, and a confidence judgment. This machinery converts the inherently subjective question of annotation quality into measurable preference votes that can be compared across three datasets with different label-space sizes and domains. For the pipeline experiments, the pre-filtering machinery uses a recall-oriented prompt that asks GPT-4 to mark each candidate label as 'yes' or 'no' to avoid dropping valid labels, while the post-filtering machinery applies a simple exact-overlap rule: samples where GPT-4 and human labels share no labels are removed. The three datasets, ISEAR (7 classes), SemEval (11 classes), and GoEmotions (28 classes), provide controlled variation in label-space complexity.

What would settle it

Retrain the post-filter comparison on a GoEmotions test set annotated by a fresh panel of consensus annotators who do not use the GPT-4 overlap rule, and check whether models trained on the GPT-4-filtered set still beat models trained on the full set; if they do not, the post-filtering quality claim collapses.

Watch

Extended reading notes

Core claim

The paper establishes that, when shown the same text with two anonymized label sets, human evaluators prefer GPT-4's zero-shot emotion predictions over the original human annotations in 62% of preference votes overall (60.66% for ISEAR, 58.16% for SemEval, and 66.41% for GoEmotions), and that this preference grows with label-space size. It interprets this as evidence that the common practice of benchmarking LLMs against human labels as ground truth underestimates their performance and may mislead conclusions about their capabilities. Complementing this, the paper shows that a GPT-4 pre-filter can remove about 70% of candidate labels while retaining over 90% of labels that human annotators choose from the full set, reducing cognitive load and annotation time without losing coverage. It also shows that a post-filter which drops samples where GPT-4 and human labels share no overlap produces a training set under 40% of the original size on which BERT and DistilBERT achieve better F1 and UAR than on the full human-labeled set, suggesting that GPT-4 can flag low-quality labels and improve training efficiency.

Load-bearing premise

The evidence that post-filtering improves label quality assumes the filtered test set is an independent quality benchmark, but that test set is built by the same exact-overlap rule that created the filtered training set, so the two are not independent.

Editorial extensions

If this is right

  • Models evaluated only against original human labels may be underrated, so future evaluations should include human preference judgments alongside agreement metrics.
  • GPT-4 pre-filtering can make large emotion label spaces practical by cutting cognitive load and annotation time while preserving label coverage, supporting more granular emotion taxonomies.
  • GPT-4 post-filtering can shrink training data size without sacrificing, and possibly improving, downstream model performance, lowering the cost of training data preparation.
  • Larger label spaces yield more annotations rated as totally accurate by evaluators for both humans and GPT-4, so label-space design should explicitly weigh granularity against annotator effort and agreement.
  • Disagreement patterns between GPT-4 and human annotation are asymmetric and dataset-specific, meaning hybrid pipelines should treat the two as complementary perspectives rather than interchangeable sources.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If GPT-4's labels are systematically preferred by evaluators, many existing emotion benchmarks encode a particular human perspective that may not match the majority read; future datasets might report multiple label sets or annotator-perspective metadata to make this visible.
  • The pre-filtering approach likely generalizes to other subjective text-annotation tasks with large label spaces, such as hate-speech or stance detection, though the recall-oriented prompt design would need recalibration for each task's false-negative tolerance.
  • The paper's finding that preference for GPT-4 grows with label space suggests a testable curve relating preference advantage to the number of candidate classes, which could predict the optimal label-space size for hybrid annotation pipelines.
  • The post-filtering result, if confirmed on an independently constructed test set, would imply that the effective information content of a training set matters more than its raw size, potentially changing how annotation budgets are allocated.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper investigates whether GPT-4's zero-shot emotion annotations can be used as an alternative or complement to human annotations. It reports a crowdsourcing study in which 59 evaluators compared GPT-4 labels with original human labels on 990 samples drawn from ISEAR, SemEval, and GoEmotions; GPT-4 labels were preferred in about 62% of preference votes (60.7% ISEAR, 58.2% SemEval, 66.4% GoEmotions). The paper then proposes two GPT-4-assisted annotation pipelines: pre-filtering, where GPT-4 reduces the label set shown to human annotators, and post-filtering, where samples with no overlap between GPT-4 and human labels are removed from training data. Experiments on GoEmotions indicate that pre-filtering reduces cognitive load and time while preserving label coverage, and that post-filtering improves downstream model F1 and UAR on the filtered test set. The authors conclude that evaluating LLMs against human labels alone may underestimate their performance and advocate for human-in-the-loop annotation with LLM support.

Significance. If the preference result held on a representative sample, it would meaningfully challenge the common practice of treating human labels as the sole ground truth in emotion recognition. The study's strengths include a real crowdsourcing evaluation with independent evaluators, three datasets with different label spaces, and a concrete demonstration of LLM-assisted pre- and post-filtering. The release of prompts and code is also valuable. However, the preference estimate is computed only on disagreement samples, and the post-filtering evaluation uses a test set constructed by the same filter rule as the training set. These two issues directly limit the strength of the paper's central claims, so the results should be interpreted with caution pending re-analysis.

major comments (3)
  1. [§4.2.1, §4.4.1] The 62% preference figure is computed on the 990 samples remaining after dropping all samples where GPT-4 and human labels exactly agree, as well as samples rejected by GPT-4's content policy. Consequently, the "overall 62%" (Section 4.4.1) is conditional on disagreement; it does not estimate the fraction of all samples on which GPT-4 is preferred, because agreement samples are excluded by construction. To support the conclusion that evaluating LLMs against human labels is "prone to underestimate their performance," the authors should either explicitly reweight the preference estimate by the proportion of agreement samples in each dataset or restrict the wording to "among samples where the two sources disagree." In addition, the claim of significance is not backed by a statistical test; a mixed-effects logistic regression with evaluator and sample random effects would be appropriate given the nested structure of the data.
  2. [§5.2, Table 3] The post-filtering performance gain is measured on a "Filter" test set that is created by applying the exact-overlap rule (drop samples with no overlapping label between GPT-4 and humans) to the test split, the same rule used to construct the Filter training set. The higher F1/UAR on the F column may therefore reflect train/test distribution alignment on a truncated label space rather than removal of low-quality labels. The independent human-labeled test set (H column) shows the Filter-trained BERT actually has lower F1 than the Human-trained BERT (0.442 vs 0.472), which the paper acknowledges as an exception. The paper states in Section 5.2.1 that its primary test set is the human-validated 438-sample set from Section 4.2.1, but this set is not reported in Table 3. The post-filtering claim needs to be demonstrated on an independently labeled test set that is not constructed by the same overlap rule, or the conclusions should be softened accordingly.
  3. [§5.2, Table 3] The "Random F" baseline controls only for the reduction in training-set size; it does not control for the selection rule itself. A more informative comparison would include another quality-based filter, such as removing samples with low inter-annotator agreement in the original GoEmotions data or using annotator confidence scores. Without such a baseline, the observed improvement could be attributed to the removal of inherently ambiguous samples rather than to GPT-4's specific ability to flag low-quality labels. This distinction is important because the paper's stated goal is to evaluate GPT-4 as a quality checker, not merely any filtering heuristic.
minor comments (5)
  1. [§4.4.2, Figure 3] The pooled rating distributions in Figure 3 and Table 2 ignore the fact that multiple evaluations come from the same sample and the same evaluator. Reporting intraclass correlations or fitting a multilevel model would strengthen the claim that the rating difference is not driven by a few influential evaluators or samples.
  2. [§4.4.4] The logistic regression for GPT-4's weaknesses uses t-test-based feature selection on the same data and then fits the regression on the same data without cross-validation, which risks overfitting. The authors appropriately note the limited sample size, but the analysis should be described as exploratory and the reported p-values interpreted with that caveat.
  3. [§5.1.1] The between-subject design for the pre-filtering study means that differences between the Small, Large, and Pre-Filtered groups could partly reflect annotator-level differences. It would help to report the random assignment procedure and any demographic summary for the three groups, or to run a sensitivity analysis with participant as a random effect.
  4. [§5.2.1] The paper says the human-validated 438-sample set from Section 4.2.1 is the primary test set, but Table 3 does not include results on that set. Please either add those results or clarify why they were omitted, since the current table makes the post-filtering evaluation look weaker than the paper's own description suggests.
  5. [Throughout] Some reference formatting issues appear (for example, extra spaces in author initials in the bibliography), and the figure captions for Figure 1 repeat the same confusion matrix across panels. These should be cleaned up in a final version.

Circularity Check

1 steps flagged · score 6.0 of 10

Post-filtering's Table 3 gain is measured on a Filter test set built by the same rule as the Filter training set, so the evidence for label-quality improvement is partially circular.

  1. self definitional [Section 5.2, Section 5.2.1, Table 3]
    "Specifically, we compare the labels from human and GPT-4 annotation and drop the samples where the two sources totally disagree: i.e., they selected different labels for single-label classification datasets, or where they do not contain any overlapping labels for multi-label classification datasets. By applying this filtering step to GoEmotions, we obtained a much smaller Filtered set of 16,592 samples (out of 42,287). ... We report performance on both the whole test set and a filtered test set."

    The 'F' test set in Table 3 is produced by applying the same filtering rule used to create the Filter training set: samples with no overlap between GPT-4 and human labels are removed from both train and test. A model trained on the Filter set is therefore evaluated on the same truncated label distribution it was trained on, while the Human-trained model is evaluated out-of-distribution relative to that truncation. The higher F1/UAR of Filter-trained models on the F column is partly a train/test domain-alignment artifact, not independent evidence that the removed labels were low quality. The 'Random F' control only matches sample size, not the truncation rule.

full rationale

The paper's human-preference result is not circular: it uses independent Prolific evaluators and blind comparisons of GPT-4 and human labels, and the pre-filtering analysis is validated against human-selected labels with an external coverage benchmark. The post-filtering claim, however, is partially circular. The Filter training set is defined by dropping samples where GPT-4 and human labels share no overlap, and the 'F' test column in Table 3 is produced by the same truncation of the test split. A model trained on the Filter set is therefore evaluated in the same truncated distribution it was trained on, while the Human-trained model is evaluated out-of-distribution for that subset; the F-column gains are partly a domain-alignment artifact, not an independent measure of label quality. The 'Random F' control only matches sample size and cannot remove this train/test rule-alignment confound. The independent H column shows the Filter-trained BERT has lower F1 than the Human-trained BERT (0.442 vs 0.472), and the human-validated primary test set mentioned in Section 5.2.1 is not reported in Table 3. These issues make the post-filtering evidence partially self-confirming. Because the paper also contains genuinely external evaluations (the preference study and pre-filtering coverage), the overall circularity is moderate rather than total, but the central applied claim about downstream model improvement is weakened by the self-constructed benchmark.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The work is empirical, so the ledger records hand-chosen thresholds and design assumptions rather than fitted theoretical constants. The main load-bearing choices are the disagreement-only sampling for the human evaluation, the exact-overlap rule for post-filtering, and the aggregation thresholds used to compute coverage.

free parameters (4)
  • Post-filter disagreement criterion = zero overlap between human and GPT-4 label sets
    Defines what counts as a low-quality sample in Section 5.2; this threshold determines that 25,695 of 42,287 training samples are removed and is never varied.
  • Aggregation threshold for label coverage = emotion selected by at least 2 of 3 annotators
    Used in Section 5.1.2 to define aggregated human labels and the 90.19% coverage figure; different aggregation rules would change coverage.
  • Feature-selection cutoff = top 10 LIWC features by lowest t-test p-value
    Section 4.4.4 uses the same data to select features and then test significance; no multiple-comparison correction.
  • Fine-tuning hyperparameters = learning rate 1e-5, 30 epochs
    Section 5.2.1 fixes these for BERT and DistilBERT with no ablation; other hyperparameters might change the filtering versus full-data comparison.
assumptions (4)
  • domain assumption Evaluator preference between two label sets is a valid proxy for annotation quality.
    Section 4 uses 7-point accuracy ratings and forced preference to conclude GPT-4 labels are better than human labels; emotion has no independent ground truth.
  • ad hoc to paper Removing exact-agreement samples does not bias the preference estimate.
    Section 4.2.1 drops all samples where label sets matched exactly, then Section 4.4.1 reports overall 62% preference without reweighting.
  • domain assumption GPT-4 content-policy refusals are ignorable.
    Section 4.1 excludes 17, 12, and 14 samples in ISEAR, SemEval, and GoEmotions; no analysis of what content is lost.
  • ad hoc to paper A test set filtered by the same human-GPT overlap rule is a valid independent evaluation set.
    Section 5.2 constructs both training and test sets with the same zero-overlap drop rule, which makes the Filter test column in Table 3 partly in-domain rather than independent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rethinking Emotion Annotations in the Era of Large Language Models." pith.science (2026). https://pith.science/paper/MJSZGQI7

@misc{pith2026241207906,
  author       = {Pith},
  title        = {Pith review of: Rethinking Emotion Annotations in the Era of Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MJSZGQI7}},
  note         = {Machine review of arXiv:2412.07906}
}
read the original abstract

Modern affective computing systems rely heavily on datasets with human-annotated emotion labels, for training and evaluation. However, human annotations are expensive to obtain, sensitive to study design, and difficult to quality control, because of the subjective nature of emotions. Meanwhile, Large Language Models (LLMs) have shown remarkable performance on many Natural Language Understanding tasks, emerging as a promising tool for text annotation. In this work, we analyze the complexities of emotion annotation in the context of LLMs, focusing on GPT-4 as a leading model. In our experiments, GPT-4 achieves high ratings in a human evaluation study, painting a more positive picture than previous work, in which human labels served as the only ground truth. On the other hand, we observe differences between human and GPT-4 emotion perception, underscoring the importance of human input in annotation studies. To harness GPT-4's strength while preserving human perspective, we explore two ways of integrating GPT-4 into emotion annotation pipelines, showing its potential to flag low-quality labels, reduce the workload of human annotators, and improve downstream model learning performance and efficiency. Together, our findings highlight opportunities for new emotion labeling practices and suggest the use of LLMs as a promising tool to aid human annotation.

Figures

Figures reproduced from arXiv: 2412.07906 by the authors.

Figure 1
Figure 1. Disagreements between Human and GPT-4 Annotations, visualized as confusion matrices. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Proportion of human evaluators’ votes favoring hu [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Comparison of the cognitive load on different label [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

77 extracted references · 58 canonical work pages

  1. [1]

    R. W. Picard, Affective computing. MIT press, 2000

    2000

  2. [2]

    Affective com- puting: Recent advances, challenges, and future trends,

    G. Pei, H. Li, Y. Lu, Y. Wang, S. Hua, and T. Li, “Affective com- puting: Recent advances, challenges, and future trends,”Intelligent Computing, vol. 3, p. 0076, 2024

    work page 2024

  3. [3]

    Affective computing in education: A systematic review and future research,

    E. Yadegaridehkordi, N. F. B. M. Noor, M. N. B. Ayub, H. B. Affal, and N. B. Hussin, “Affective computing in education: A systematic review and future research,” Computers & education , vol. 142, p. 103649, 2019

    work page 2019

  4. [4]

    Affective computing for healthcare: Recent trends, applications, challenges, and beyond,

    Y. Liu, K. Wang, L. Wei, J. Chen, Y. Zhan, D. Tao, and Z. Chen, “Affective computing for healthcare: Recent trends, applications, challenges, and beyond,” arXiv preprint arXiv:2402.13589, 2024

    arXiv 2024

  5. [5]

    L. Tian, S. Oviatt, M. Muszynski, B. Chamberlain, J. Healey, and A. Sano, Applied Affective Computing. Morgan & Claypool, 2022

    work page 2022

  6. [6]

    A systematic review on affective computing: Emotion models, databases, and recent advances,

    Y. Wang, W. Song, W. Tao, A. Liotta, D. Yang, X. Li, S. Gao, Y. Sun, W. Ge, W. Zhang et al., “A systematic review on affective computing: Emotion models, databases, and recent advances,” Information Fusion, vol. 83, pp. 19–52, 2022

    work page 2022

  7. [7]

    Objective assessment of subjective tasks in crowd- sourcing applications,

    G. Haralabopoulos, M. Tsikandilakis, M. Torres Torres, and D. McAuley, “Objective assessment of subjective tasks in crowd- sourcing applications,” in LREC 2020 Workshop on” Citizen Linguis- tics in Language Resource Development”, 2020

    work page 2020

  8. [8]

    The ambiguous world of emotion representation,

    V . Sethu, E. M. Provost, J. Epps, C. Busso, N. Cummins, and S. Narayanan, “The ambiguous world of emotion representation,” arXiv preprint arXiv:1909.00360, 2019

    arXiv 1909

Show all 77 references
  1. [9]

    Com- paring the utility of different classification schemes for emotive language analysis,

    L. Williams, M. Arribas-Ayllon, A. Artemiou, and I. Spasi ´c, “Com- paring the utility of different classification schemes for emotive language analysis,” Journal of Classification , vol. 36, pp. 619–648, 2019

    2019

  2. [10]

    Muse-ing on the impact of utterance ordering on crowdsourced emotion annotations,

    M. Jaiswal, Z. Aldeneh, C.-P . Bara, Y. Luo, M. Burzo, R. Mihal- cea, and E. M. Provost, “Muse-ing on the impact of utterance ordering on crowdsourced emotion annotations,” in ICASSP 2019- 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). ...

    2019

  3. [11]

    Toward effective automatic recognition systems of emotion in speech,

    C. Busso, M. Bulut, S. Narayanan, J. Gratch, and S. Marsella, “Toward effective automatic recognition systems of emotion in speech,” Social emotions in nature and artifact: emotions in human and human-computer interaction, vol. 7, no. 17, pp. 110–127, 2013

    2013

  4. [12]

    Challenges in annotation: Annotator experiences from a crowdsourced emotion annotation task

    E. ¨Ohman, “Challenges in annotation: Annotator experiences from a crowdsourced emotion annotation task.” in DHN, 2020, pp. 293– 301

    2020

  5. [13]

    Influence of contextual informa- tion in emotion annotation for spoken dialogue systems,

    Z. Callejas and R. Lopez-Cozar, “Influence of contextual informa- tion in emotion annotation for spoken dialogue systems,” Speech Communication, vol. 50, no. 5, pp. 416–433, 2008

    2008

  6. [14]

    https://github.com/chailab-umich/GPT-4-Emotion-Annotation 10

  7. [15]

    Facial expression recognition with inconsistently annotated datasets,

    J. Zeng, S. Shan, and X. Chen, “Facial expression recognition with inconsistently annotated datasets,” in Proceedings of the European conference on computer vision (ECCV), 2018, pp. 222–237

    2018

  8. [16]

    Annotation and processing of continuous emotional attributes: Challenges and opportunities,

    A. Metallinou and S. Narayanan, “Annotation and processing of continuous emotional attributes: Challenges and opportunities,” in 2013 10th IEEE international conference and workshops on automatic face and gesture recognition (FG). IEEE, 2013, pp. 1–8

    2013

  9. [17]

    Challenges in real- life emotion annotation and machine learning based detection,

    L. Devillers, L. Vidrascu, and L. Lamel, “Challenges in real- life emotion annotation and machine learning based detection,” Neural Networks, vol. 18, no. 4, pp. 407–422, 2005

    2005

  10. [18]

    Gender differences in nonverbal communication of emotion,

    J. A. Hall, J. D. Carter, and T. G. Horgan, “Gender differences in nonverbal communication of emotion,” Gender and emotion: Social psychological perspectives, pp. 97–117, 2000

    2000

  11. [19]

    Emotion regulation and emo- tion perception in aging: A perspective on age-related differences and similarities,

    J. W. Gurera and D. M. Isaacowitz, “Emotion regulation and emo- tion perception in aging: A perspective on age-related differences and similarities,” Progress in brain research , vol. 247, pp. 329–351, 2019

    2019

  12. [20]

    Race and sex in the perception of emotion,

    A. G. Gitter, H. Black, and D. Mostofsky, “Race and sex in the perception of emotion,” Journal of Social Issues , vol. 28, no. 4, pp. 63–78, 1972

    1972

  13. [21]

    EmoLabel: Semi-automatic methodology for emotion annotation of social media text,

    L. Canales, W. Daelemans, E. Boldrini, and P . Martinez-Barco, “EmoLabel: Semi-automatic methodology for emotion annotation of social media text,” IEEE Trans. Affect. Comput., vol. 13, no. 2, pp. 579–591, Apr. 2022

    2022

  14. [22]

    Tradeoff between quality and quantity of emotional annotations to characterize expressive behaviors,

    A. Burmania, M. Abdelwahab, and C. Busso, “Tradeoff between quality and quantity of emotional annotations to characterize expressive behaviors,” in 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2016, pp. 5190–5194

    2016

  15. [23]

    Large language models for data annotation: A survey,

    Z. Tan, D. Li, S. Wang, A. Beigi, B. Jiang, A. Bhattacharjee, M. Karami, J. Li, L. Cheng, and H. Liu, “Large language models for data annotation: A survey,” arXiv preprint arXiv:2402.13446, 2024

    2024 arXiv

  16. [24]

    ChatGPT outperforms crowd workers for text-annotation tasks,

    F. Gilardi, M. Alizadeh, and M. Kubli, “ChatGPT outperforms crowd workers for text-annotation tasks,” Proc. Natl. Acad. Sci. U. S. A., vol. 120, no. 30, p. e2305016120, Jul. 2023

    2023

  17. [25]

    Affect recognition in conversations using large language models,

    S. Feng, G. Sun, N. Lubis, C. Zhang, and M. Ga ˇsi´c, “Affect recognition in conversations using large language models,” IEEE Computational Intelligence Magazine, Sep. 2023

    2023

  18. [26]

    Bias in emotion recognition with ChatGPT,

    N. Wake, A. Kanehira, K. Sasabuchi, J. Takamatsu, and K. Ikeuchi, “Bias in emotion recognition with ChatGPT,” arXiv [cs.RO], Oct. 2023

    2023

  19. [27]

    Can large language models aid in annotating speech emotional data? uncov- ering new frontiers,

    S. Latif, M. Usama, M. I. Malik, and B. W. Schuller, “Can large language models aid in annotating speech emotional data? uncov- ering new frontiers,” arXiv preprint arXiv:2307.06090, 2023

    2023 arXiv

  20. [28]

    Refashioning emotion recognition modelling: The advent of gen- eralised large models,

    Z. Zhang, L. Peng, T. Pang, J. Han, H. Zhao, and B. W. Schuller, “Refashioning emotion recognition modelling: The advent of gen- eralised large models,” IEEE Transactions on Computational Social Systems, 2024

    2024

  21. [29]

    Sentiment analysis in the era of large language models: A reality check,

    W. Zhang, Y. Deng, B. Liu, S. Pan, and L. Bing, “Sentiment analysis in the era of large language models: A reality check,” in Findings of the Association for Computational Linguistics: NAACL , 2024, pp. 3881–3906

    2024

  22. [30]

    From text to emotion: Unveiling the emotion annotation capabilities of llms,

    M. Niu, M. Jaiswal, and E. Mower Provost, “From text to emotion: Unveiling the emotion annotation capabilities of llms,” in Proc. Interspeech, 2024, pp. 2650–2654

    2024

  23. [31]

    Towards the improvement of automatic emotion pre-annotation with polarity and subjective information,

    L. Canales, University of Alicante, Alicante, Spain, W. Daelemans, E. Boldrini, and P . Mart ´ınez-Barco, “Towards the improvement of automatic emotion pre-annotation with polarity and subjective information,” in RANLP 2017 - Recent Advances in Natural Language Processing Meet...

    2017

  24. [32]

    An argument for basic emotions,

    P . Ekman, “An argument for basic emotions,” Cognition & emotion, vol. 6, no. 3-4, pp. 169–200, 1992

    1992

  25. [33]

    A circumplex model of affect

    J. A. Russell, “A circumplex model of affect.” Journal of personality and social psychology, vol. 39, no. 6, p. 1161, 1980

    1980

  26. [34]

    Emobank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis,

    S. Buechel and U. Hahn, “Emobank: Studying the impact of annotation perspective and representation format on dimensional emotion analysis,” EACL 2017, p. 578, 2017

    2017

  27. [35]

    Dailydialog: A manually labelled multi-turn dialogue dataset,

    Y. Li, H. Su, X. Shen, W. Li, Z. Cao, and S. Niu, “Dailydialog: A manually labelled multi-turn dialogue dataset,” in Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2017, pp. 986–995

    2017

  28. [36]

    How universal and specific is emotional experience? evidence from 27 countries on five conti- nents,

    H. G. Wallbott and K. R. Scherer, “How universal and specific is emotional experience? evidence from 27 countries on five conti- nents,” Social Science Information, vol. 25, no. 4, 1986

    1986

  29. [37]

    SemEval-2018 task 1: Affect in tweets,

    S. Mohammad, F. Bravo-Marquez, M. Salameh, and S. Kiritchenko, “SemEval-2018 task 1: Affect in tweets,” in Proceedings of The 12th International Workshop on Semantic Evaluation . Stroudsburg, PA, USA: Association for Computational Linguistics, 2018, pp. 1–17

    2018

  30. [38]

    Goemotions: A dataset of fine-grained emotions,

    D. Demszky, D. Movshovitz-Attias, J. Ko, A. Cowen, G. Nemade, and S. Ravi, “Goemotions: A dataset of fine-grained emotions,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 4040–4054

    2020

  31. [39]

    Man is to computer programmer as woman is to homemaker? debiasing word embeddings,

    T. Bolukbasi, K.-W. Chang, J. Y. Zou, V . Saligrama, and A. T. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,” Advances in neural information pro- cessing systems, vol. 29, 2016

    2016

  32. [40]

    Beyond the turk: Alternative platforms for crowdsourcing behavioral re- search,

    E. Peer, L. Brandimarte, S. Samat, and A. Acquisti, “Beyond the turk: Alternative platforms for crowdsourcing behavioral re- search,” Journal of experimental social psychology , vol. 70, pp. 153– 163, 2017

    2017

  33. [41]

    Increasing the reliability of crowdsourcing evaluations using online quality as- sessment,

    A. Burmania, S. Parthasarathy, and C. Busso, “Increasing the reliability of crowdsourcing evaluations using online quality as- sessment,” IEEE Transactions on Affective Computing , vol. 7, no. 4, pp. 374–388, 2015

    2015

  34. [42]

    Scaling laws for neural language models,

    J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei, “Scaling laws for neural language models,” arXiv preprint arXiv:2001.08361, 2020

    2001 arXiv

  35. [43]

    Emotional intelligence of large language models,

    X. Wang, X. Li, Z. Yin, Y. Wu, and J. Liu, “Emotional intelligence of large language models,” Journal of Pacific Rim Psychology , vol. 17, 2023

    2023

  36. [44]

    Is chatgpt equipped with emotional dialogue capabilities?

    W. Zhao, Y. Zhao, X. Lu, S. Wang, Y. Tong, and B. Qin, “Is chatgpt equipped with emotional dialogue capabilities?” arXiv preprint arXiv:2304.09582, 2023

    2023 arXiv

  37. [45]

    Is GPT a computational model of emo- tion?

    A. N. Tak and J. Gratch, “Is GPT a computational model of emo- tion?” in 2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII). IEEE, 2023, pp. 1–8

    2023

  38. [46]

    Emotionally numb or empathetic? evaluating how llms feel using emotionbench,

    J.-t. Huang, M. H. Lam, E. J. Li, S. Ren, W. Wang, W. Jiao, Z. Tu, and M. R. Lyu, “Emotionally numb or empathetic? evaluating how llms feel using emotionbench,” arXiv preprint arXiv:2308.03656 , 2023

    2023 arXiv

  39. [47]

    The biases of pre- trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,

    R. Mao, Q. Liu, K. He, W. Li, and E. Cambria, “The biases of pre- trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,” IEEE Transactions on Affective Computing, vol. 14, no. 3, pp. 1743–1753, 2023

    2023

  40. [48]

    Gpt-4 emulates average-human emo- tional cognition from a third-person perspective,

    A. N. Tak and J. Gratch, “Gpt-4 emulates average-human emo- tional cognition from a third-person perspective,” arXiv preprint arXiv:2408.13718, 2024

    2024 arXiv

  41. [49]

    Is gpt-3 a good data annotator?

    B. Ding, C. Qin, L. Liu, Y. K. Chia, B. Li, S. Joty, and L. Bing, “Is gpt-3 a good data annotator?” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023, pp. 11 173–11 195

    2023

  42. [50]

    Foundation model assisted automatic speech emotion recognition: Transcribing, annotating, and aug- menting,

    T. Feng and S. Narayanan, “Foundation model assisted automatic speech emotion recognition: Transcribing, annotating, and aug- menting,” in ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2024, pp. 12 116–12 120

    2024

  43. [51]

    Quantifying the persona effect in llm simulations,

    T. Hu and N. Collier, “Quantifying the persona effect in llm simulations,” arXiv preprint arXiv:2402.10811, 2024

    2024 arXiv

  44. [52]

    Potato: The portable text annotation tool,

    J. Pei, A. Ananthasubramaniam, X. Wang, N. Zhou, A. Dedeloudis, J. Sargent, and D. Jurgens, “Potato: The portable text annotation tool,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , 2022, pp. 327– 337

    2022

  45. [53]

    Choice overload: A conceptual review and meta-analysis,

    A. Chernev, U. B ¨ockenholt, and J. Goodman, “Choice overload: A conceptual review and meta-analysis,” Journal of Consumer Psychology, vol. 25, no. 2, pp. 333–358, 2015

    2015

  46. [54]

    The development and psychometric properties of LIWC2015,

    J. W. Pennebaker, R. L. Boyd, K. Jordan, and K. Blackburn, “The development and psychometric properties of LIWC2015,” 2015

    2015

  47. [55]

    Incorporating LIWC in neural networks to improve human trait and behavior analysis in low resource scenarios,

    I. Y. Kilic and S. Pan, “Incorporating LIWC in neural networks to improve human trait and behavior analysis in low resource scenarios,” in Proceedings of the Thirteenth Language Resources and Evaluation Conference, 2022, pp. 4532–4539

    2022

  48. [56]

    A multi-layered psychological- based reference model for citizen need assessment using ai- powered models,

    R. Alharthi and A. El Saddik, “A multi-layered psychological- based reference model for citizen need assessment using ai- powered models,” SN Computer Science, vol. 1, no. 5, p. 291, 2020

    2020

  49. [57]

    Detecting fake news spreaders with behavioural, lexical and psycholinguistic features

    H. R. M. Bello, L. Heilmann, and E. Ronan, “Detecting fake news spreaders with behavioural, lexical and psycholinguistic features.” in CLEF (Working Notes), 2020

    2020

  50. [58]

    Common pitfalls in statistical analysis: logistic regression,

    P . Ranganathan, C. Pramesh, and R. Aggarwal, “Common pitfalls in statistical analysis: logistic regression,” Perspectives in clinical research, vol. 8, no. 3, pp. 148–151, 2017. 11

    2017

  51. [59]

    Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models,

    Z. Chu, J. Chen, Q. Chen, W. Yu, H. Wang, M. Liu, and B. Qin, “Timebench: A comprehensive evaluation of temporal reasoning abilities in large language models,” arXiv preprint arXiv:2311.17667, 2023

    2023 arXiv

  52. [60]

    Do llms understand social knowledge? evaluating the sociability of large language models with socket benchmark,

    M. Choi, J. Pei, S. Kumar, C. Shu, and D. Jurgens, “Do llms understand social knowledge? evaluating the sociability of large language models with socket benchmark,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 11 370–11 403

    2023

  53. [61]

    Exploring the sensitivity of llms’ decision-making capabilities: Insights from prompt vari- ations and hyperparameters,

    M. Loya, D. Sinha, and R. Futrell, “Exploring the sensitivity of llms’ decision-making capabilities: Insights from prompt vari- ations and hyperparameters,” in Findings of the Association for Computational Linguistics: EMNLP 2023, 2023, pp. 3711–3716

    2023

  54. [62]

    Nasa-task load index (nasa-tlx); 20 years later,

    S. G. Hart, “Nasa-task load index (nasa-tlx); 20 years later,” in Pro- ceedings of the human factors and ergonomics society annual meeting , vol. 50, no. 9. Sage publications Sage CA: Los Angeles, CA, 2006, pp. 904–908

    2006

  55. [63]

    A comparison of emotion annotation approaches for text,

    I. D. Wood, J. P . McCrae, V . Andryushechkin, and P . Buitelaar, “A comparison of emotion annotation approaches for text,” Informa- tion, vol. 9, no. 5, p. 117, 2018

    2018

  56. [64]

    Development of nasa-tlx (task load index): Results of em- pirical and theoretical research,

    S. Hart, “Development of nasa-tlx (task load index): Results of em- pirical and theoretical research,” Human mental workload/Elsevier , 1988

    1988

  57. [65]

    The distribution of the flora in the alpine zone. 1,

    P . Jaccard, “The distribution of the flora in the alpine zone. 1,”New phytologist, vol. 11, no. 2, pp. 37–50, 1912

    1912

  58. [66]

    How reliable are annotations via crowd- sourcing: a study about inter-annotator agreement for multi-label image annotation,

    S. Nowak and S. R ¨uger, “How reliable are annotations via crowd- sourcing: a study about inter-annotator agreement for multi-label image annotation,” in Proceedings of the international conference on Multimedia information retrieval, 2010, pp. 557–566

    2010

  59. [67]

    Bert: Pre- training of deep bidirectional transformers for language under- standing,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language under- standing,” in Proceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Techn...

    2019

  60. [68]

    Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter,

    V . Sanh, “Distilbert, a distilled version of bert: Smaller, faster, cheaper and lighter,” arXiv preprint arXiv:1910.01108, 2019

    1910 arXiv

  61. [69]

    Bert: a review of applications in natural language processing and understanding,

    M. V . Koroteev, “Bert: a review of applications in natural language processing and understanding,” arXiv preprint arXiv:2103.11943 , 2021

    2021 arXiv

  62. [70]

    Joint audio and speech understanding,

    Y. Gong, A. H. Liu, H. Luo, L. Karlinsky, and J. Glass, “Joint audio and speech understanding,” in 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU) . IEEE, 2023, pp. 1–8

    2023

  63. [71]

    Emotion analysis as a regression problem–dimensional models and their implications on emotion representation and metrical evaluation,

    S. Buechel and U. Hahn, “Emotion analysis as a regression problem–dimensional models and their implications on emotion representation and metrical evaluation,” in ECAI 2016. IOS Press, 2016, pp. 1114–1122

    2016

  64. [72]

    Tracing vocal expression of emotion along the speech chain: Do listeners perceive what speakers feel?

    S. Biersack and V . Kempe, “Tracing vocal expression of emotion along the speech chain: Do listeners perceive what speakers feel?” in ISCA Workshop on Plasticity in Speech Perception , 2005

    2005

  65. [73]

    The expression and perception of emotions: comparing assessments of self versus others

    C. Busso and S. S. Narayanan, “The expression and perception of emotions: comparing assessments of self versus others.” in Interspeech, 2008, pp. 257–260

    2008

  66. [74]

    Label quality in affectnet: results of crowd-based re-annotation,

    D. Y. Kim and C. Wallraven, “Label quality in affectnet: results of crowd-based re-annotation,” in Asian Conference on Pattern Recognition. Springer, 2021, pp. 518–531

    2021

  67. [75]

    Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,

    P . Liu, W. Yuan, J. Fu, Z. Jiang, H. Hayashi, and G. Neubig, “Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing,” ACM Comput. Surv. , vol. 55, no. 9, pp. 1–35, Jan. 2023

    2023

  68. [76]

    Using cognitive psychology to understand gpt-3,

    M. Binz and E. Schulz, “Using cognitive psychology to understand gpt-3,” Proceedings of the National Academy of Sciences of the United States of America, vol. 120, no. 6, p. e2218523120, 2023

    2023

  69. [77]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al. , “Chain-of-thought prompting elicits reasoning in large language models,” Advances in neural information processing systems, vol. 35, pp. 24 824–24 837, 2022

    2022

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.