{"id":"ebd67505-3454-4029-83b9-bf91b8aa1970","arxiv_id":"2412.07906","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Human evaluators preferred GPT-4's zero-shot emotion labels over original human labels in 62% of disagreement samples, and GPT-4 pre-filtering and post-filtering can reduce annotation workload and improve training efficiency.","lead":"This paper tests whether GPT-4 can label emotions in text as well as people do, and finds that human evaluators prefer GPT-4's labels over the original human labels in about 62% of disputed cases. It also proposes two ways to use GPT-4 alongside human annotators: narrowing the label menu before annotation and flagging suspicious labels afterward.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Post-filtering's performance gain in Table 3 is measured on a Filter test set built by the same rule as the Filter training set, so the gain may reflect distribution alignment rather than improved label quality.","rationale":"The reader's weakest assumption is correct and is the most load-bearing gap. The paper's headline preference result (Section 4.4) is well supported: the preference for GPT-4 labels is consistent across three datasets, 53/59 evaluators, and is corroborated by perceived-accuracy ratings; I do not see a comparably specific threat to that central claim. However, the post-filtering experiment is an advertised contribution ('show its potential to flag low-quality labels... improve downstream model learning') and it is the part that most directly proposes a practical pipeline. The Filter test set is not an independent oracle: it uses the same rule that defines the Filter training set, so the Filter model's higher scores on the 'F' column can be explained by distribution alignment. The fact that the Filter-trained model actually underperforms on the human-labeled test set for F1 (BERT: 0.442 vs 0.472) makes the reliance on the filtered test column especially problematic. The paper even states that the annotated set from Section 4.2.1 is the primary test set, but Table 3 does not show it. Reporting that result, or running the independent evaluation, is a concrete and feasible fix. If the independent evaluation also shows gains, the claim lands; if not, the post-filtering claim should be softened. This does not require changing the reader's CONDITIONAL verdict; it points to the exact condition that should be satisfied.","tokens_in":17641,"tokens_out":9606,"duration_ms":90990,"concrete_test":"Re-run the Table 3 evaluation exactly, but with the independent human-validated GoEmotions set from Section 4.2.1 (three crowd annotations per sample, aggregated by majority or at-least-two rule) as the only test set, comparing models trained on Human, Filter, and Random F. If Filter-trained BERT/DistilBERT do not outperform on this independent set, the post-filtering label-quality claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 5.2, the post-filtering claim that GPT-4 can flag low-quality labels and improve downstream model learning rests on Table 3. The 'F' test column is constructed by applying the same exact-overlap rule to the GoEmotions test split: samples with no overlap between GPT-4 and human labels are dropped. Models trained on the Filter set are therefore evaluated on the same truncated label distribution they were trained on, and the higher F1/UAR on the 'F' column could be caused by train/test domain alignment rather than by the removal of bad labels. On the independent human-labeled test set ('H'), the Filter-trained BERT actually has lower F1 than the Human-trained BERT (0.442 vs 0.472), which the paper acknowledges as an exception. The paper's stated primary test set, the human-validated 438-sample set from Section 4.2.1, is not reported in Table 3, so the central applied claim lacks an independent benchmark.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether GPT-4's zero-shot emotion annotations can be used as an alternative or complement to human annotations. It reports a crowdsourcing study in which 59 evaluators compared GPT-4 labels with original human labels on 990 samples drawn from ISEAR, SemEval, and GoEmotions; GPT-4 labels were preferred in about 62% of preference votes (60.7% ISEAR, 58.2% SemEval, 66.4% GoEmotions). The paper then proposes two GPT-4-assisted annotation pipelines: pre-filtering, where GPT-4 reduces the label set shown to human annotators, and post-filtering, where samples with no overlap between GPT-4 and human labels are removed from training data. Experiments on GoEmotions indicate that pre-filtering reduces cognitive load and time while preserving label coverage, and that post-filtering improves downstream model F1 and UAR on the filtered test set. The authors conclude that evaluating LLMs against human labels alone may underestimate their performance and advocate for human-in-the-loop annotation with LLM support.","tokens_in":17874,"tokens_out":4467,"duration_ms":45377,"significance":"If the preference result held on a representative sample, it would meaningfully challenge the common practice of treating human labels as the sole ground truth in emotion recognition. The study's strengths include a real crowdsourcing evaluation with independent evaluators, three datasets with different label spaces, and a concrete demonstration of LLM-assisted pre- and post-filtering. The release of prompts and code is also valuable. However, the preference estimate is computed only on disagreement samples, and the post-filtering evaluation uses a test set constructed by the same filter rule as the training set. These two issues directly limit the strength of the paper's central claims, so the results should be interpreted with caution pending re-analysis.","major_comments":[{"comment":"The 62% preference figure is computed on the 990 samples remaining after dropping all samples where GPT-4 and human labels exactly agree, as well as samples rejected by GPT-4's content policy. Consequently, the \"overall 62%\" (Section 4.4.1) is conditional on disagreement; it does not estimate the fraction of all samples on which GPT-4 is preferred, because agreement samples are excluded by construction. To support the conclusion that evaluating LLMs against human labels is \"prone to underestimate their performance,\" the authors should either explicitly reweight the preference estimate by the proportion of agreement samples in each dataset or restrict the wording to \"among samples where the two sources disagree.\" In addition, the claim of significance is not backed by a statistical test; a mixed-effects logistic regression with evaluator and sample random effects would be appropriate given the nested structure of the data.","section":"§4.2.1, §4.4.1"},{"comment":"The post-filtering performance gain is measured on a \"Filter\" test set that is created by applying the exact-overlap rule (drop samples with no overlapping label between GPT-4 and humans) to the test split, the same rule used to construct the Filter training set. The higher F1/UAR on the F column may therefore reflect train/test distribution alignment on a truncated label space rather than removal of low-quality labels. The independent human-labeled test set (H column) shows the Filter-trained BERT actually has lower F1 than the Human-trained BERT (0.442 vs 0.472), which the paper acknowledges as an exception. The paper states in Section 5.2.1 that its primary test set is the human-validated 438-sample set from Section 4.2.1, but this set is not reported in Table 3. The post-filtering claim needs to be demonstrated on an independently labeled test set that is not constructed by the same overlap rule, or the conclusions should be softened accordingly.","section":"§5.2, Table 3"},{"comment":"The \"Random F\" baseline controls only for the reduction in training-set size; it does not control for the selection rule itself. A more informative comparison would include another quality-based filter, such as removing samples with low inter-annotator agreement in the original GoEmotions data or using annotator confidence scores. Without such a baseline, the observed improvement could be attributed to the removal of inherently ambiguous samples rather than to GPT-4's specific ability to flag low-quality labels. This distinction is important because the paper's stated goal is to evaluate GPT-4 as a quality checker, not merely any filtering heuristic.","section":"§5.2, Table 3"}],"minor_comments":[{"comment":"The pooled rating distributions in Figure 3 and Table 2 ignore the fact that multiple evaluations come from the same sample and the same evaluator. Reporting intraclass correlations or fitting a multilevel model would strengthen the claim that the rating difference is not driven by a few influential evaluators or samples.","section":"§4.4.2, Figure 3"},{"comment":"The logistic regression for GPT-4's weaknesses uses t-test-based feature selection on the same data and then fits the regression on the same data without cross-validation, which risks overfitting. The authors appropriately note the limited sample size, but the analysis should be described as exploratory and the reported p-values interpreted with that caveat.","section":"§4.4.4"},{"comment":"The between-subject design for the pre-filtering study means that differences between the Small, Large, and Pre-Filtered groups could partly reflect annotator-level differences. It would help to report the random assignment procedure and any demographic summary for the three groups, or to run a sensitivity analysis with participant as a random effect.","section":"§5.1.1"},{"comment":"The paper says the human-validated 438-sample set from Section 4.2.1 is the primary test set, but Table 3 does not include results on that set. Please either add those results or clarify why they were omitted, since the current table makes the post-filtering evaluation look weaker than the paper's own description suggests.","section":"§5.2.1"},{"comment":"Some reference formatting issues appear (for example, extra spaces in author initials in the bibliography), and the figure captions for Figure 1 repeat the same confusion matrix across panels. These should be cleaned up in a final version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper reports a well-executed crowdsourcing study and a thoughtful discussion of label-space effects, and I believe the authors can address the concerns above with reanalysis and reframing. The main issue is that the headline preference result and the post-filtering result are both weaker than the text suggests once the sample exclusion rules and test-set construction are taken into account. I would encourage the editor to ask for a revised version that either reweights the preference estimate and evaluates on an independent test set, or substantially softens the conclusions. I do not see evidence of bad faith; these are standard methodological pitfalls in annotation studies."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper does two things: it gives a careful crowdsourced comparison of GPT-4 and human emotion labels, and it tests two ways to use GPT-4 in annotation pipelines. The first half is the stronger half. The finding that human evaluators preferred GPT-4's zero-shot labels over original human labels in 62% of preference votes, consistently across three datasets and across 53 of 59 evaluators, is a genuine and useful result. It's a real extension of the authors' earlier small-scale study, and the analysis of label-space effects is thoughtful.\n\nThe pre-filtering experiment is also solid. Using GPT-4 to dynamically trim the label set before presenting options to annotators reduced options by over 70% while preserving about 90% of the labels humans later chose, with no loss in agreement and lower self-reported mental demand. That is a concrete, practical contribution.\n\nThe soft spots are in the post-filtering section. The preference result is conditional: evaluators only saw samples where human and GPT-4 labels disagreed, so 62% is not a statement about overall label quality. The paper's framing in Section 4.4.1 slightly overstates this. More concerning is Table 3. The \"Filter\" test column is built by the same exact-overlap rule used to build the Filter training set, so the higher F1/UAR there may just reflect that train and test are drawn from the same truncated label distribution, not that the filtering removed bad labels. The independent human-labeled test column actually shows Filter-trained BERT with lower F1 than Human-trained BERT (0.442 vs 0.472). The paper acknowledges this as an exception, but it's the exception that points to the problem. The authors say their primary test set is the human-validated 438-sample set from the earlier section, but that set does not appear in Table 3. Without an independent test set, the label-quality claim is not established.\n\nThe paper is honest about its limitations, and the pre-filtering part stands on its own. I'd like to see a revised version that reports the human-validated set in the post-filtering evaluation and that clearly states the conditional nature of the preference result.\n\nWho gets value: anyone building emotion annotation pipelines, and anyone thinking about whether human labels should be the default ground truth for evaluating LLMs. It deserves a serious referee; the issues are fixable with a cleaner evaluation design.\n\nRecommend: send to peer review, with a clear request for an independent post-filtering test set.","headline":"A useful, honest study of GPT-4 in emotion annotation, but the post-filtering evidence is weaker than claimed because the test set is built with the same rule as the training set.","tokens_in":18344,"tokens_out":2274,"would_cite":true,"duration_ms":20920,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A crowdsourced study finds that human evaluators prefer GPT-4's zero-shot emotion labels over original human annotations in 62% of comparisons, suggesting that benchmarking LLMs against human labels as ground truth underestimates them.","keywords":["emotion annotation","large language models","GPT-4","human evaluation","crowdsourcing","multi-label classification","annotation pipelines","ground truth"],"falsifier":"Retrain the post-filter comparison on a GoEmotions test set annotated by a fresh panel of consensus annotators who do not use the GPT-4 overlap rule, and check whether models trained on the GPT-4-filtered set still beat models trained on the full set; if they do not, the post-filtering quality claim collapses.","tokens_in":17466,"feed_emoji":"🤖","tokens_out":4629,"duration_ms":41745,"temperature":0.7,"pith_summary":"The paper argues that human emotion labels should not be treated as the sole ground truth when evaluating large language models. In a crowdsourced evaluation, independent human evaluators preferred GPT-4's zero-shot emotion annotations over the original human labels in 62% of comparisons overall, with the preference strongest on the largest label space. The authors conclude that evaluating LLMs against human labels alone is prone to underestimation and can produce misleading results. They also propose two ways to integrate GPT-4 into human annotation pipelines: a pre-annotation filter that narrows label choices and a post-annotation filter that flags low-quality samples, reporting benefits in cognitive load, coverage, and downstream training efficiency.","feed_headline":"GPT-4 emotion labels beat human labels in 62% of votes","feed_subtitle":"A crowdsourced study says judging LLMs against human labels may underestimate them and suggests hybrid pipelines.","key_machinery":"The central object is the pairwise human evaluation protocol: evaluators read a text and are shown two label sets, one from GPT-4 zero-shot prompting and one from the original human annotations, without knowing the source, and are asked for a forced preference, a 7-point perceived-accuracy rating, and a confidence judgment. This machinery converts the inherently subjective question of annotation quality into measurable preference votes that can be compared across three datasets with different label-space sizes and domains. For the pipeline experiments, the pre-filtering machinery uses a recall-oriented prompt that asks GPT-4 to mark each candidate label as 'yes' or 'no' to avoid dropping valid labels, while the post-filtering machinery applies a simple exact-overlap rule: samples where GPT-4 and human labels share no labels are removed. The three datasets, ISEAR (7 classes), SemEval (11 classes), and GoEmotions (28 classes), provide controlled variation in label-space complexity.","core_discovery":"The paper establishes that, when shown the same text with two anonymized label sets, human evaluators prefer GPT-4's zero-shot emotion predictions over the original human annotations in 62% of preference votes overall (60.66% for ISEAR, 58.16% for SemEval, and 66.41% for GoEmotions), and that this preference grows with label-space size. It interprets this as evidence that the common practice of benchmarking LLMs against human labels as ground truth underestimates their performance and may mislead conclusions about their capabilities. Complementing this, the paper shows that a GPT-4 pre-filter can remove about 70% of candidate labels while retaining over 90% of labels that human annotators choose from the full set, reducing cognitive load and annotation time without losing coverage. It also shows that a post-filter which drops samples where GPT-4 and human labels share no overlap produces a training set under 40% of the original size on which BERT and DistilBERT achieve better F1 and UAR than on the full human-labeled set, suggesting that GPT-4 can flag low-quality labels and improve training efficiency.","pith_inferences":["If GPT-4's labels are systematically preferred by evaluators, many existing emotion benchmarks encode a particular human perspective that may not match the majority read; future datasets might report multiple label sets or annotator-perspective metadata to make this visible.","The pre-filtering approach likely generalizes to other subjective text-annotation tasks with large label spaces, such as hate-speech or stance detection, though the recall-oriented prompt design would need recalibration for each task's false-negative tolerance.","The paper's finding that preference for GPT-4 grows with label space suggests a testable curve relating preference advantage to the number of candidate classes, which could predict the optimal label-space size for hybrid annotation pipelines.","The post-filtering result, if confirmed on an independently constructed test set, would imply that the effective information content of a training set matters more than its raw size, potentially changing how annotation budgets are allocated."],"forward_implications":["Models evaluated only against original human labels may be underrated, so future evaluations should include human preference judgments alongside agreement metrics.","GPT-4 pre-filtering can make large emotion label spaces practical by cutting cognitive load and annotation time while preserving label coverage, supporting more granular emotion taxonomies.","GPT-4 post-filtering can shrink training data size without sacrificing, and possibly improving, downstream model performance, lowering the cost of training data preparation.","Larger label spaces yield more annotations rated as totally accurate by evaluators for both humans and GPT-4, so label-space design should explicitly weigh granularity against annotator effort and agreement.","Disagreement patterns between GPT-4 and human annotation are asymmetric and dataset-specific, meaning hybrid pipelines should treat the two as complementary perspectives rather than interchangeable sources."],"supporting_citations":[{"why":"The authors' previous small-scale study that first reported human evaluators preferring GPT-4 emotion annotations, which this paper expands with a larger sample, more evaluators, and a crowdsourced protocol.","marker":"[29]"},{"why":"ISEAR dataset used as one of the three evaluation datasets, providing 7-class self-reported emotion labels from cross-cultural participants.","marker":"[35]"},{"why":"SemEval-2018 Task 1 dataset used as the second evaluation dataset, providing 11-class multilabel crowdsourced tweet annotations.","marker":"[36]"},{"why":"GoEmotions dataset used as the third and largest label-space dataset (28 classes), and the dataset on which the pre-filtering and post-filtering pipeline experiments are conducted.","marker":"[37]"},{"why":"Prior work that benchmarks LLMs against human labels and finds underperformance, which the paper cites as an example of the evaluation approach it argues is prone to underestimation.","marker":"[25]"},{"why":"Another human-label-as-ground-truth benchmark of LLM sentiment and emotion analysis, whose conclusions the paper's human-preference results directly contrast with.","marker":"[28]"},{"why":"Work suggesting GPT-4 emulates average-human emotional cognition from a third-person perspective, providing a plausible mechanism for why evaluators prefer GPT-4 labels.","marker":"[47]"},{"why":"Evidence that ChatGPT outperforms crowd workers on text-annotation tasks, motivating the paper's proposal to use LLMs in annotation pipelines.","marker":"[23]"}],"fun_headline_variants":["GPT-4 emotion labels win 62% of human votes","Hybrid human-GPT4 labeling beats human-only emotion data","GPT-4 pre-filter cuts emotion annotation workload by 70%","Human plus GPT-4 labels improve emotion model training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evidence that post-filtering improves label quality assumes the filtered test set is an independent quality benchmark, but that test set is built by the same exact-overlap rule that created the filtered training set, so the two are not independent.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4 emotion labels win 62% of human votes","Hybrid human-GPT4 labeling beats human-only emotion data","GPT-4 pre-filter cuts emotion annotation workload by 70%","Human plus GPT-4 labels improve emotion model training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001583,"raw_usage":{"total_tokens":6337,"prompt_tokens":994,"completion_tokens":5343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":5282}},"tokens_in":610,"tokens_out":5343,"duration_ms":41980,"temperature":1.0,"reasoning_tokens":5282,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:25:58.672871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the post-filter comparison on a GoEmotions test set annotated by a fresh panel of consensus annotators who do not use the GPT-4 overlap rule, and check whether models trained on the GPT-4-filtered set still beat models trained on the full set; if they do not, the post-filtering quality claim collapses.","supporting_citations":[{"cited_title":"Sentiment analysis in the era of large language models: A reality check,","cited_arxiv_id":null,"evidence_quote":"The authors' previous small-scale study that first reported human evaluators preferring GPT-4 emotion annotations, which this paper expands with a larger sample, more evaluators, and a crowdsourced protocol."},{"cited_title":"Dailydialog: A manually labelled multi-turn dialogue dataset,","cited_arxiv_id":null,"evidence_quote":"ISEAR dataset used as one of the three evaluation datasets, providing 7-class self-reported emotion labels from cross-cultural participants."},{"cited_title":"How universal and specific is emotional experience? evidence from 27 countries on five conti- nents,","cited_arxiv_id":null,"evidence_quote":"SemEval-2018 Task 1 dataset used as the second evaluation dataset, providing 11-class multilabel crowdsourced tweet annotations."},{"cited_title":"SemEval-2018 task 1: Affect in tweets,","cited_arxiv_id":null,"evidence_quote":"GoEmotions dataset used as the third and largest label-space dataset (28 classes), and the dataset on which the pre-filtering and post-filtering pipeline experiments are conducted."},{"cited_title":"Affect recognition in conversations using large language models,","cited_arxiv_id":null,"evidence_quote":"Prior work that benchmarks LLMs against human labels and finds underperformance, which the paper cites as an example of the evaluation approach it argues is prone to underestimation."},{"cited_title":"Refashioning emotion recognition modelling: The advent of gen- eralised large models,","cited_arxiv_id":null,"evidence_quote":"Another human-label-as-ground-truth benchmark of LLM sentiment and emotion analysis, whose conclusions the paper's human-preference results directly contrast with."},{"cited_title":"The biases of pre- trained language models: An empirical study on prompt-based sentiment analysis and emotion detection,","cited_arxiv_id":null,"evidence_quote":"Work suggesting GPT-4 emulates average-human emotional cognition from a third-person perspective, providing a plausible mechanism for why evaluators prefer GPT-4 labels."}],"review_version":1}