{"id":"68325cf3-9dff-4012-94dc-f0548cff7ca9","arxiv_id":"2607.00968","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Frontier LLMs reach a shared zero-shot ceiling of ~39% accuracy on 13-class emotion classification with Gemini highest but no pairwise statistical differences.","lead":"This paper tests three frontier LLMs on zero-shot classification of 1,000 sentences into 13 fine-grained emotions using one identical prompt. All models reach roughly 39% accuracy with no significant differences, exposing consistent failures on emotions like love and shame.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Single uniform prompt + 1k stratified sample may not produce unbiased estimate of true zero-shot ceiling on 13-class task","rationale":"The reader’s weakest_assumption directly identifies the same load-bearing point; the abstract-only limitation is noted but does not alter the technical concern. No other internal inconsistency (e.g., statistical test misuse or dataset description) rises to the same level of centrality for the convergence claim.","tokens_in":1817,"tokens_out":359,"duration_ms":22988,"concrete_test":"Re-evaluate all three models on the identical 1000 sentences using two additional prompt variants (one that adds explicit “output only the emotion label” instruction and one that lists classes in random order); recompute accuracies, macro-F1, and McNemar p-values. If relative ordering or significance changes by more than the original 1–2 point accuracy gap, the ceiling claim is prompt-dependent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result (Gemini 39.9% acc / 0.363 macro-F1, no pairwise McNemar significance at p>0.10, convergence at shared ceiling) depends on the claim that one fixed zero-shot prompt applied to a stratified 1000-sentence draw from boltuix/emotions yields a representative performance snapshot. Because the three models differ sharply in macro-F1 (0.363 vs 0.291 vs 0.159) while accuracies are close, any prompt-specific bias or sampling artifact that affects class-wise calibration would directly alter the “no significant difference” and “shared ceiling” conclusions. The paper does not report prompt ablations, multiple independent samples, or label-quality checks on the source dataset.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper conducts a zero-shot evaluation of three commercial LLMs—Claude (claude-sonnet-4-6), GPT-5.4, and Gemini (gemini-2.5-flash)—on a 13-class fine-grained emotion classification task. Using a single uniform prompt without exemplars on a stratified sample of 1,000 sentences from the boltuix/emotions dataset (131,306 sentences total), it reports Gemini achieving the highest accuracy of 39.9% and macro-F1 of 0.363, followed by GPT-5.4 (38.8%, 0.291) and Claude (38.0%, 0.159). McNemar tests show no significant pairwise differences (p > 0.10), leading to the conclusion of convergence at a shared zero-shot performance ceiling, with models performing well on sarcasm and desire but poorly on love, confusion, and shame. Claude shows class-imbalance bias.","tokens_in":1996,"tokens_out":603,"duration_ms":27853,"significance":"If the empirical results are robust, the paper provides a timely benchmark demonstrating the limitations of current frontier LLMs in zero-shot fine-grained affective computing tasks. This has direct relevance for applications in conversational AI and mental health, quantifying an 'affective gap' that suggests further advances are needed beyond current model scales and training paradigms.","major_comments":[{"comment":"The central claim of a 'shared zero-shot ceiling' and lack of statistically significant differences depends on the representativeness of the single uniform prompt and the 1,000-sentence stratified sample. The manuscript does not include the exact prompt text, any prompt ablations, or results from multiple independent samples, which leaves open the possibility that prompt-specific or sampling artifacts drive the observed macro-F1 variations (0.363 vs 0.291 vs 0.159) and the McNemar outcomes.","section":"Evaluation Setup"},{"comment":"The reported McNemar tests (p > 0.10) are used to support no pairwise differences, but without accompanying confusion matrices, per-class precision/recall breakdowns, or error analysis on the 13 classes, it is difficult to assess whether the close accuracy scores reflect true convergence or masking effects from class imbalances and prediction biases noted for Claude.","section":"Results"}],"minor_comments":[{"comment":"The dataset citation 'boltuix/emotions' should include a reference or link to the source for reproducibility.","section":"Abstract"},{"comment":"Model version names like 'GPT-5.4' and 'gemini-2.5-flash' appear non-standard; clarify if these are placeholders or specific API endpoints used in April 2026.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for these constructive comments, which help clarify the presentation of our evaluation. We respond to each major comment below, indicating planned revisions where appropriate.","responses":[{"response":"We will include the exact prompt text verbatim in the revised Methods section for full reproducibility. The single uniform prompt was intentionally selected to enable a controlled, apples-to-apples zero-shot comparison across the three models without introducing variability from different prompting techniques or few-shot examples. The 1,000-sentence sample was drawn via stratified sampling to preserve the class distribution of the full 131,306-sentence dataset. While prompt ablations and repeated independent samples would provide additional robustness checks, they fall outside the scope of the current study, which focuses on convergence under a standardized zero-shot protocol. We will add explicit discussion of these design decisions and their limitations in the revised manuscript.","revision_made":"partial","referee_comment":"[Evaluation Setup] The central claim of a 'shared zero-shot ceiling' and lack of statistically significant differences depends on the representativeness of the single uniform prompt and the 1,000-sentence stratified sample. The manuscript does not include the exact prompt text, any prompt ablations, or results from multiple independent samples, which leaves open the possibility that prompt-specific or sampling artifacts drive the observed macro-F1 variations (0.363 vs 0.291 vs 0.159) and the McNemar outcomes."},{"response":"We agree that the addition of these materials will improve interpretability. In the revised version we will append full confusion matrices and per-class precision, recall, and F1 scores for all three models. We will also expand the Results and Discussion sections with a targeted error analysis covering the classes where models perform well (sarcasm, desire) and poorly (love, confusion, shame), along with further detail on Claude's observed class-imbalance bias. The McNemar tests remain appropriate for assessing paired accuracy differences on identical instances; the macro-F1 scores already surface distributional differences that we discuss as evidence of bias despite non-significant accuracy gaps.","revision_made":"yes","referee_comment":"[Results] The reported McNemar tests (p > 0.10) are used to support no pairwise differences, but without accompanying confusion matrices, per-class precision/recall breakdowns, or error analysis on the 13 classes, it is difficult to assess whether the close accuracy scores reflect true convergence or masking effects from class imbalances and prediction biases noted for Claude."}],"tokens_in":1542,"tokens_out":538,"duration_ms":24036,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The core result is that Gemini, GPT-5.4, and Claude all land between 38-40% accuracy on the boltuix/emotions 13-class sample under a single uniform zero-shot prompt, with McNemar tests showing no statistical differences. They handle sarcasm and desire better than love, confusion, or shame, and Claude shows more class-imbalance bias in its predictions.\n\nWhat the paper does cleanly is run the three models through identical API calls on a public stratified 1k subset, report accuracy plus macro-F1, and apply a standard pairwise test. That gives a concrete snapshot of current commercial model behavior on fine-grained affect.\n\nThe soft spot is the evaluation design itself. The shared-ceiling claim and the \"no significant difference\" conclusion rest on one fixed prompt and one 1000-sentence draw. The macro-F1 spread (0.363 down to 0.159) already hints that class-wise calibration differs across models, so a different prompt or another sample could move the numbers and the statistical outcome. No prompt variants, no repeated draws, and no label-quality check on the source data are shown, which leaves the representativeness open.\n\nThis is useful for affective-computing groups that need a quick current baseline on this taxonomy. It is not a methods paper and does not claim broader theoretical reach. A serious editor should send it to review so the prompt and sampling choices can be stress-tested, but it does not require major new experiments to be worth referee time.","headline":"Three frontier LLMs top out near 39% accuracy on a 13-class emotion task with one zero-shot prompt, showing no significant pairwise differences but clear macro-F1 gaps.","tokens_in":2520,"tokens_out":389,"would_cite":false,"duration_ms":14029,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Three leading LLMs reach similar accuracies around 39 percent on zero-shot 13-class emotion classification with no significant differences.","keywords":["zero-shot evaluation","emotion classification","large language models","affective computing","fine-grained taxonomy","LLM performance ceiling","13-class emotions"],"falsifier":"Repeating the evaluation on a fresh stratified sample or with an altered prompt template and finding that one model pulls ahead by a statistically significant margin.","tokens_in":2741,"feed_emoji":"🤖","tokens_out":723,"duration_ms":22808,"temperature":0.7,"pith_summary":"The paper tests Claude, GPT-5.4, and Gemini on assigning sentences to one of 13 specific emotion categories using only a single prompt and no examples. It reports that the three models post nearly identical results, with Gemini at 39.9 percent accuracy, GPT-5.4 at 38.8 percent, and Claude at 38.0 percent, and statistical tests confirm the differences are not meaningful. The models succeed on sarcasm and desire yet fail on love, confusion, and shame, while Claude shows extra bias toward majority classes. A reader would care because emotion classification supports mental-health tools and conversational systems that need reliable fine-grained understanding. The work therefore frames current frontier models as having hit a common zero-shot limit on this taxonomy.","feed_headline":"LLMs converge at 39% on 13-class zero-shot emotion task","feed_subtitle":"Gemini, GPT-5.4 and Claude post statistically indistinguishable results, exposing a shared performance limit.","key_machinery":"Uniform zero-shot prompt applied identically across models to the stratified sample, evaluated by accuracy and macro-F1 on the 13-class taxonomy.","core_discovery":"On a stratified 1,000-sentence sample drawn from the 13-class boltuix/emotions dataset, a single uniform zero-shot prompt produces accuracies of 39.9 percent for Gemini, 38.8 percent for GPT-5.4, and 38.0 percent for Claude, with macro-F1 scores of 0.363, 0.291, and 0.159 respectively; McNemar tests find no pairwise differences at p greater than 0.10, indicating the models converge at a shared zero-shot ceiling while Claude additionally exhibits class-imbalance bias.","pith_inferences":["The ceiling may indicate that zero-shot prompting alone is insufficient for reliable use in affective applications without further adaptation.","Testing whether few-shot examples or task-specific fine-tuning lift performance above this level would directly test the limit.","Mental-health and conversational systems relying on these models may need hybrid approaches that combine LLMs with other signals."],"forward_implications":["Gemini, GPT-5.4, and Claude show no statistically significant accuracy differences on the task.","All three models perform well on sarcasm and desire but consistently fail on love, confusion, and shame.","Claude's lower macro-F1 reveals a stronger class-imbalance prediction bias than the other two models.","Current frontier models appear to share a performance ceiling in zero-shot fine-grained emotion classification."],"fun_headline_variants":["Zero-shot LLMs converge at 39% on 13 emotions","LLMs reach shared 39% limit in emotion classification","13-class emotion task exposes 39% LLM ceiling","Major LLMs post similar 39% scores on emotions"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The single uniform prompt and the 1,000-sentence stratified sample produce an unbiased estimate of each model's true zero-shot capability on the full 13-class taxonomy.","fun_headline_variants_meta":{"raw":{"variants":["Zero-shot LLMs converge at 39% on 13 emotions","LLMs reach shared 39% limit in emotion classification","13-class emotion task exposes 39% LLM ceiling","Major LLMs post similar 39% scores on emotions"]},"model":"grok-4.3","cost_usd":0.00905,"raw_usage":{"total_tokens":4110,"prompt_tokens":765,"num_sources_used":0,"completion_tokens":59,"cost_in_usd_ticks":90499500,"prompt_tokens_details":{"text_tokens":765,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3286,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":765,"tokens_out":59,"duration_ms":31802,"temperature":1.0,"reasoning_tokens":3286,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-02T13:00:53.926484+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Repeating the evaluation on a fresh stratified sample or with an altered prompt template and finding that one model pulls ahead by a statistically significant margin.","supporting_citations":[],"review_version":1}