Pith. sign in

REVIEW 3 major objections 4 minor 20 references

Few-shot text-based emotion detection

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A large few-shot prompt, not reasoning or fine-tuning, wins on low-resource emotion detection.

desk verdict A competent SemEval system description whose Emakhuwa first place is real and whose interpretive claims are softer than the headline. read the letter →

arxiv 2507.05918 v1 pith:4BLXLTRK submitted 2025-07-08 cs.CL

Teodor-George Marchitan , Claudiu Creanga , Liviu P. Dinu This is my paper
classification cs.CL
keywords emotiondetectionfew-shotpromptinglargelanguagemodelsmultilinguallow-resourcelanguagesmulti-labelclassificationpromptengineeringSemEval2025Task11
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper sets out to find the best recipe for a multilingual, multi-label emotion-detection task on three languages with very different amounts of resources. It claims that a simple few-shot prompt — up to 600 labelled example sentences in the prompt, with no requested reasoning steps — is the winning recipe, especially for a low-resource language. The reported result is first place among 31 teams on Emakhuwa with an F1-macro (average per-emotion F1) of 0.3250, alongside 26th of 96 on English and 35th of 36 on Mozambican Portuguese. Few-shot prompting beat zero-shot prompting by about 7 percent F1-macro, and chain-of-thought and tree-of-thought prompting both made scores worse. The practical payoff, if the result holds, is that a language with almost no annotated data can get a competitive emotion detector from a large prompt alone.

What carries the argument

The central mechanism is few-shot in-context learning: each prompt contains a target sentence plus a large pool of labelled examples (up to 600 sentences covering the six emotion categories), and the model is asked to output only the emotions present. The paper varies the pool size from 6 to 600 and shows F1-macro improving at every step for Gemini Flash, rising from 0.69 to 0.77 on the validation set. This mechanism does the argument's work because it isolates demonstration quantity and prompt simplicity as the performance drivers, while the comparisons show that adding explicit reasoning traces or fine-tuning smaller LLMs does not beat it.

What would settle it

Have a second, independent group of annotators re-label a random sample of the Emakhuwa and Mozambican Portuguese test sentences, then score the paper's system against the new labels; if the F1-macro changes enough to alter team rankings, the conclusion is not robust to label subjectivity.

Watch

Extended reading notes

Core claim

The authors report that their final system, Gemini Flash with few-shot prompting, achieves an F1-macro of 0.7546 (26th of 96 teams) on English, 0.1727 (35th of 36) on Mozambican Portuguese, and 0.3250 (1st of 31) on Emakhuwa in the multi-label track. Their controlled comparisons show that few-shot prompting outperforms zero-shot prompting by about 7 percent (Qwen2.5-14B: 0.63 to 0.70 F1-macro), that performance keeps rising as the prompt example set grows from 6 to 100 to 300 to 600, and that chain-of-thought and tree-of-thought prompting degrade performance on the perception-based labels. The best fine-tuned BERT models reach about 0.70 validation F1-macro, below the LLM prompts, while the best fine-tuned LLM (DeepSeek R1 Distill Llama 8B) reaches a test F1-macro of 0.7441 on English, close to but below the Gemini few-shot result. The error analysis shows a consistent tendency to over-predict fear and under-predict joy in both the prompting and fine-tuning approaches. The authors attribute the chain-of-thought failure to the subjectivity of 'perceived emotion' labels, where a forced reasoning path can diverge from the annotator's interpretation even when the model's alternative reading is plausible.

Load-bearing premise

The reported scores stand only if the official test labels for perceived emotion — which are individual annotators' subjective judgments — are accepted as the ground truth.

Editorial extensions

If this is right

  • A language with almost no annotated resources can get usable emotion detection from a large few-shot prompt instead of a fine-tuning run.
  • Prompt-design effort on subjective tasks should validate reasoning prompts per task; here chain-of-thought and tree-of-thought actively lowered F1-macro.
  • Scaling the demonstration count is a cheap and repeatable lever: Gemini Flash improved from 0.69 with 6 examples to 0.77 with 600 on validation.
  • Fine-tuning a local model can come close to an API-prompted system on high-resource English (0.7441 versus 0.7546 test F1-macro), leaving a cost-performance trade-off.
  • The same recipe does not transfer uniformly: the system placed 35th of 36 on Mozambican Portuguese, so language-specific factors still dominate.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If perceived-emotion labels are subjective, an evaluation that compares systems against a single gold label may hide how much of the ranking is annotator-specific; the paper's own observation that diverging model reasoning was often plausible points toward evaluating against multiple annotators.
  • The fear-over-prediction and joy-under-prediction pattern appears in both prompting and fine-tuning, which suggests a dataset-level label imbalance rather than a model-specific failure; rebalancing prompt examples or adjusting decision thresholds is a direct test.
  • A natural extension is to run the same 600-example protocol on the other low-resource languages in the dataset to see whether Emakhuwa's first-place result is repeatable or an outlier.
  • The near-bottom Mozambican Portuguese score suggests example quality and model familiarity with the language variety matter more than prompt size; swapping Emakhuwa examples into the Portuguese prompt would test cross-lingual transfer directly.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper describes the Unibuc-NLP system for SemEval-2025 Task 11, Track A (multi-label emotion detection). The authors experiment with three approaches: fine-tuning BERT-family models (mBERT, DeBERTa, XLM-RoBERTa), few-shot prompting of LLMs (Gemini Flash, Qwen2.5, DeepSeek), and LoRA fine-tuning of LLMs (DeepSeek-R1-Distill-Llama-8B, Mistral 7B, Qwen2.5 variants). Their final submitted system is Gemini 2.0 Flash with few-shot prompting using 600 examples, achieving official test F1-macro scores of 0.7546 (26/96) for English, 0.1727 (35/36) for Mozambican Portuguese, and 0.325 (1/31) for Emakhuwa. The paper also reports that few-shot prompting outperforms zero-shot prompting by about 7%, that more examples improve performance, and that Chain-of-Thought and Tree-of-Thought prompting hurt performance, which the authors attribute to the subjective nature of perceived emotion annotation.

Significance. The paper is a shared-task system report whose main value is the official test-set results and the concrete recipe behind them. The first-place Emakhuwa result (F1-macro 0.325, 1/31 teams) is a meaningful empirical data point for a low-resource language, and the paper transparently reports LoRA hyperparameters, prompt templates, and training details. The contribution is not methodological: the few-shot prompting and LoRA fine-tuning recipes are standard, and the paper's interpretive claims about prompting strategy and low-resource adaptation are weakened by uncontrolled comparisons and by the paper's own unquantified caveat about label reliability. If the official scores are accurate, the paper still provides a useful benchmark for SemEval-2025 participants, but the stated conclusions require additional controlled experiments and label-quality analysis.

major comments (3)
  1. [Section 3.2, Table 3] The evidence for the claim that CoT and ToT degrade performance is confounded by model choice and example count. The zero-shot CoT row is for Qwen2.5, whereas the ToT rows are for Gemini; within Gemini, ToT is tested only with 6 and 20 examples, while the best few-shot setting uses 500 or 600 examples. To support the conclusion that CoT/ToT make models perform worse, the authors should compare plain few-shot, few-shot CoT, and few-shot ToT with the same base model and the same number of examples across the 6-to-600 range.
  2. [Section 3.2] The qualitative statement that the models were 'conviced that the models were right in most of the cases' when they disagreed with the ground-truth labels is unquantified. The paper reports no inter-annotator agreement, no mislabeling rate, no sample size, and no independent adjudication for English, Mozambican Portuguese, or Emakhuwa. Because the headline Emakhuwa F1-macro of 0.325 (rank 1/31) is computed against those labels, this claim needs quantitative support or removal; otherwise the official score's interpretation as a measure of emotion-detection quality is insecure.
  3. [Section 4, Table 6] The baseline comparison is incomplete. The text states that the system was above the baseline on two of three languages, but the baseline values are never reported, and the very poor Mozambican Portuguese result (0.1727, rank 35/36) is left unexplained beyond 'language-specific nuances'. Section 5's conclusion about adaptation to low-resource languages should be accompanied by the official baseline numbers and a concrete analysis of why Emakhuwa succeeds while Mozambican Portuguese fails, given that both are low-resource varieties in the same task.
minor comments (4)
  1. [Table 3] The claim that increasing examples from 6 to 100, 300, and 600 improves performance is not fully supported by the table, which shows only 6, 500, and 600; the authors should either include the missing intermediate points or state which example counts were actually tested.
  2. [Table 5] The caption says 'results on both train and validation datasets', but the columns include Test F1 Micro and Test F1 Macro; the caption should state that the test columns are the official test-set scores.
  3. [Section 3.1] The model is described as having 6 output neurons 'corresponding to the 6 emotion categories', but Appendix A lists five named emotions (Anger, Fear, Joy, Sadness, Surprise) plus 'None'; the authors should clarify that the sixth category is the absence of an emotion.
  4. [Throughout] There are several typos and inconsistencies: 'exemples' in Table 3's caption, 'quanitized' in Section 3.3, mixed model identifiers ('Qwen2.5' vs 'Qwen2.5-14B', 'Gemini 2.0 Flash Exp'), and the GitHub repository is mentioned without a URL; these should be cleaned up.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: results are official external benchmark scores, and the few self-citations are routine training-recipe references that do not define the outcome.

full rationale

The paper is a SemEval-2025 system description. Its central claims are the official F1-macro scores and ranks on the external BRIGHTER test sets, which are measured against a public benchmark rather than derived from the paper's own definitions or fitted parameters. The few-shot prompting and fine-tuning configurations are chosen based on validation-set experiments (Tables 2, 3, 4, 5), and the final test results (Table 6) come from the official evaluation. There is no step in which a parameter fitted to a subset is later renamed as a prediction, and no equation in the paper defines the claimed result in terms of its inputs. The self-citations (Marchitan et al., 2024; Creanga and Dinu, 2024) are referenced only for training recipes such as the two-stage fine-tuning procedure and truncation strategy; they do not supply the benchmark scores, the emotion taxonomy, or the evaluation labels. The authors' observation in §3.2 that models sometimes appeared right when disagreeing with the ground truth raises a possible data-quality concern about the gold labels, but that is a validity-of-evaluation issue, not a circularity of derivation. Consequently, the circularity burden is zero.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on official dataset labels, undocumented few-shot example selection, and a set of training hyperparameters. No new entities are introduced and no mathematical derivation is attempted.

free parameters (3)
  • Number of few-shot examples = 600
    Selected on validation; Table 3 shows F1 improves from 0.69 with 6 examples to 0.77 with 600.
  • Prompt template and example ordering = Not specified
    The exact 600 examples and their ordering are not given, yet the result depends on them; this is a hand-chosen configuration without reported variance.
  • LoRA hyperparameters (r, alpha, dropout, LR) = r=4, alpha=8, dropout=0.05, LR varied
    For the fine-tuned baselines, values were chosen per model; they affect the reported fine-tuning scores.
assumptions (3)
  • domain assumption BRIGHTER perceived-emotion labels are the gold standard
    The paper accepts the annotations as ground truth for evaluation and notes they are subjective perceptions, not objective facts.
  • domain assumption The six emotion categories are sufficient for all test texts
    The task fixes the label set; the system only predicts from this list, including 'None'.
  • domain assumption The official SemEval scoring was applied correctly
    The paper reports the official F1-macro values without reproducing the scoring script; any scoring bug would change the claim.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Few-shot text-based emotion detection." pith.science (2026). https://pith.science/paper/4BLXLTRK

@misc{pith2026250705918,
  author       = {Pith},
  title        = {Pith review of: Few-shot text-based emotion detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4BLXLTRK}},
  note         = {Machine review of arXiv:2507.05918}
}
abstract

This paper describes the approach of the Unibuc - NLP team in tackling the SemEval 2025 Workshop, Task 11: Bridging the Gap in Text-Based Emotion Detection. We mainly focused on experiments using large language models (Gemini, Qwen, DeepSeek) with either few-shot prompting or fine-tuning. With our final system, for the multi-label emotion detection track (track A), we got an F1-macro of $0.7546$ (26/96 teams) for the English subset, $0.1727$ (35/36 teams) for the Portuguese (Mozambican) subset and $0.325$ (\textbf{1}/31 teams) for the Emakhuwa subset.

Figures

Figures reproduced from arXiv: 2507.05918 by the authors.

Figure 1
Figure 1. Track A: Distribution of token length for the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Confusion Matrix for Fear (Counts and Per [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Confusion Matrix for Joy (Counts and Per [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Confusion Matrix for Anger (Counts and Per [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Confusion Matrix for Sadness (Counts and [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Confusion Matrix for Surprise (Counts and [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Comparison of F1 macro score on each emotion between best model using few-shot prompt￾ing (Gemini 2.0 Flash) and fine-tuning (DeepSeek R1 Distill Llama 8B) [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

20 extracted references · 3 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/arXiv:1911.02116 Unsupervised cross-lingual representation learning at scale

    arXiv 2019

  4. [4]

    Claudiu Creanga and Liviu P. Dinu. 2024. https://doi.org/10.18653/v1/2024.semeval-1.95 ISDS - NLP at S em E val-2024 task 10: Transformer based neural networks for emotion recognition in conversations . In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 649--654, Mexico City, Mexico. Association for Computationa...

    work page doi:10.18653/v1/2024.semeval-1.95 2024

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. http://arxiv.org/abs/arXiv:1810.04805 Bert: Pre-training of deep bidirectional transformers for language understanding

    arXiv 2018

  6. [6]

    DeepSeek-AI et. al. 2025. http://arxiv.org/abs/2501.12948 Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    arXiv 2025

  7. [7]

    Gemini Team et. al. 2024. http://arxiv.org/abs/2403.05530 Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    arXiv 2024

  8. [8]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://openreview.net/forum?id=XPZIaotutsD Deberta: Decoding-enhanced bert with disentangled attention . In International Conference on Learning Representations

    2021

Show all 20 references
  1. [9]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  2. [10]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  3. [11]

    Yoon Kim. 2014. https://doi.org/10.3115/v1/D14-1181 Convolutional neural networks for sentence classification . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1746--1751, Doha, Qatar. Association for Computational Li...

    2014 doi

  4. [12]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. http://arxiv.org/abs/arXiv:1907.11692 Roberta: A robustly optimized bert pretraining approach

    2019 arXiv

  5. [13]

    Jieyi Long. 2023. http://arxiv.org/abs/arXiv:2305.08291 Large language model guided tree-of-thought

    2023 arXiv

  6. [14]

    Ilya Loshchilov and Frank Hutter. 2019. http://arxiv.org/abs/1711.05101 Decoupled weight decay regularization

    2019 arXiv

  7. [15]

    Teodor-george Marchitan, Claudiu Creanga, and Liviu P. Dinu. 2024. https://doi.org/10.18653/v1/2024.semeval-1.63 Team U nibuc - NLP at S em E val-2024 task 8: Transformer and hybrid deep learning based models for machine-generated text detection . In Proceedings of the 18th In...

    2024 doi

  8. [16]

    Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine de Kock, Nirmal Surange, Daniela Teodorescu, Ibrahim Said Ahmad, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino D. M. A. Ali, Ilseyar Alimova, Vladimir...

    2025 arXiv

  9. [17]

    Mohammad

    Shamsuddeen Hassan Muhammad, Nedjma Ousidhoum, Idris Abdulmumin, Seid Muhie Yimam, Jan Philip Wahle, Terry Ruas, Meriem Beloucif, Christine De Kock, Tadesse Destaw Belay, Ibrahim Said Ahmad, Nirmal Surange, Daniela Teodorescu, David Ifeoluwa Adelani, Alham Fikri Aji, Felermino...

    2025

  10. [18]

    Soujanya Poria, Erik Cambria, Devamanyu Hazarika, Navonil Majumder, Amir Zadeh, and Louis-Philippe Morency. 2017. https://doi.org/10.18653/v1/P17-1081 Context-dependent sentiment analysis in user-generated videos . In Proceedings of the 55th Annual Meeting of the Association f...

    2017 doi

  11. [19]

    Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, ...

    2025 arXiv

  12. [20]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. http://arxiv.org/abs/arXiv:2201.11903 Chain-of-thought prompting elicits reasoning in large language models

    2022 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.