REVIEW 4 major objections 5 minor 27 references
L3Cube-MahaEmotions: A Marathi Emotion Recognition Dataset with Synthetic Annotations using CoTR prompting and Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper claims that generic LLMs such as GPT-4 and Llama3-405B generalize better than fine-tuned BERT models for complex low-resource emotion recognition, demonstrated on a new 11-label Marathi benchmark where GPT-4 with…
desk verdict Useful new Marathi emotion dataset, but the LLM-beats-BERT claim is confounded by training BERT on synthetic labels. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is Chain-of-Translation (CoTR) prompting, a single prompt that first translates a Marathi sentence into English and then predicts one of eleven emotion labels (Happiness, Sadness, Anger, Fear, Surprise, Disgust, Excitement, Pride, Respect, Sarcasm, Neutral). It works by letting the LLM classify in English, where its language understanding is stronger, before the result is assigned to the original Marathi sentence. The surrounding machinery is the dataset pipeline: 15,000 sentences from the MahaSent sentiment corpus, GPT-4 producing the 12k training labels, and manually labeled 1.5k validation and 1.5k test sets serving as the evaluation ground truth.
What would settle it
Have a fresh set of annotators independently re-label the 1,500 test sentences and measure inter-annotator agreement; if agreement is low or the new labels differ substantially from the paper's gold labels, the reported gaps between GPT-4 and BERT could shrink or reverse. Alternatively, train MahaBERT on a larger, fully human-labeled training set (matched for genre) and compare to GPT-4; if the fine-tuned BERT then matches or exceeds GPT-4, the claim that generic LLMs generalize better would fail.
Extended reading notes
Core claim
The central discovery is the consistent advantage of generic LLMs over fine-tuned BERT on a gold-standard Marathi emotion test set. GPT-4 with Chain-of-Translation prompting scores 0.86 accuracy and 0.86 F1, compared with 0.63/0.64 for MahaBERT-V2, 0.59/0.60 for MuRIL, and 0.70/0.72 for Llama3-405B with CoTR. The gap persists when MahaBERT is trained on the 12,000 synthetic labels generated by GPT-4, which the authors interpret as evidence that fine-tuning on automatically annotated data does not necessarily distill the LLM's ability. They also show CoTR itself is responsible for part of the gain: GPT-4's accuracy rises from 0.83 to 0.86 when Marathi is translated to English before classification.
Load-bearing premise
The load-bearing premise is that the 1,500-sentence manually labeled test set is a reliable gold standard; the paper declares it ground truth without reporting inter-annotator agreement or adjudication, so if these labels are noisy or inconsistent, every accuracy and F1 comparison in the paper loses its foundation.
Editorial extensions
If this is right
- For low-resource languages, a practical labeling recipe is to use a generic LLM with CoTR on a small human-checked evaluation set rather than spending on large-scale manual annotation.
- Fine-tuning a domain BERT model on LLM-generated labels should not be assumed to beat the teacher; the student can remain worse even when the labels come from a strong model.
- Translation-based prompting is a simple, reusable intervention for multilingual LLM tasks: it moved GPT-4 from 0.83 to 0.86 accuracy on this test set.
- The new 11-label benchmark gives the Marathi NLP community a common test set for comparing emotion recognition approaches.
Reading between the lines
- A testable extension is to measure inter-annotator agreement on the gold test set; the paper's comparisons are only as strong as the consistency of that 1,500-sentence annotation, which is not reported.
- The reported evaluation merges Anger and Disgust into one class; re-running the comparison with the full 11-class scheme could change the magnitude of the GPT-4 lead.
- If the LLM advantage generalizes beyond Marathi, then the marginal value of expensively curated training corpora in other low-resource languages may be smaller than commonly assumed, at least for fine-grained emotion tasks.
- A direct way to separate noise from task complexity would be to denoise the 12k synthetic labels (e.g., by consistency filtering) and retrain MahaBERT; if the student then approaches GPT-4, residual label noise is the likely culprit.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces L3Cube-MahaEmotions, a Marathi emotion recognition dataset built from MahaSent-GT, containing roughly 12,000 training sentences annotated by GPT-4 using Chain-of-Translation (CoTR) prompting and 1,500-sentence validation and test sets that were manually labeled. The authors benchmark GPT-4, Llama3-405B, MahaBERT-V2, and MuRIL on the test set, reporting that GPT-4 with CoTR reaches 0.86 accuracy/F1, while MahaBERT-V2 and MuRIL reach 0.63/0.64 and 0.59/0.60, respectively. The abstract's central claim is that generic LLMs like GPT-4 and Llama3-405B generalize better than fine-tuned BERT for complex low-resource emotion recognition, with BERT models trained on GPT-4's synthetic labels failing to surpass GPT-4. The paper also claims an absolute 6% improvement from CoTR prompting over direct prompting.
Significance. If the claims were fully supported, the dataset would be a useful public resource for Marathi emotion recognition, and the LLM-versus-BERT comparison would be an interesting empirical finding for low-resource NLP. The authors deserve credit for releasing the dataset and models, for manually constructing a held-out test set, and for applying the CoTR prompting idea from their prior work in a new language/task setting. However, the headline claim about LLMs generalizing better than fine-tuned BERT is not established by the current experimental design because the BERT models were trained on GPT-4's synthetic labels rather than on clean human labels. The gold-standard reliability, the 11-label claim, and the CoTR improvement magnitude also need attention. The resource contribution may survive these issues, but the paper's central scientific message requires substantial rework.
major comments (4)
- [§4.2, Table 3; §3.4; §1] The central claim that generic LLMs generalize better than fine-tuned BERT is confounded by training-data quality. MahaBERT-V2 and MuRIL were fine-tuned on the 12,000-sentence training set labeled by GPT-4 (Sections 3.4 and 4.2), while GPT-4 and Llama3-405B were evaluated directly on the human-labeled test set. The comparison therefore conflates architecture with the source of the training labels; the result shows at most that GPT-4's synthetic labels do not distill effectively into BERT, not that LLMs generalize better than BERT in general. The paper itself acknowledges that residual noise in the training labels may hinder BERT generalization (Section 1 and Limitations). To support the abstract's claim, the authors should fine-tune BERT models on human-labeled training data matched in size and domain, or explicitly reframe the claim as 'LLM-synthetic annotations do not distill into fine-tuned BERT.'
- [Abstract, §3.2, Table 3] The paper consistently promises 11 fine-grained emotion labels, but the note to Table 3 states that Anger and Disgust are merged into a single class during both training and evaluation. This means the main benchmark evaluates 10 effective classes, so the '11 fine-grained emotion labels' contribution and the associated accuracy/F1 numbers are not for the full taxonomy. Additionally, Section 3.4 reports MahaBERT-V2 accuracy 63% and F1 0.47, while Table 3 reports F1 0.64, and MuRIL is reported as 60% accuracy/F1 0.42 in Section 3.4 but 0.59/0.60 in Table 3. These discrepancies need reconciliation, and the 11-class results should be reported or the abstract revised.
- [§4.1, §3.2] The manually labeled validation and test sets are declared to be ground truth, but no inter-annotator agreement, annotation guidelines, annotator qualifications, or adjudication procedure are reported. The annotation rule 'the strongest or most obvious emotion is selected' (Section 3.2) is highly subjective for a fine-grained 11-class scheme, especially for subtle or mixed-emotion sentences. Without reliability evidence, every accuracy and F1 comparison in Table 3 rests on an unvalidated gold standard. The authors should report at least Cohen's kappa or Krippendorff's alpha on a subset, and describe how disagreements were resolved.
- [Abstract, Contributions, Table 3] The claimed 'absolute 6% improvement in the GPT-4 performance using CoTR prompting' is not supported by Table 3, which shows GPT-4 at 0.83 accuracy and 0.83 F1 versus GPT-4 (CoTR) at 0.86 accuracy and 0.86 F1—a 3 percentage point improvement, not 6. The underlying numbers should be corrected, or the non-CoTR result that produces the 6% figure should be reported. In addition, the abstract mentions label aggregation strategies such as Union and Intersection, but these are never defined or analyzed in the paper; only an 'At Least One Correct' count appears in Table 2.
minor comments (5)
- [Table 1 / Table 2 captions] Table 1 is captioned 'Number of samples per emotion label' but its body contains GPT-4/Llama correctness statistics; either retitle the table or provide the actual label distribution, which would also help the reader assess class imbalance.
- [§3.4] The model is called 'GPT-4o' in Section 3.4 but 'GPT-4' throughout the rest of the paper, including Table 3 and the abstract; please clarify which exact model version was used for annotation and evaluation.
- [Throughout] Model naming is inconsistent: the paper uses 'Llama 3.1 405B', 'Llama3-405B', and 'LLaMA-3'; please standardize the references.
- [§3.4 / §4.2] No training details are given for MahaBERT-V2 and MuRIL (e.g., number of epochs, learning rate, batch size, maximum sequence length), which limits reproducibility of the BERT baseline results.
- [§4.2 / Table 3] The paper reports point estimates but no statistical significance testing; on a 1,500-example test set, the 0.86 versus 0.83 difference between GPT-4 (CoTR) and GPT-4 may be within noise, so a paired test such as McNemar's would strengthen the comparison.
Circularity Check
Headline LLM-vs-BERT comparison is confounded: BERT is trained on GPT-4's own labels, so 'BERT fails to surpass GPT-4' is partly built into the evaluation design.
-
fitted input called prediction
[Abstract; Section 4.2 (GPT-4 vs Llama3-405B), Table 3; Section 3.4 Models Used]
"Given the OR of both models’ predictions is similar with GPT-4’s performance (1384 for OR vs. 1284 for GPT), we decided to tag the training data exclusively using GPT-4 for the final classifier model. After annotation, we trained a classifier on the GPT-labeled dataset. ... While GPT-4 predictions outperform fine-tuned BERT models, BERT-based models trained on synthetic labels fail to surpass GPT-4. ... An important finding of this work is that generic LLMs like GPT-4 and Llama3-405B generalize better than fine-tuned BERT for complex low-resource emotion recognition tasks."
The BERT baselines (MahaBERT-V2, MuRIL) are fitted to GPT-4's own predictions, because the training data is synthetically annotated by GPT-4, while GPT-4 itself is evaluated on the human gold test set. The comparison is therefore 'GPT-4 vs a model distilled from GPT-4's labels', not an independent test of architecture. The BERT models never receive clean human supervision, so their test accuracy is statistically governed by the quality of GPT-4's labels; the reported failure to surpass GPT-4 is an artifact of the annotation pipeline rather than evidence that LLMs generalize better. The abstract's 'important finding' thus largely restates the training-data construction.
full rationale
The dataset contribution is independently anchored to a human-labeled validation/test benchmark, and the CoTR-vs-non-CoTR comparison is a genuine empirical test on the same gold test set, so the CoTR self-citation (Deshpande et al., 2024) is not load-bearing circularity. However, the headline claim that 'generic LLMs like GPT-4 and Llama3-405B generalize better than fine-tuned BERT' is supported by an experiment in which the BERT models are trained exclusively on GPT-4's synthetic labels. Because the only supervision available to BERT is GPT-4's output, the observation 'BERT fails to surpass GPT-4' is partly predetermined by the design; it does not isolate architectural generalization. This is a fitted-input-called-prediction pattern and warrants a partial-circularity score of 6 rather than 2. The paper's 'absolute 6% improvement' claim for CoTR vs the 0.83-to-0.86 (3-point) gain in Table 3 is a numerical inconsistency, but that is a reporting/correctness issue, not circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption Human-annotated validation and test labels are treated as a reliable gold standard, but no inter-annotator agreement is reported.
- domain assumption The 11-emotion taxonomy is sufficient, non-overlapping, and each sentence has a single primary emotion.
- domain assumption Translating Marathi to English preserves the emotional content of the original sentence.
- domain assumption MahaSent-GT Twitter data is a representative source of Marathi emotional text.
Cite this review
Pith. "Pith review of L3Cube-MahaEmotions: A Marathi Emotion Recognition Dataset with Synthetic Annotations using CoTR prompting and Large Language Models." pith.science (2026). https://pith.science/paper/4J4AMAAK
@misc{pith2026250600863,
author = {Pith},
title = {Pith review of: L3Cube-MahaEmotions: A Marathi Emotion Recognition Dataset with Synthetic Annotations using CoTR prompting and Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/4J4AMAAK}},
note = {Machine review of arXiv:2506.00863}
}
read the original abstract
Emotion recognition in low-resource languages like Marathi remains challenging due to limited annotated data. We present L3Cube-MahaEmotions, a high-quality Marathi emotion recognition dataset with 11 fine-grained emotion labels. The training data is synthetically annotated using large language models (LLMs), while the validation and test sets are manually labeled to serve as a reliable gold-standard benchmark. Building on the MahaSent dataset, we apply the Chain-of-Translation (CoTR) prompting technique, where Marathi sentences are translated into English and emotion labeled via a single prompt. GPT-4 and Llama3-405B were evaluated, with GPT-4 selected for training data annotation due to superior label quality. We evaluate model performance using standard metrics and explore label aggregation strategies (e.g., Union, Intersection). While GPT-4 predictions outperform fine-tuned BERT models, BERT-based models trained on synthetic labels fail to surpass GPT-4. This highlights both the importance of high-quality human-labeled data and the inherent complexity of emotion recognition. An important finding of this work is that generic LLMs like GPT-4 and Llama3-405B generalize better than fine-tuned BERT for complex low-resource emotion recognition tasks. The dataset and model are shared publicly at https://github.com/l3cube-pune/MarathiNLP
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Evan Heetderks Alexandre Magueresse, Vincent Carles. 2020. https://arxiv.org/abs/2006.07264 Low-resource languages: A review of past work and future challenges
arXiv 2020
-
[4]
Akshita Aggarwal Anshul Wadhawan. 2021. https://arxiv.org/abs/2102.09943 Towards emotion recognition in hindi-english code-mixed data: A transformer based approach . In Computation and Language
work page Pith review arXiv 2021
-
[5]
Dr.R.R.Deshmukh Bharati Borade. 2023. http://www.joaasr.com/index.php/joaasr/article/view/936 Emotional speech recognition for marathi language
work page 2023
-
[6]
Tejas Deshpande, Nidhi Kowtal, and Raviraj Joshi. 2024. Chain-of-translation prompting (cotr): A novel prompting technique for low resource languages. arXiv preprint arXiv:2409.04512
arXiv 2024
-
[7]
Kishor Bhangale; Dipali Dhake; Rupali Kawade; Triveni Dhamale; Vaishnavi Patil; Nehul Gupta. 2023. Deep learning-based analysis of affective computing for marathi corpus. In 2023 3rd International Conference on Intelligent Technologies (CONIT)
work page 2023
-
[8]
Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, and Raviraj Joshi. 2024. On limitations of llm as annotator for low resource languages. arXiv preprint arXiv:2411.17637
arXiv 2024
Show all 27 references
-
[9]
Charibeth Cheng Jan Christian Blaise Cruz. 2020. https://arxiv.org/abs/2005.02068 Establishing baselines for text classification in low-resource languages
2020 arXiv
-
[10]
Raviraj Joshi. 2022 a . L3cube-mahacorpus and mahabert: Marathi monolingual corpus, marathi bert language models, and resources. In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference, pages 97--101
2022
-
[11]
Raviraj Joshi. 2022 b . L3cube-mahanlp: Marathi natural language processing datasets, models, and library. arXiv preprint arXiv:2205.14728
2022 arXiv
-
[12]
Jimmy Lin Kelechi Ogueji, Yuxin Zhu. 2021. https://aclanthology.org/2021.mrl-1.11/?utm_source=miragenews&utm_medium=miragenews&utm_campaign=news Small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages . ACL Anthology
2021
-
[13]
Alvarez, Adria Recasens, and Agata Lapedriza
Ronak Kosti, Jose M. Alvarez, Adria Recasens, and Agata Lapedriza. 2017. https://openaccess.thecvf.com/content_cvpr_2017/html/Kosti_Emotion_Recognition_in_CVPR_2017_paper.html Emotion recognition in context . In Proceedings of the IEEE Conference on Computer Vision and Pattern...
2017
-
[14]
Kołakowska, A
A. Kołakowska, A. Landowska, M. Szwoch, W. Szwoch, and M. R. Wróbel. 2014. https://link.springer.com/chapter/10.1007/978-3-319-08491-6_5 Emotion recognition and its applications . In Proceedings of the 13th International Joint Conference on Natural Language Processing and the ...
2014 doi
-
[15]
Atharva Kulkarni, Meet Mandhane, Manali Likhitkar, Gayatri Kshirsagar, and Raviraj Joshi. 2021. L3cubemahasent: A marathi tweet-based sentiment analysis dataset. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Ana...
2021
-
[16]
Tapesh Kumar and Mehul Mahrishi andf Girish Sharma. 2023. https://link.springer.com/article/10.1007/s11042-023-15150-1 Emotion recognition in hindi text using multilingual bert transformer
2023 doi
-
[17]
Le, and Thang Luong
Minh-Thang Luong, Quoc V. Le, and Thang Luong. 2023. https://transacl.org/ojs/index.php/tacl/article/view/2112 Multilingual neural machine translation with a special focus on low-resource languages . Transactions of the Association for Computational Linguistics (TACL)
2023
-
[18]
Sanjib Narzary, Maharaj Brahma, and Mwnthai Narzary. 2022. Generating monolingual dataset for low resource language bodo from old books using google keep. In Proceedings of ACL
2022
-
[19]
Krish Patel, Gaurav Keshari, Dhaval Powle, Saad Ansari, Tejaswini Chavan, and Anindita Khade. 2024. Hybrid nlp model for multilingual sentiment and emotion analysis in poetry. In 2024 International Conference on Artificial Intelligence and Quantum Computation-Based Sensor Appl...
2024
-
[20]
Patil and Satish R
Pravin K. Patil and Satish R. Kolhe. 2024. https://link.springer.com/chapter/10.1007/978-981-99-7962-2_15 Sarcasm detection for marathi and the role of emoticons
2024 doi
-
[21]
Aabha Pingle, Aditya Vyawahare, Isha Joshi, Rahul Tangsali, and Raviraj Joshi. 2023. L3cube-mahasent-md: A multi-domain marathi sentiment analysis dataset and transformer models. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages...
2023
-
[22]
Mona Diab Shabnam Tafreshi, Shubham Vatsal. 2024. https://arxiv.org/abs/2402.18424 Emotion classification in low and moderate resource languages . arxiv
2024 arXiv
-
[23]
Gopendra Vikram Singh, Priyanshu Priya, Mauajama Firdaus, Asif Ekbal, and Pushpak Bhattacharyya. 2022. https://arxiv.org/abs/2205.13908 Emoinhindi: A multi-label emotion and intensity annotated dataset in hindi for emotion recognition in dialogues . LREC 2022
2022 arXiv
-
[24]
Donald Jefferson Thabah and Bipul Syam Purkayastha
N. Donald Jefferson Thabah and Bipul Syam Purkayastha. 2021. https://link.springer.com/chapter/10.1007/978-981-33-4084-8_1 Low resource neural machine translation from english to khasi: A transformer-based approach . In Low Resource Neural Machine Translation from English to K...
2021 doi
-
[25]
Abhishek Velankar, Hrushikesh Patil, and Raviraj Joshi. 2022. L3cube-mahahate: A tweet-based marathi hate speech detection dataset and bert models. Aggression and Cyberbullying (TRAC 2022), page 1
2022
-
[26]
Bradley Hauer Xiang Zhang, Senyu Li. 2023. https://aclanthology.org/2023.emnlp-main.491/ Don’t trust chatgpt when your question is not in english: A study of multilingual abilities and types of llms . In Proceedings of the 2023 Conference on Empirical Methods in Natural Langua...
2023
-
[27]
Yuqing Yang, Jie Fu, and Pascal Poupart. 2023. https://aclanthology.org/2023.acl-main.123 Prompt learning for low-resource language understanding with pretrained models . In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)
2023
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.