REVIEW 4 major objections 4 minor 29 references
Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa
T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Fine-tuned DeBERTa-v3-large is the most reliable Japanese binary emotion classifier among the compared models, with mean F1 0.662.
desk verdict Useful incremental benchmark for Japanese emotion classification with a released package, but the headline result rests on a single split and the abstract misreports its own accuracy table. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the binary decomposition of the task: each of the eight Plutchik emotions is predicted by its own fine-tuned transformer as present ($\geq 1$) or absent ($0$) from reader-averaged intensity labels. Converting the four-level scale to binary reduces the extreme skew (for Anger 97% and Trust 96% of posts are intensity 0) so that F1, not accuracy, becomes the informative metric. The paper fine-tunes BERT, RoBERTa, and DeBERTa-v3 base and large with the same schedule (30 epochs, learning rate $1\times10^{-6}$, F1-based early stopping) and evaluates accuracy and F1 on the 20% test split.
What would settle it
Recompute the mean accuracy from Table 2: DeBERTa-v3-base is 0.870 while DeBERTa-v3-large is 0.860, which already contradicts the abstract's 'best mean accuracy' claim. Then re-run the comparison with a writer-grouped split or several random seeds and check whether DeBERTa-v3-large still leads on F1; if the F1 gap over the base model closes, the central claim would not survive.
Extended reading notes
Core claim
The paper's central claim, stated in its own terms, is that fine-tuning a Japanese DeBERTa-v3-large model on binary labels derived from WRIME's reader-averaged intensity scores yields the best overall emotion classifier among the six systems it compares. That model reaches a mean F1 of 0.662, ahead of DeBERTa-v3-base (0.630), RoBERTa (0.613), BERT (0.572), ChatGPT-4o (0.527), and TinySwallow-1.5B-Instruct (0.292), and it posts the highest F1 on seven of the eight individual emotions; ChatGPT-4o wins only on Joy. On accuracy the abstract reports 0.860 for the large model, although Table 2 lists DeBERTa-v3-base at 0.870 and the large model at 0.860, so the F1 result is the internally consistent basis for the ranking.
Load-bearing premise
The load-bearing premise is that the single random 80/20 split of the 43,000 WRIME posts keeps every writer's posts entirely inside either training or test, with early stopping on a properly separated validation set; if posts from the same writer straddle the split, the reported scores are inflated and the model ranking might change.
Editorial extensions
If this is right
- For Japanese binary emotion detection, fine-tuned DeBERTa-v3-large with mean F1 0.662 is the best among the compared models, and the author releases it as a pip-installable package for direct use.
- Fine-tuned PLMs of the BERT family outperform prompted GPT-4o (F1 0.527) and TinySwallow-1.5B (0.292) by a large margin, so zero-shot prompting is not currently a substitute for fine-tuning.
- Per-emotion F1 remains low for rare emotions (Anger 0.549, Trust 0.496), so practical systems should expect weak detection on those categories and need targeted data collection or augmentation.
- Because the binary decomposition turns an eight-way multi-label task into eight independent classifiers, adding or removing an emotion does not require retraining the others.
Reading between the lines
- A writer-grouped split or several random seeds would be a stronger test; the single 80/20 split with no mention of writer overlap means the reported margins could shrink, possibly changing the ranking.
- Since accuracy is dominated by the majority 'absent' class, practitioners should judge models by F1; the 0.662-versus-0.630 gap between DeBERTa-v3-large and base is small enough that cost may favor the base model in production.
- A single multi-label model sharing one encoder across the eight emotions could cut the eight-model download and might match per-emotion F1, but the paper does not test this.
- The LLMs were evaluated with criteria but no demonstrations; adding a few-shot examples or requiring a short rationale before the label might close part of the F1 gap, a testable extension.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript reports a benchmark for binary emotion classification of Japanese text on the WRIME corpus, covering Plutchik's eight emotions. The authors convert reader-averaged intensity scores into binary labels, fine-tune BERT, RoBERTa, DeBERTa-v3-base, and DeBERTa-v3-large, and also prompt two LLMs (TinySwallow-1.5B-Instruct and ChatGPT-4o). Accuracy and F1 are reported per emotion, and the central claim is that DeBERTa-v3-large achieves the best mean F1 (0.662), with the fine-tuned PLMs outperforming the LLMs. The paper also releases a pip-installable prediction package.
Significance. If the empirical ranking is reliable, the paper provides a useful practical benchmark for Japanese emotion classification and a reusable artifact in the released DeBERTa-based package. The choice of F1 as the primary metric is appropriate given the strong class imbalance documented in Table 1, and the comparison against two LLMs is a useful addition. The public availability of the WRIME dataset and the released models supports reproducibility. However, the evaluation currently lacks the statistical and methodological support needed to establish the claimed ranking, and the abstract's accuracy claim is internally inconsistent with Table 2, so the contribution is not yet ready for publication.
major comments (4)
- [Abstract and Section 4, Table 2] The abstract states that "DeBERTa-v3-large attains the best mean accuracy (0.860) and F1-score (0.662), outperforming all other models." This accuracy claim is contradicted by Table 2, which reports DeBERTa-v3-base with mean accuracy 0.870 and DeBERTa-v3-large with 0.860. The body text in Section 4 correctly reports both models as outperforming the other PLMs but does not claim large is the accuracy leader. The abstract's wording must be corrected to match the table, or the table must be corrected if the reported averages are erroneous.
- [Section 3 and Section 4] The core ranking claim rests on a single random 80/20 split with no repeated seeds, no writer-level grouping, and no confidence intervals or significance tests. WRIME is a corpus of posts by individual writers; if posts from the same writer appear in both the training and test sets, writer-specific stylistic cues can inflate all reported scores and potentially change the model ordering. The gaps between some models are small (e.g., DeBERTa-v3-base mean F1 0.630 vs DeBERTa-v3-large 0.662; several per-emotion gaps are around 0.01-0.02), and with a single split there is no way to tell whether these differences are meaningful. The authors should use writer-disjoint splits, run multiple random seeds, and report variance or at least clearly state the limitation that the ranking is based on one split.
- [Appendix A and Section 3] The construction of the validation set used for early stopping and epoch selection is never specified. Section 3 describes only an 80% training / 20% test division, while Appendix A reports validation loss and F1 per emotion without stating whether the validation set is carved from the training portion, held out separately, or reused in any way. If the validation set overlaps with the test set, or if the test set is used to guide epoch selection, the reported test scores could be optimistically biased. This must be clarified because the reported F1 ranking is the paper's central result.
- [Appendix B and Section 4] The LLM comparison is described with a single prompt and no information about decoding temperature, number of repeated calls, or how malformed or non-array outputs were handled. Since the paper makes a comparative claim that all PLMs outperform the LLMs in F1, the LLM evaluation protocol needs to be specified in enough detail to allow replication and to rule out prompt-sensitivity artifacts. This is secondary to the PLM ranking, but it affects the completeness of the benchmark.
minor comments (4)
- [Section 2.1] The text says the WRIME subset contains 43,000 posts, but the four intensity rows in Table 1 each sum to 43,200 (e.g., Joy: 29883+7682+4512+1123). Please correct the corpus size consistently.
- [Section 3 and Appendix A] Section 3 states Epochs: 30, but Appendix A reports per-emotion early stopping at variable epochs, including RoBERTa Anger at epoch 32. The relationship between the stated epoch budget and the reported stopping epochs should be clarified, e.g., by explicitly saying early stopping with a maximum of 30 epochs and noting why one entry exceeds this.
- [Appendix A] The validation F1 values in Tables A1-A4 are surprisingly high relative to the test F1 values in Table 3 (e.g., BERT Joy validation F1 0.937 vs test F1 0.712; DeBERTa-v3-large Anger validation F1 0.967 vs test F1 0.549). This large gap is worth a sentence of discussion, as it may indicate overfitting, distribution shift, or a mismatch between the validation and test splits.
- [Table 3] The emotion column header appears to include a stray line break in “BER T”, and the model column header “DeBER T a” is visually unclear; please reformat the table headers.
Circularity Check
No circularity: the evaluation is an empirical benchmark with direct held-out measurements, and no prediction reduces to a fitted input.
full rationale
This paper is an empirical benchmark, not a derivation. The central claims (DeBERTa-v3-large has the best mean F1 of 0.662, and PLMs outperform LLMs) are direct measurements on a held-out 20% test split of WRIME, as stated in Section 3: "The entire WRIME corpus (43,000 posts) was divided into 80% training and 20% test sets." The labels are taken from reader-averaged intensity scores in Section 3 and are not defined in terms of any model output. Model selection by F1 on a validation set, as documented in Appendix A, is standard early stopping and does not constitute circularity because the reported test metrics are not the selection metric fitted to the same data. No claim is justified by a self-citation chain: references to WRIME, BERT, RoBERTa, and DeBERTa are external sources for the data and architectures, and no "uniqueness theorem" or prior result by the same author is invoked to force the outcome. The abstract's statement that DeBERTa-v3-large attains the best mean accuracy (0.860) is contradicted by Table 2, where DeBERTa-v3-base has 0.870, but this is an internal numerical inconsistency about reporting, not circularity. Likewise, the single-split, no-writer-grouping, and no-repeated-seed design raises methodological risk about benchmark reliability, but it does not make any derived quantity equivalent to its input by construction. Therefore no circular step is present.
Assumptions & free parameters
free parameters (3)
- Binary label threshold =
1 (reader-averaged intensity >= 1)
- Per-emotion training epochs =
DeBERTa-large: Joy 8, Sadness 8, Anticipation 11, Surprise 10, Anger 10, Fear 11, Disgust 20, Trust 8; others in…
- Fine-tuning hyperparameters =
lr=1e-6, warmup=2500, batch=16, grad accum=2, weight decay=0.01
assumptions (3)
- domain assumption Reader-averaged intensity annotations in WRIME are a valid ground truth for emotion presence after thresholding at >= 1.
- domain assumption The single random 80/20 split yields independent, representative test data without writer-level leakage.
- standard math Standard transformer fine-tuning with AdamW and cross-entropy loss is appropriate for this binary classification task.
Cite this review
Pith. "Pith review of Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa." pith.science (2026). https://pith.science/paper/DKLWIG5B
@misc{pith2026250500013,
author = {Pith},
title = {Pith review of: Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa},
year = {2026},
howpublished = {\url{https://pith.science/paper/DKLWIG5B}},
note = {Machine review of arXiv:2505.00013}
}
read the original abstract
Background Practical applications such as social media monitoring and customer-feedback analysis require accurate emotion detection for Japanese text, yet resource scarcity and class imbalance hinder model performance. Objective This study aims to build a high-accuracy model for predicting the presence or absence of eight Plutchik emotions in Japanese sentences. Methods Using the WRIME corpus, we transform reader-averaged intensity scores into binary labels and fine-tune four pre-trained language models (BERT, RoBERTa, DeBERTa-v3-base, DeBERTa-v3-large). For context, we also assess two large language models (TinySwallow-1.5B-Instruct and ChatGPT-4o). Accuracy and F1-score serve as evaluation metrics. Results DeBERTa-v3-large attains the best mean accuracy (0.860) and F1-score (0.662), outperforming all other models. It maintains robust F1 across both high-frequency emotions (e.g., Joy, Anticipation) and low-frequency emotions (e.g., Anger, Trust). The LLMs lag, with ChatGPT-4o and TinySwallow-1.5B-Instruct scoring 0.527 and 0.292 in mean F1, respectively. Conclusion The fine-tuned DeBERTa-v3-large model currently offers the most reliable solution for binary emotion classification in Japanese. We release this model as a pip-installable package (pip install deberta-emotion-predictor). Future work should augment data for rare emotions, reduce model size, and explore prompt engineering to improve LLM performance. This manuscript is under review for possible publication in New Generation Computing.
Reference graph
Works this paper leans on
-
[1]
: Opinion mining and sentiment analysis
Pang, B., Lee, L., et al. : Opinion mining and sentiment analysis. Foundations and Trends® in information retrieval 2(1–2), 1–135 (2008)
work page 2008
-
[2]
Liu, B.: Sentiment Analysis and Opinion Mining. Springer, ??? (2022)
work page 2022
-
[3]
arXiv p reprint arXiv:2311.11250 (2023)
Kumar, S., Roy, P.P., Dogra, D.P., Kim, B.-G.: A comprehensive review on sentiment analysis: Tasks, approaches and applications. arXiv p reprint arXiv:2311.11250 (2023)
-
[4]
Journal of computational science 2(1), 1–8 (2011)
Bollen, J., Mao, H., Zeng, X.: Twitter mood predicts the stock mark et. Journal of computational science 2(1), 1–8 (2011)
work page 2011
-
[5]
Coppersmith, G., Dredze, M., Harman, C.: Quantifying mental hea lth signals in twitter. In: Proceedings of the Workshop on Computational Ling uistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pp. 51–6 0 (2014)
work page 2014
-
[6]
Saha, K., De Choudhury, M.: Modeling stress with social media arou nd incidents of gun violence on college campuses. Proceedings of the ACM on Huma n- Computer Interaction 1(CSCW), 1–27 (2017) 1https://pypi.org/project/deberta-emotion-predictor 11
work page 2017
-
[7]
McAuley, J., Targett, C., Shi, Q., Van Den Hengel, A.: Image-based recommen- dations on styles and substitutes. In: Proceedings of the 38th In ternational ACM SIGIR Conference on Research and Development in Information Re trieval, pp. 43–52 (2015)
work page 2015
-
[8]
In: Proceedings of the 25th International Conference on World Wide Web, pp
He, R., McAuley, J.: Ups and downs: Modeling the visual evolution of fash- ion trends with one-class collaborative filtering. In: Proceedings of the 25th International Conference on World Wide Web, pp. 507–517 (2016)
work page 2016
Show all 29 references
-
[9]
I n: Theories of Emotion, pp
Plutchik, R.: A general psychoevolutionary theory of emotion. I n: Theories of Emotion, pp. 3–33. Elsevier, ??? (1980)
1980
-
[10]
arXiv preprint arXiv:2 011.01612 (2020)
¨Ohman, E., P` amies, M., Kajava, K., Tiedemann, J.: Xed: A multilingual d ataset for sentiment analysis and emotion detection. arXiv preprint arXiv:2 011.01612 (2020)
2020
-
[11]
arXiv preprint arXiv:1910.11769 (2019)
Liu, C., Osama, M., De Andrade, A.: Dens: A dataset for multi-clas s emotion analysis. arXiv preprint arXiv:1910.11769 (2019)
2019 arXiv
-
[12]
In: Proc
Bujnowski, P., Kuzma, B., Paziewski, B., Rutkowski, J., Marhula, J ., Bordzicka, Z., Andruszkiewicz, P.: Samsemo: New dataset for multilingual and mu ltimodal emotion recognition. In: Proc. Interspeech 2024, pp. 2925–292 9 (2024)
2024
-
[13]
IEEE Access 12, 19752–19764 (2024)
Xin, D., Jiang, J., Takamichi, S., Saito, Y., Aizawa, A., Saruwatari, H .: Jvnv: A corpus of japanese emotional speech with verbal content and n onverbal expressions. IEEE Access 12, 19752–19764 (2024)
2024
-
[14]
Speech Communica tion 156, 103004 (2024)
Xin, D., Takamichi, S., Saruwatari, H.: Jnv corpus: A corpus of ja panese nonverbal vocalizations with diverse phrases and emotions. Speech Communica tion 156, 103004 (2024)
2024
-
[15]
Scientific Reports 13(1), 21785 (2023)
Namba, S., Sato, W., Namba, S., Nomiya, H., Shimokawa, K., Osumi, M .: Devel- opment of the riken database for dynamic facial expressions with m ultiple angles. Scientific Reports 13(1), 21785 (2023)
2023
-
[16]
Applied Mathematics and Nonlinear Sciences 10 (2025) https://doi.org/10.2478/amns-2025-0606
Li, X.: Sentiment analysis and validity evaluation of japanese langu age under the transfer learning model. Applied Mathematics and Nonlinear Sciences 10 (2025) https://doi.org/10.2478/amns-2025-0606
2025 doi
-
[17]
arX iv preprint arXiv:2411.18383 (2024)
Sun, Y., Tsuruta, H., Kumagai, M., Kurosaki, K.: Topic modeling and sentiment analysis on japanese online media’s coverage of nuclear energy. arX iv preprint arXiv:2411.18383 (2024)
2024 arXiv
-
[18]
In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human L anguage 12 Technologies, pp
Kajiwara, T., Chu, C., Takemura, N., Nakashima, Y., Nagahara, H .: Wrime: A new dataset for emotional intensity estimation with subjective an d objec- tive annotations. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Li...
2021
-
[19]
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-train ing of deep bidirectional transformers for language understanding. In: Pro ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...
2019
-
[20]
In: P roceedings of the 30th Annual Meeting of the Association for Natural Langua ge Process- ing (NLP2024), Nagoya, Japan, pp
Okadome, T., Shirai, K.: 感 情の 顕 現性を 考 慮した 書き 手の 感 情 強度 推定 [estimating writer emotion intensity with emotion manifestness]. In: P roceedings of the 30th Annual Meeting of the Association for Natural Langua ge Process- ing (NLP2024), Nagoya, Japan, pp. 1079–1083 (2024). Associat...
2024
-
[21]
In: Foru m on Data Engineering and Information Management (DEIM2024), Paper T1- B-8-03, Japan (2024)
Xue, P., Kobayashi, A.: 大 規 模 言 語 モ デ ルによる 細かな 感 情 推定 手 法 [fine-grained emotion estimation with large language models]. In: Foru m on Data Engineering and Information Management (DEIM2024), Paper T1- B-8-03, Japan (2024). In Japanese
2024
-
[22]
In: 2020 17 th International Computer Conference on Wavelet Active Media Technology and Info rmation Processing (ICCW AMTIP), pp
Adoma, A.F., Henry, N.-M., Chen, W.: Comparative analyses of ber t, roberta, dis- tilbert, and xlnet for text-based emotion recognition. In: 2020 17 th International Computer Conference on Wavelet Active Media Technology and Info rmation Processing (ICCW AMTIP), pp. 117–121 (2...
2020
-
[23]
CoRR abs/1907.11692 (2019) 1907.11692
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis , M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019) 1907.11692
2019 arXiv
-
[24]
arXiv preprint arXiv:2006.03654 (2020)
He, P., Liu, X., Gao, J., Chen, W.: Deberta: Decoding-enhanced b ert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020)
2020 arXiv
-
[25]
arXiv preprint ar Xiv:2005.00547 (2020)
Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., Ravi, S.: Goemotions: A dataset of fine-grained emotions. arXiv preprint ar Xiv:2005.00547 (2020)
2020 arXiv
-
[26]
ArXiv abs/1908.11860 (2019)
Rietzler, A., Stabinger, S., Opitz, P., Engl, S.: Adapt or get left be hind: Domain adaptation through bert language model finetuning for aspect-t arget sentiment classification. ArXiv abs/1908.11860 (2019)
2019 arXiv
-
[27]
https://arxiv.org/abs/2503.18253
Belay, T.D., Gete, D.K., Ayele, A.A., Kolesnikova, O., Sidorov, G., Yima m, S.M.: Enhancing Multi-Label Emotion Analysis and Corresponding Inte nsities for Ethiopian Languages (2025). https://arxiv.org/abs/2503.18253
2025
-
[28]
https://arxiv.org/abs/2501.16937 13
Shing, M., Misaki, K., Bao, H., Yokoi, S., Akiba, T.: TAID: Temporally A daptive Interpolated Distillation for Efficient Knowledge Transfer in Languag e Models (2025). https://arxiv.org/abs/2501.16937 13
2025 arXiv
-
[29]
https://openai.com/index/gpt-4o
OpenAI: GPT-4o Technical Report. https://openai.com/index/gpt-4o . Accessed: 2025-03-19 (2024) 14
2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.