Pith. sign in

REVIEW 4 major objections 4 minor 29 references

Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa

T0 review · 4 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Pith's one-line read Fine-tuned DeBERTa-v3-large is the most reliable Japanese binary emotion classifier among the compared models, with mean F1 0.662.

desk verdict Useful incremental benchmark for Japanese emotion classification with a released package, but the headline result rests on a single split and the abstract misreports its own accuracy table. read the letter →

arxiv 2505.00013 v1 pith:DKLWIG5B submitted 2025-04-22 cs.CL cs.AI

classification cs.CLcs.AI
keywords JapaneseemotionclassificationDeBERTaRoWRIMEdatasetbinarypre-trainedlanguagemodelslargefine-tuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks which pre-trained language model best detects whether each of eight basic emotions is present in Japanese social-media posts. It turns the WRIME corpus's reader-averaged four-level intensity scores into binary labels and fine-tunes BERT, RoBERTa, and two DeBERTa-v3 variants, then compares them with prompted GPT-4o and TinySwallow-1.5B-Instruct. Its headline result is that DeBERTa-v3-large is the strongest classifier, with a mean F1 of 0.662 and a claimed mean accuracy of 0.860, and that every fine-tuned model beats both LLMs on F1 by a wide margin. If the comparison holds, Japanese emotion detection for social-media monitoring or customer feedback is best served today by fine-tuned DeBERTa-v3-large rather than by zero-shot prompting of current LLMs. The paper's own accuracy table, however, gives DeBERTa-v3-base a higher mean accuracy (0.870) than the large model (0.860), so the F1 result is the one that actually supports the ranking.

What carries the argument

The machinery is the binary decomposition of the task: each of the eight Plutchik emotions is predicted by its own fine-tuned transformer as present ($\geq 1$) or absent ($0$) from reader-averaged intensity labels. Converting the four-level scale to binary reduces the extreme skew (for Anger 97% and Trust 96% of posts are intensity 0) so that F1, not accuracy, becomes the informative metric. The paper fine-tunes BERT, RoBERTa, and DeBERTa-v3 base and large with the same schedule (30 epochs, learning rate $1\times10^{-6}$, F1-based early stopping) and evaluates accuracy and F1 on the 20% test split.

What would settle it

Recompute the mean accuracy from Table 2: DeBERTa-v3-base is 0.870 while DeBERTa-v3-large is 0.860, which already contradicts the abstract's 'best mean accuracy' claim. Then re-run the comparison with a writer-grouped split or several random seeds and check whether DeBERTa-v3-large still leads on F1; if the F1 gap over the base model closes, the central claim would not survive.

Watch

Extended reading notes

Core claim

The paper's central claim, stated in its own terms, is that fine-tuning a Japanese DeBERTa-v3-large model on binary labels derived from WRIME's reader-averaged intensity scores yields the best overall emotion classifier among the six systems it compares. That model reaches a mean F1 of 0.662, ahead of DeBERTa-v3-base (0.630), RoBERTa (0.613), BERT (0.572), ChatGPT-4o (0.527), and TinySwallow-1.5B-Instruct (0.292), and it posts the highest F1 on seven of the eight individual emotions; ChatGPT-4o wins only on Joy. On accuracy the abstract reports 0.860 for the large model, although Table 2 lists DeBERTa-v3-base at 0.870 and the large model at 0.860, so the F1 result is the internally consistent basis for the ranking.

Load-bearing premise

The load-bearing premise is that the single random 80/20 split of the 43,000 WRIME posts keeps every writer's posts entirely inside either training or test, with early stopping on a properly separated validation set; if posts from the same writer straddle the split, the reported scores are inflated and the model ranking might change.

Editorial extensions

If this is right

  • For Japanese binary emotion detection, fine-tuned DeBERTa-v3-large with mean F1 0.662 is the best among the compared models, and the author releases it as a pip-installable package for direct use.
  • Fine-tuned PLMs of the BERT family outperform prompted GPT-4o (F1 0.527) and TinySwallow-1.5B (0.292) by a large margin, so zero-shot prompting is not currently a substitute for fine-tuning.
  • Per-emotion F1 remains low for rare emotions (Anger 0.549, Trust 0.496), so practical systems should expect weak detection on those categories and need targeted data collection or augmentation.
  • Because the binary decomposition turns an eight-way multi-label task into eight independent classifiers, adding or removing an emotion does not require retraining the others.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A writer-grouped split or several random seeds would be a stronger test; the single 80/20 split with no mention of writer overlap means the reported margins could shrink, possibly changing the ranking.
  • Since accuracy is dominated by the majority 'absent' class, practitioners should judge models by F1; the 0.662-versus-0.630 gap between DeBERTa-v3-large and base is small enough that cost may favor the base model in production.
  • A single multi-label model sharing one encoder across the eight emotions could cut the eight-model download and might match per-emotion F1, but the paper does not test this.
  • The LLMs were evaluated with criteria but no demonstrations; adding a few-shot examples or requiring a short rationale before the label might close part of the F1 gap, a testable extension.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The manuscript reports a benchmark for binary emotion classification of Japanese text on the WRIME corpus, covering Plutchik's eight emotions. The authors convert reader-averaged intensity scores into binary labels, fine-tune BERT, RoBERTa, DeBERTa-v3-base, and DeBERTa-v3-large, and also prompt two LLMs (TinySwallow-1.5B-Instruct and ChatGPT-4o). Accuracy and F1 are reported per emotion, and the central claim is that DeBERTa-v3-large achieves the best mean F1 (0.662), with the fine-tuned PLMs outperforming the LLMs. The paper also releases a pip-installable prediction package.

Significance. If the empirical ranking is reliable, the paper provides a useful practical benchmark for Japanese emotion classification and a reusable artifact in the released DeBERTa-based package. The choice of F1 as the primary metric is appropriate given the strong class imbalance documented in Table 1, and the comparison against two LLMs is a useful addition. The public availability of the WRIME dataset and the released models supports reproducibility. However, the evaluation currently lacks the statistical and methodological support needed to establish the claimed ranking, and the abstract's accuracy claim is internally inconsistent with Table 2, so the contribution is not yet ready for publication.

major comments (4)
  1. [Abstract and Section 4, Table 2] The abstract states that "DeBERTa-v3-large attains the best mean accuracy (0.860) and F1-score (0.662), outperforming all other models." This accuracy claim is contradicted by Table 2, which reports DeBERTa-v3-base with mean accuracy 0.870 and DeBERTa-v3-large with 0.860. The body text in Section 4 correctly reports both models as outperforming the other PLMs but does not claim large is the accuracy leader. The abstract's wording must be corrected to match the table, or the table must be corrected if the reported averages are erroneous.
  2. [Section 3 and Section 4] The core ranking claim rests on a single random 80/20 split with no repeated seeds, no writer-level grouping, and no confidence intervals or significance tests. WRIME is a corpus of posts by individual writers; if posts from the same writer appear in both the training and test sets, writer-specific stylistic cues can inflate all reported scores and potentially change the model ordering. The gaps between some models are small (e.g., DeBERTa-v3-base mean F1 0.630 vs DeBERTa-v3-large 0.662; several per-emotion gaps are around 0.01-0.02), and with a single split there is no way to tell whether these differences are meaningful. The authors should use writer-disjoint splits, run multiple random seeds, and report variance or at least clearly state the limitation that the ranking is based on one split.
  3. [Appendix A and Section 3] The construction of the validation set used for early stopping and epoch selection is never specified. Section 3 describes only an 80% training / 20% test division, while Appendix A reports validation loss and F1 per emotion without stating whether the validation set is carved from the training portion, held out separately, or reused in any way. If the validation set overlaps with the test set, or if the test set is used to guide epoch selection, the reported test scores could be optimistically biased. This must be clarified because the reported F1 ranking is the paper's central result.
  4. [Appendix B and Section 4] The LLM comparison is described with a single prompt and no information about decoding temperature, number of repeated calls, or how malformed or non-array outputs were handled. Since the paper makes a comparative claim that all PLMs outperform the LLMs in F1, the LLM evaluation protocol needs to be specified in enough detail to allow replication and to rule out prompt-sensitivity artifacts. This is secondary to the PLM ranking, but it affects the completeness of the benchmark.
minor comments (4)
  1. [Section 2.1] The text says the WRIME subset contains 43,000 posts, but the four intensity rows in Table 1 each sum to 43,200 (e.g., Joy: 29883+7682+4512+1123). Please correct the corpus size consistently.
  2. [Section 3 and Appendix A] Section 3 states Epochs: 30, but Appendix A reports per-emotion early stopping at variable epochs, including RoBERTa Anger at epoch 32. The relationship between the stated epoch budget and the reported stopping epochs should be clarified, e.g., by explicitly saying early stopping with a maximum of 30 epochs and noting why one entry exceeds this.
  3. [Appendix A] The validation F1 values in Tables A1-A4 are surprisingly high relative to the test F1 values in Table 3 (e.g., BERT Joy validation F1 0.937 vs test F1 0.712; DeBERTa-v3-large Anger validation F1 0.967 vs test F1 0.549). This large gap is worth a sentence of discussion, as it may indicate overfitting, distribution shift, or a mismatch between the validation and test splits.
  4. [Table 3] The emotion column header appears to include a stray line break in “BER T”, and the model column header “DeBER T a” is visually unclear; please reformat the table headers.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the evaluation is an empirical benchmark with direct held-out measurements, and no prediction reduces to a fitted input.

full rationale

This paper is an empirical benchmark, not a derivation. The central claims (DeBERTa-v3-large has the best mean F1 of 0.662, and PLMs outperform LLMs) are direct measurements on a held-out 20% test split of WRIME, as stated in Section 3: "The entire WRIME corpus (43,000 posts) was divided into 80% training and 20% test sets." The labels are taken from reader-averaged intensity scores in Section 3 and are not defined in terms of any model output. Model selection by F1 on a validation set, as documented in Appendix A, is standard early stopping and does not constitute circularity because the reported test metrics are not the selection metric fitted to the same data. No claim is justified by a self-citation chain: references to WRIME, BERT, RoBERTa, and DeBERTa are external sources for the data and architectures, and no "uniqueness theorem" or prior result by the same author is invoked to force the outcome. The abstract's statement that DeBERTa-v3-large attains the best mean accuracy (0.860) is contradicted by Table 2, where DeBERTa-v3-base has 0.870, but this is an internal numerical inconsistency about reporting, not circularity. Likewise, the single-split, no-writer-grouping, and no-repeated-seed design raises methodological risk about benchmark reliability, but it does not make any derived quantity equivalent to its input by construction. Therefore no circular step is present.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a standard empirical protocol plus two specific assumptions: the thresholded reader labels define the task, and the single data split is leakage-free. No new theoretical entities are introduced; the only hand-chosen quantities are the label threshold, fixed hyperparameters, and per-emotion early-stopping epochs.

free parameters (3)
  • Binary label threshold = 1 (reader-averaged intensity >= 1)
    Emotion presence is defined by thresholding the mean of three reader intensity scores; this modeling choice changes the task and is not varied or justified beyond mitigating imbalance (Section 3).
  • Per-emotion training epochs = DeBERTa-large: Joy 8, Sadness 8, Anticipation 11, Surprise 10, Anger 10, Fear 11, Disgust 20, Trust 8; others in…
    Early stopping is selected on validation F1 per emotion and model; these are fitted to validation data, and the chosen values affect the reported test scores.
  • Fine-tuning hyperparameters = lr=1e-6, warmup=2500, batch=16, grad accum=2, weight decay=0.01
    Chosen by hand and fixed across models; they are part of the recipe but are not fitted per model, yet they influence all reported results.
assumptions (3)
  • domain assumption Reader-averaged intensity annotations in WRIME are a valid ground truth for emotion presence after thresholding at >= 1.
    Section 3 derives all labels from mean reader scores; the paper never validates this binarization against annotator agreement or alternative thresholds.
  • domain assumption The single random 80/20 split yields independent, representative test data without writer-level leakage.
    Section 3 describes only the split ratio; no writer grouping, seed, or validation/test separation is specified, and Table 1 counts sum to 43,200 rather than the stated 43,000.
  • standard math Standard transformer fine-tuning with AdamW and cross-entropy loss is appropriate for this binary classification task.
    This is conventional practice in the cited PLM literature; it is not the point of the paper but is relied on implicitly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa." pith.science (2026). https://pith.science/paper/DKLWIG5B

@misc{pith2026250500013,
  author       = {Pith},
  title        = {Pith review of: Performance Evaluation of Emotion Classification in Japanese Using RoBERTa and DeBERTa},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DKLWIG5B}},
  note         = {Machine review of arXiv:2505.00013}
}
read the original abstract

Background Practical applications such as social media monitoring and customer-feedback analysis require accurate emotion detection for Japanese text, yet resource scarcity and class imbalance hinder model performance. Objective This study aims to build a high-accuracy model for predicting the presence or absence of eight Plutchik emotions in Japanese sentences. Methods Using the WRIME corpus, we transform reader-averaged intensity scores into binary labels and fine-tune four pre-trained language models (BERT, RoBERTa, DeBERTa-v3-base, DeBERTa-v3-large). For context, we also assess two large language models (TinySwallow-1.5B-Instruct and ChatGPT-4o). Accuracy and F1-score serve as evaluation metrics. Results DeBERTa-v3-large attains the best mean accuracy (0.860) and F1-score (0.662), outperforming all other models. It maintains robust F1 across both high-frequency emotions (e.g., Joy, Anticipation) and low-frequency emotions (e.g., Anger, Trust). The LLMs lag, with ChatGPT-4o and TinySwallow-1.5B-Instruct scoring 0.527 and 0.292 in mean F1, respectively. Conclusion The fine-tuned DeBERTa-v3-large model currently offers the most reliable solution for binary emotion classification in Japanese. We release this model as a pip-installable package (pip install deberta-emotion-predictor). Future work should augment data for rare emotions, reduce model size, and explore prompt engineering to improve LLM performance. This manuscript is under review for possible publication in New Generation Computing.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 23 canonical work pages

  1. [1]

    : Opinion mining and sentiment analysis

    Pang, B., Lee, L., et al. : Opinion mining and sentiment analysis. Foundations and Trends® in information retrieval 2(1–2), 1–135 (2008)

  2. [2]

    Springer, ??? (2022)

    Liu, B.: Sentiment Analysis and Opinion Mining. Springer, ??? (2022)

  3. [3]

    arXiv p reprint arXiv:2311.11250 (2023)

    Kumar, S., Roy, P.P., Dogra, D.P., Kim, B.-G.: A comprehensive review on sentiment analysis: Tasks, approaches and applications. arXiv p reprint arXiv:2311.11250 (2023)

  4. [4]

    Journal of computational science 2(1), 1–8 (2011)

    Bollen, J., Mao, H., Zeng, X.: Twitter mood predicts the stock mark et. Journal of computational science 2(1), 1–8 (2011)

  5. [5]

    In: Proceedings of the Workshop on Computational Ling uistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pp

    Coppersmith, G., Dredze, M., Harman, C.: Quantifying mental hea lth signals in twitter. In: Proceedings of the Workshop on Computational Ling uistics and Clinical Psychology: From Linguistic Signal to Clinical Reality, pp. 51–6 0 (2014)

  6. [6]

    Proceedings of the ACM on Huma n- Computer Interaction 1(CSCW), 1–27 (2017) 1https://pypi.org/project/deberta-emotion-predictor 11

    Saha, K., De Choudhury, M.: Modeling stress with social media arou nd incidents of gun violence on college campuses. Proceedings of the ACM on Huma n- Computer Interaction 1(CSCW), 1–27 (2017) 1https://pypi.org/project/deberta-emotion-predictor 11

  7. [7]

    In: Proceedings of the 38th In ternational ACM SIGIR Conference on Research and Development in Information Re trieval, pp

    McAuley, J., Targett, C., Shi, Q., Van Den Hengel, A.: Image-based recommen- dations on styles and substitutes. In: Proceedings of the 38th In ternational ACM SIGIR Conference on Research and Development in Information Re trieval, pp. 43–52 (2015)

  8. [8]

    In: Proceedings of the 25th International Conference on World Wide Web, pp

    He, R., McAuley, J.: Ups and downs: Modeling the visual evolution of fash- ion trends with one-class collaborative filtering. In: Proceedings of the 25th International Conference on World Wide Web, pp. 507–517 (2016)

Show all 29 references
  1. [9]

    I n: Theories of Emotion, pp

    Plutchik, R.: A general psychoevolutionary theory of emotion. I n: Theories of Emotion, pp. 3–33. Elsevier, ??? (1980)

  2. [10]

    arXiv preprint arXiv:2 011.01612 (2020)

    ¨Ohman, E., P` amies, M., Kajava, K., Tiedemann, J.: Xed: A multilingual d ataset for sentiment analysis and emotion detection. arXiv preprint arXiv:2 011.01612 (2020)

  3. [11]

    arXiv preprint arXiv:1910.11769 (2019)

    Liu, C., Osama, M., De Andrade, A.: Dens: A dataset for multi-clas s emotion analysis. arXiv preprint arXiv:1910.11769 (2019)

  4. [12]

    In: Proc

    Bujnowski, P., Kuzma, B., Paziewski, B., Rutkowski, J., Marhula, J ., Bordzicka, Z., Andruszkiewicz, P.: Samsemo: New dataset for multilingual and mu ltimodal emotion recognition. In: Proc. Interspeech 2024, pp. 2925–292 9 (2024)

  5. [13]

    IEEE Access 12, 19752–19764 (2024)

    Xin, D., Jiang, J., Takamichi, S., Saito, Y., Aizawa, A., Saruwatari, H .: Jvnv: A corpus of japanese emotional speech with verbal content and n onverbal expressions. IEEE Access 12, 19752–19764 (2024)

  6. [14]

    Speech Communica tion 156, 103004 (2024)

    Xin, D., Takamichi, S., Saruwatari, H.: Jnv corpus: A corpus of ja panese nonverbal vocalizations with diverse phrases and emotions. Speech Communica tion 156, 103004 (2024)

  7. [15]

    Scientific Reports 13(1), 21785 (2023)

    Namba, S., Sato, W., Namba, S., Nomiya, H., Shimokawa, K., Osumi, M .: Devel- opment of the riken database for dynamic facial expressions with m ultiple angles. Scientific Reports 13(1), 21785 (2023)

  8. [16]

    Applied Mathematics and Nonlinear Sciences 10 (2025) https://doi.org/10.2478/amns-2025-0606

    Li, X.: Sentiment analysis and validity evaluation of japanese langu age under the transfer learning model. Applied Mathematics and Nonlinear Sciences 10 (2025) https://doi.org/10.2478/amns-2025-0606

  9. [17]

    arX iv preprint arXiv:2411.18383 (2024)

    Sun, Y., Tsuruta, H., Kumagai, M., Kurosaki, K.: Topic modeling and sentiment analysis on japanese online media’s coverage of nuclear energy. arX iv preprint arXiv:2411.18383 (2024)

  10. [18]

    In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human L anguage 12 Technologies, pp

    Kajiwara, T., Chu, C., Takemura, N., Nakashima, Y., Nagahara, H .: Wrime: A new dataset for emotional intensity estimation with subjective an d objec- tive annotations. In: Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Li...

  11. [19]

    Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-train ing of deep bidirectional transformers for language understanding. In: Pro ceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog...

  12. [20]

    In: P roceedings of the 30th Annual Meeting of the Association for Natural Langua ge Process- ing (NLP2024), Nagoya, Japan, pp

    Okadome, T., Shirai, K.: 感 情の 顕 現性を 考 慮した 書き 手の 感 情 強度 推定 [estimating writer emotion intensity with emotion manifestness]. In: P roceedings of the 30th Annual Meeting of the Association for Natural Langua ge Process- ing (NLP2024), Nagoya, Japan, pp. 1079–1083 (2024). Associat...

  13. [21]

    In: Foru m on Data Engineering and Information Management (DEIM2024), Paper T1- B-8-03, Japan (2024)

    Xue, P., Kobayashi, A.: 大 規 模 言 語 モ デ ルによる 細かな 感 情 推定 手 法 [fine-grained emotion estimation with large language models]. In: Foru m on Data Engineering and Information Management (DEIM2024), Paper T1- B-8-03, Japan (2024). In Japanese

  14. [22]

    In: 2020 17 th International Computer Conference on Wavelet Active Media Technology and Info rmation Processing (ICCW AMTIP), pp

    Adoma, A.F., Henry, N.-M., Chen, W.: Comparative analyses of ber t, roberta, dis- tilbert, and xlnet for text-based emotion recognition. In: 2020 17 th International Computer Conference on Wavelet Active Media Technology and Info rmation Processing (ICCW AMTIP), pp. 117–121 (2...

  15. [23]

    CoRR abs/1907.11692 (2019) 1907.11692

    Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis , M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019) 1907.11692

  16. [24]

    arXiv preprint arXiv:2006.03654 (2020)

    He, P., Liu, X., Gao, J., Chen, W.: Deberta: Decoding-enhanced b ert with disentangled attention. arXiv preprint arXiv:2006.03654 (2020)

  17. [25]

    arXiv preprint ar Xiv:2005.00547 (2020)

    Demszky, D., Movshovitz-Attias, D., Ko, J., Cowen, A., Nemade, G., Ravi, S.: Goemotions: A dataset of fine-grained emotions. arXiv preprint ar Xiv:2005.00547 (2020)

  18. [26]

    ArXiv abs/1908.11860 (2019)

    Rietzler, A., Stabinger, S., Opitz, P., Engl, S.: Adapt or get left be hind: Domain adaptation through bert language model finetuning for aspect-t arget sentiment classification. ArXiv abs/1908.11860 (2019)

  19. [27]

    https://arxiv.org/abs/2503.18253

    Belay, T.D., Gete, D.K., Ayele, A.A., Kolesnikova, O., Sidorov, G., Yima m, S.M.: Enhancing Multi-Label Emotion Analysis and Corresponding Inte nsities for Ethiopian Languages (2025). https://arxiv.org/abs/2503.18253

  20. [28]

    https://arxiv.org/abs/2501.16937 13

    Shing, M., Misaki, K., Bao, H., Yokoi, S., Akiba, T.: TAID: Temporally A daptive Interpolated Distillation for Efficient Knowledge Transfer in Languag e Models (2025). https://arxiv.org/abs/2501.16937 13

  21. [29]

    https://openai.com/index/gpt-4o

    OpenAI: GPT-4o Technical Report. https://openai.com/index/gpt-4o . Accessed: 2025-03-19 (2024) 14

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.