UWB at WASSA-2024 Shared Task 2: Cross-lingual Emotion Detection

Jakub \v{S}m\'id; Pavel Kr\'al; Pavel P\v{r}ib\'a\v{n}

REVIEW 1 major objections 1 minor 5 references

UWB at WASSA-2024 Shared Task 2: Cross-lingual Emotion Detection

T0 review · 1 major / 1 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pipeline that translates English training tweets into four languages and fine-tunes Orca 2 with QLoRA, XLM-R, and mT5 ranks first in numerical trigger-word detection, third in binary trigger-word detection, and seventh in emotion…

desk verdict A solid, clearly written shared-task system description whose rankings are believable, but whose general conclusion that translated data and trigger-word switching help is not supported by its own table. read the letter →

arxiv 2508.08650 v1 pith:YBCVUOSH submitted 2025-08-12 cs.CL

Jakub Šmíd , Pavel Přibáň , Pavel Král This is my paper · ORCID

classification cs.CL

keywords cross-lingualemotiondetectiontriggerwordQLoRAOrca2XLM-RmT5machinetranslationlabelprojection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper describes a system built for the WASSA-2024 cross-lingual emotion detection task, in which tweets in five languages must receive one of six emotion labels and the words triggering that emotion must be marked. Its central claim is that machine-translating the English training tweets into the other four languages, combined with a marker-based label projection and trigger-word switching, transfers emotion supervision across languages without explicit word alignment. On the task's test sets the system ranks first in numerical trigger-word detection, third in binary trigger-word detection, and seventh in emotion classification. If this holds, an English-centric large language model can be made multilingual for emotion analysis with cheap translation, and token-level trigger labels can be projected using special symbols rather than alignment tools.

What carries the argument

The central mechanism is the alignment-free label projection used in the pipeline. Trigger words in an English sentence are wrapped in special symbols such as brackets before machine translation; after translation the symbols mark the trigger words in the target language, and sentences where the translation drops the symbols are discarded. A second operation, trigger-word switching, builds two extra bilingual training sentences by replacing the trigger words in each language pair, creating code-switched data without any alignment tool. QLoRA fine-tuning of Orca 2 uses the low-rank update $W_0 + BA$ on a frozen, 4-bit quantized backbone, so a 13-billion-parameter model can be trained on a single 48 GB GPU. The machinery matters because it converts token-level annotation into a translation-survival problem: the pseudo-labels are only as good as the translation's ability to carry the marker symbols.

What would settle it

Rerun the trigger-word experiments with a translation engine that frequently drops or shifts the bracket markers (for example, an MT system with a low measured marker-preservation rate) and compare the token-level F1 and numerical attribution scores with the reported XLM-R score of 59.19 and mT5 score of 70.52; a large drop would show the result rides on marker-preserving translation rather than on the label-projection idea itself.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a pipeline combining a QLoRA-fine-tuned Orca 2 for sentence-level emotion labelling with XLM-R and mT5 for token-level trigger words, all trained on machine-translated versions of the English data, is enough to rank at or near the top of a five-language shared task. The strongest reported evidence is numerical trigger-word attribution, where mT5 trained on original English, translated, and trigger-word-switched datasets reaches 70.52 and takes first place; XLM-R reaches 59.19 token-level F1 for binary trigger words and ranks third; Orca 2 reaches 59.10 macro F1 for emotion detection and ranks seventh. Translated training data lifts Orca 2 by more than three F1 points on test, while the multilingual mT5 gains about eight points on the numerical task only when the translated and switched datasets are all used.

Load-bearing premise

The load-bearing premise is that machine translation keeps the special symbols that mark trigger words and preserves what each marked word is doing emotionally, because any sentence where the symbols are dropped is thrown away and the method needs enough surviving sentences to train on.

Editorial extensions

If this is right

Machine translation of the training set is a reliable way to improve cross-lingual emotion classification for an English-centric LLM, with the paper's reported test gain of over three F1 points for Orca 2.
A token-level annotation task can be transferred across languages by marking and translating, so word-alignment tools are not required to create pseudo-labels.
The reported mT5 numerical score improves by about eight points when translated and code-switched trigger-word data are added, suggesting that the full combination of original, translated, and switched data is needed for the top result.
Solving emotion detection and trigger-word detection independently with separate models is sufficient to be competitive, leaving joint modeling as an untested alternative.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If marker-preserving machine translation is the real source of the gain, the same recipe could transfer other token-level annotations, such as aspect terms or event triggers, to low-resource languages without training aligners.
The trigger-word-switched datasets can be read as a form of contrastive augmentation: exposing the model to the same tweet with trigger words in different languages may push it to rely on trigger content rather than language-specific surface form.
A testable extension is to run the same pipeline with a translation engine known to drop bracketed markers frequently; the expected result is that binary and numerical trigger scores degrade in proportion to marker-loss rate, which would locate the method's boundary.
The paper treats the two subtasks independently, so a natural next step is a joint model that predicts emotion and triggers together; the paper does not test this, and the current results do not show whether it would help.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth knowing: this is a system description, not a methods paper. The rankings are the contribution, and they are credible. The paper's broader claim—that translated data and trigger-word switching improve performance—does not survive contact with Table 3.

The new part is narrow: it is the first application of this specific stack (QLoRA-fine-tuned Orca 2 for emotion detection, XLM-R and mT5 for trigger words, plus Google-translated pseudo-labels and trigger-word switching) to the WASSA-2024 cross-lingual emotion task. That is a legitimate contribution. The system description is clear, the code is public, the evaluation is against an external shared-task test set, and the reported rankings (1st numerical, 3rd binary, 7th emotion) are consistent with the tables. For practical purposes, the paper gives a useful recipe.

The soft spot is real and central enough to matter. The abstract and conclusion say that additional translated data improves performance. Table 3 shows the opposite for most configurations: XLM-R binary drops from 58.59 to 56.69 with DS+DT and to 53.18 with DS+DSt+DTs; mT5 binary drops to 48.61; XLM-R numerical drops from 70.06 to 66.20 with DS+DT. The only large gain is mT5 numerical with the full combination (60.00 to 70.52), and it is unreplicated. The paper itself notes that seed variability can obscure differences, but every cell is a single run. That makes the causal claim unsupported. The official ranks are still facts about that submission, but they cannot carry the general conclusion.

A smaller risk is the label projection in Section 2.2: it assumes the translation keeps the special markers and that the marked word keeps its semantic role. The paper handles dropped markers by discarding the sentence, but it does not report how many sentences survive, so we cannot tell how much noise enters the pseudo-labels.

Who should read this: anyone working on cross-lingual emotion detection or practical QLoRA fine-tuning, especially as a shared-task baseline. The citations are appropriate and the writing is direct.

Recommendation: it deserves peer review as a workshop system description, but the authors should be pushed to either add multi-seed variance estimates or soften the causal claim in the abstract and conclusion. As it stands, the rankings are solid and the description is useful; the interpretation overreaches.

Referee Report

1 major / 1 minor

Summary. The paper describes the authors' submission to the WASSA-2024 Shared Task 2 on cross-lingual emotion detection. For emotion classification (six classes, five languages), the authors fine-tune a 4-bit quantized Orca 2 13B model with QLoRA, optionally adding machine-translated training data. For trigger word detection, they fine-tune XLM-R and mT5 encoders, using machine translation and a trigger-word-switching augmentation that mixes source and target-language trigger words. They report official shared-task results: 7th in emotion detection (F1 59.10), 3rd in binary trigger detection (XLM-R, token F1 59.19), and 1st in numerical trigger detection (mT5, 70.52). The central claimed contribution is that machine translation and trigger-word switching improve performance across both subtasks.

Significance. If the system's gains are real, the paper demonstrates a practical recipe for cross-lingual emotion and trigger-word detection that combines quantized LLM fine-tuning with multilingual encoders and cheap data augmentation. The official shared-task rankings are externally validated facts, and the code is publicly available, which aids reproducibility. The main scientific value is the evidence on which augmentation strategies help which model families. However, the paper's own results are mixed: most configurations with added translated data or trigger-word switching do not improve over the English-only baseline, and the one large gain (mT5 for numerical trigger detection) is a single run without variance estimates. The strength of the claimed causal conclusion is therefore substantially weaker than the text suggests.

major comments (1)

[Section 3.2, Table 3; Sections 3.3 and 4] Section 2.2 (label projection) assumes that the translation system preserves the special marker symbols and that a trigger word in the translated sentence carries the same semantic role. The authors state that sentences where markers are dropped are discarded, but they do not report how many sentences are discarded or how often the marker-based extraction yields a word that corresponds semantically to the original trigger. Without such statistics or a quality check, the quality of the pseudo-labels for the augmented datasets is unknown. The authors should report the survival rate and ideally a small manual or automatic validation of the projected trigger words.

minor comments (1)

[Section 2.1, Equation (2)] The notation h = {h}L i=1 should be h = {h_i}_{i=1}^L, and the explanation that numerical values are obtained by applying softmax to the logits for class 1 should be stated more explicitly, since the numerical score is not a probability over classes in the usual sense.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper reports empirical shared-task results against an external test set; the augmentation claims are experimental observations, not derivations from fitted inputs.

full rationale

This is an empirical system-description paper for the WASSA-2024 shared task. The authors fine-tune Orca 2, XLM-R, and mT5 on the official English training data plus machine-translated and trigger-switched variants, then evaluate on the competition's held-out test set. There is no first-principles derivation whose conclusion is equivalent to its input by construction: the label-projection procedure is a data-augmentation step that creates pseudo-labeled translated sentences, and the downstream evaluation is measured on an external benchmark, not on the pseudo-labeled data used for training. The claim that adding translated data improves performance is supported by the observed dev/test scores in Tables 1 and 3; that the evidence is mixed (many configurations degrade while one mT5 numerical setting improves substantially) is a weakness in the empirical argument, not a circularity. Similarly, selecting the best configuration on the dev set is standard model selection, not fitting a parameter and then calling a closely related quantity a prediction. The citations to QLoRA, LoRA, XLM-R, mT5, Orca 2, and the cross-lingual label-projection method are external prior work and are not load-bearing self-citations; no cited uniqueness theorem is invoked to force a choice. The paper makes no claim that would reduce to a renamed known result or to a self-definitional identity. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The central claim is an empirical performance result, so the ledger includes key hyperparameters and modeling assumptions chosen by the authors. There are no invented entities or new theoretical postulates.

free parameters (5)

LoRA rank r = 64
LoRA rank chosen without reported search; directly affects adapter capacity and the final emotion detection model.
LoRA alpha = 16
LoRA scaling parameter set alongside rank; part of QLoRA setup.
Orca 2 learning rate = 2e-4
Learning rate for Orca 2 QLoRA fine-tuning, chosen without a reported grid; affects convergence.
Batch size = 16
Uniform batch size for all models; a pragmatic choice that affects optimization.
Validation split fraction = 0.10
10% of the English training set held out for validation in all experiments; chosen by hand and affects model selection.

assumptions (3)

domain assumption The marker-based label projection preserves trigger-word semantics after machine translation.
Section 2.2 assumes the translation system retains the special symbols and that a marked word in the translated sentence corresponds to the same emotional trigger.
domain assumption The 10% English validation split is representative of the five target languages for model selection.
Section 2.5 states this split is used for all experiments and simulates true unsupervised settings. If the languages differ substantially, the selected hyperparameters may be suboptimal.
domain assumption Multilingual pre-trained models (Orca 2, XLM-R, mT5) provide transferable cross-lingual representations.
This is the premise for fine-tuning mostly English data and expecting it to transfer to Dutch, Russian, Spanish, and French.

how reviews work

0 comments

Cite this review

Pith. "Pith review of UWB at WASSA-2024 Shared Task 2: Cross-lingual Emotion Detection." pith.science (2026). https://pith.science/paper/YBCVUOSH

@misc{pith2026250808650,
  author       = {Pith},
  title        = {Pith review of: UWB at WASSA-2024 Shared Task 2: Cross-lingual Emotion Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YBCVUOSH}},
  note         = {Machine review of arXiv:2508.08650}
}

read the original abstract

This paper presents our system built for the WASSA-2024 Cross-lingual Emotion Detection Shared Task. The task consists of two subtasks: first, to assess an emotion label from six possible classes for a given tweet in one of five languages, and second, to predict words triggering the detected emotions in binary and numerical formats. Our proposed approach revolves around fine-tuning quantized large language models, specifically Orca~2, with low-rank adapters (LoRA) and multilingual Transformer-based models, such as XLM-R and mT5. We enhance performance through machine translation for both subtasks and trigger word switching for the second subtask. The system achieves excellent performance, ranking 1st in numerical trigger words detection, 3rd in binary trigger words detection, and 7th in emotion detection.

Figures

Figures reproduced from arXiv: 2508.08650 by the authors.

**Figure 2.** Example of label projection method with trigger words switching (lower part) for English and Spanish [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** shows the prompt used for fine-tuning the Orca 2 model, where we train the model to output the specific emotion class in a textual format. Predict one emotion label for the given text. The possible labels are: “Love”, “Joy”, “Anger”, “Fear”, “Sadness”, “Neutral”. Answer in one following format: “Label: <emotion_label>” [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Figures from the paper (2 more)

**Figure 5.** Figure 5: Confusion matrix on test data for the Orca 2 [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Confusion matrix on test data for the Orca 2 [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 5 linked inside Pith

[2013]

In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia

A simple, fast, and effective reparameteriza- tion of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia. Association for Computational Linguistics. Hao Fei, Meishan Zhang, and Donghong Ji. 2020. Cross-lingual semant...

arXiv 2013
[2015]

Preprint, arXiv:1409.2329

Recurrent neural network regularization. Preprint, arXiv:1409.2329. Wenxuan Zhang, Ruidan He, Haiyun Peng, Lidong Bing, and Wai Lam. 2021. Cross-lingual aspect- based sentiment analysis with aspect term code- switching. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 9220–9230, Online and Punta Cana, Do- ...

arXiv 2021
[2020]

In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4411–4421

XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4411–4421. PMLR. Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted senti- me...

arXiv 2019
[2023]

Annotation guidelines for labeling emotion in multilingual tweets. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, B...

arXiv 2023
[2024]

In Proceedings of the 14th Workshop of on Computa- tional Approaches to Subjectivity, Sentiment & Social Media Analysis@ACL 2024, Bangkok, Thailand

Findings of the wassa 2024 exalt shared task on explainability for cross-lingual emotion in tweets. In Proceedings of the 14th Workshop of on Computa- tional Approaches to Subjectivity, Sentiment & Social Media Analysis@ACL 2024, Bangkok, Thailand. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap translation for cross-lingual named entity recognit...

arXiv 2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [2013]

In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia

A simple, fast, and effective reparameteriza- tion of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia. Association for Computational Linguistics. Hao Fei, Meishan Zhang, and Donghong Ji. 2020. Cross-lingual semant...

arXiv 2013

[2] [2015]

Preprint, arXiv:1409.2329

Recurrent neural network regularization. Preprint, arXiv:1409.2329. Wenxuan Zhang, Ruidan He, Haiyun Peng, Lidong Bing, and Wai Lam. 2021. Cross-lingual aspect- based sentiment analysis with aspect term code- switching. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 9220–9230, Online and Punta Cana, Do- ...

arXiv 2021

[3] [2020]

In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4411–4421

XTREME: A massively multilingual multi- task benchmark for evaluating cross-lingual gener- alisation. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4411–4421. PMLR. Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li, and Yiwei Lv. 2019. Open-domain targeted senti- me...

arXiv 2019

[4] [2023]

Annotation guidelines for labeling emotion in multilingual tweets. Hugo Touvron, Louis Martin, Kevin Stone, Peter Al- bert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, B...

arXiv 2023

[5] [2024]

In Proceedings of the 14th Workshop of on Computa- tional Approaches to Subjectivity, Sentiment & Social Media Analysis@ACL 2024, Bangkok, Thailand

Findings of the wassa 2024 exalt shared task on explainability for cross-lingual emotion in tweets. In Proceedings of the 14th Workshop of on Computa- tional Approaches to Subjectivity, Sentiment & Social Media Analysis@ACL 2024, Bangkok, Thailand. Stephen Mayhew, Chen-Tse Tsai, and Dan Roth. 2017. Cheap translation for cross-lingual named entity recognit...

arXiv 2024