{"id":"f1086508-5d0c-4a9c-8cc4-b376386fc21f","arxiv_id":"2505.16460","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Frozen BGE embeddings with per-emotion prompts and a CatBoost classifier outperform fully fine-tuned mBERT/XLM-R and surpass prior decoder baselines on 28-language emotion detection.","lead":"This paper reports the University of Indonesia's system for SemEval 2025 Task 11, multi-label emotion detection across 28 languages, comparing frozen prompt-based embedding encoders with fully fine-tuned transformers. The authors find that BGE embeddings combined with a CatBoost classifier, plus an ensemble, outperform fine-tuned mBERT/XLM-R and even a strong decoder baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline comparison is not controlled: BGE/mE5 are larger, newer encoders than the mBERT/XLM-R fine-tuning baselines, and the fine-tuning uses one untuned seed, so the 6-point gap may reflect encoder capacity rather than the classifier-only versus fine-tuning regime.","rationale":"The reader's weakest assumption focuses on untuned fine-tuning baselines, which is a real threat to the comparison. My stress-test points to a broader confound: the design varies both method and encoder family/size simultaneously, so even a well-tuned fine-tuning baseline would not fully resolve the question. The paper's own XLM-R results show fine-tuning beating frozen embeddings for the same encoder, which supports the concern that the reported advantage is specific to the BGE/mE5 encoders rather than to classifier-only training as a general strategy. I do not see this as a fatal flaw: the paper is a shared-task system description, and its practical recommendation to use BGE embeddings plus CatBoost is plausible and supported by dev/test numbers. The issue is inferential overreach in the abstract and conclusion. A controlled same-encoder comparison would settle whether the method-level claim is robust. Since the reader already issued a conditional verdict with moderate confidence, my analysis does not require moving the verdict; it strengthens the stated condition.","tokens_in":21336,"tokens_out":9305,"duration_ms":76295,"concrete_test":"Run a controlled factorial comparison on the development set: (i) fine-tune mE5-large, and BGE if compute permits (or LoRA), with the paper's hyperparameters plus a small grid of learning rates {1e-5, 2e-5, 3e-5} and at least two seeds; (ii) train classifier-only models on frozen mBERT and XLM-R embeddings with the same CatBoost setup used for BGE. If frozen BGE/mE5 plus CatBoost still beats fine-tuned mE5/BGE by a meaningful margin, the central claim survives. If fine-tuning the same encoder closes or reverses the gap, the paper's method-level conclusion is confounded by encoder choice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The claim that classifier-only training on frozen prompt-based encoders 'significantly' outperforms full fine-tuning rests on comparisons that vary two factors at once: method and encoder. Section 3 and Table 4 use BGE (bge-multilingual-gemma2) and mE5 as frozen encoders, while the fine-tuned models in Table 6 are mBERT and XLM-R only. BGE and mE5 are pretrained at a different scale and recency than the mBERT/XLM-R base-class models; no fine-tuned mE5/BGE condition and no frozen mBERT condition is reported. The paper's own controlled contrast for XLM-R goes the other way: frozen XLMR-CB-ALL scores 38.48 (Table 12) while fine-tuned XLMR-BR-LANG scores 45.61 (Table 6), so fine-tuning helps when the encoder is held constant. Moreover, Section 4 fixes lr=3e-5, 4 epochs, seed 42 with no hyperparameter search; Table 6 contains collapsed runs (XLMR-MO-LANG-AL 27.85, XLMR-SEP-LANG 21.25), and no error bars or seed averaging are given. Thus the observed BGE advantage (best dev 55.40 vs mBERT 47.10) could be attributable to pretraining quality and model scale rather than to the classifier-only regime. The abstract's 'significantly better' conclusion is therefore not established as a general claim about methods.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports the University of Indonesia's system for SemEval 2025 Task 11 Track A, multilingual multi-label emotion detection across 28 languages. The authors compare two regimes: classifier-only training on frozen pretrained encoders (mE5, BGE, Jina, XLM-R) with classical classifiers, and end-to-end fine-tuning of mBERT and XLM-R with Focal or Asymmetric loss. On the development set, BGE with emotion-specific prompts and CatBoost achieves the best average F1-macro (55.40), and the submitted ensemble achieves 56.58 average F1-macro on the official test set, which the paper reports as significantly surpassing the Qwen2.5-72B decoder baseline of the BRIGHTER paper. The central conclusion is that classifier-only training with prompt-based encoders significantly outperforms full fine-tuning.","tokens_in":21600,"tokens_out":4588,"duration_ms":37937,"significance":"If the central claim is accepted, the paper would provide a practically useful result for low-resource multilingual emotion detection: frozen modern embedding encoders plus tree-based classifiers can beat fine-tuned multilingual transformers at lower training cost, and emotion-specific prompts add measurable gains. The paper's strengths are its systematic coverage of many encoder/classifier/prompt/loss combinations, the inclusion of per-language development tables, and the use of nonparametric statistical tests. However, the main methodological comparison is confounded and the fine-tuning baselines are not tuned, so the paper's headline claim is stronger than the evidence supports. The result is still a credible system description, but the general conclusion needs either additional controlled experiments or a substantially more careful formulation.","major_comments":[{"comment":"The central claim that classifier-only training on prompt-based encoders 'significantly' outperforms full fine-tuning is not supported by a controlled comparison. The classifier-only group uses BGE, mE5, Jina, and XLM-R, while the fine-tuning group uses only mBERT and XLM-R, so the comparison varies both method and encoder. The only encoder that appears in both regimes is XLM-R, and there fine-tuning helps: XLMR-CB-ALL scores 38.48 (Table 12), while fine-tuned XLMR-BR-LANG scores 45.61 (Table 6). The Mann-Whitney U test in §5.1 therefore compares model families rather than training regimes. Please either add a frozen mBERT condition and/or a fine-tuned BGE/mE5 condition, or restrict the conclusion to the models actually compared, e.g., 'in our configuration, BGE embeddings with CatBoost outperformed untuned mBERT and XLM-R fine-tuning.'","section":"§5.1, §7, Abstract"},{"comment":"The fine-tuning baselines are not tuned and may not be representative of full fine-tuning. Section 4 states that no hyperparameter tuning was performed, fixing learning rate 3e-5, four epochs, seed 42, and batch size 32. Several fine-tuning runs collapse on many languages (e.g., XLMR-MO-LANG-AL 27.85, XLMR-MO-LANG-FL 25.61, XLMR-SEP-LANG-FL 21.25, with several languages at 0.00 in Table 12). The best fine-tuned model (mBERT-MO-ALL-AL 47.10) trails BGE (55.40), but the gap may be partly due to undertrained or unlucky runs rather than to a fundamental advantage of classifier-only training. Please add seed averaging or a modest hyperparameter search, or explicitly label these results as untuned baselines and soften the corresponding conclusion.","section":"§4, Table 3, Table 6"},{"comment":"The test-set claim that the ensemble 'significantly surpasses' the best decoder-based model from prior work should report the exact comparison conditions. The Qwen2.5-72B scores are taken from the BRIGHTER paper, but Table 1 has missing entries for seven languages; the Wilcoxon test W=285, p<0.001 should state the number of paired languages used and whether the missing languages were excluded. It would also help to state whether the external baseline was produced with the same evaluation script and test split as the submitted predictions. Without these details, the comparison is not fully auditable.","section":"§5.2, Table 1"}],"minor_comments":[{"comment":"The caption says 'Performance scores of the classifier-only training model on the test set', but Section 5.1 and the surrounding text describe development-set results; please change the caption to 'development set'.","section":"Table 5"},{"comment":"Model V2 is described as 'an ensemble of four models: BGEV2-CB-ALL, BGE-CB-LANG, BGE-CB-LANG, BGE-CB-ALL'; the two middle entries appear to be duplicates and the model names do not match the naming scheme used elsewhere (e.g., BGEV1-CB-LANG vs. BGEV2-CB-LANG). Please correct this list.","section":"§5.2"},{"comment":"The prompt-ablation sentence says 'this experiment was conducted only on CB models with two samples' and reports increases 'from 5.3% to 5.5% and from 54.0% to 55.0%'. The intended comparison is presumably BGEV1-CB-ALL (53.52) to BGEV2-CB-ALL (55.40) and BGEV1-CB-LANG (54.19) to BGEV2-CB-LANG (55.39); the text should say so and should not refer to 'two samples'.","section":"§5.1"},{"comment":"The phrase 'where we freeze the classifier' in the introduction is misleading; the frozen component is the encoder, and the classifier is trained. Please rephrase.","section":"§1"},{"comment":"The sentence 'Different prompt lead to different results' has a subject-verb agreement error and should be rephrased.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"This is a solid SemEval system-description paper, but the abstract and conclusion generalize beyond the experimental design. The core fix is to either run one controlled comparison (e.g., fine-tune BGE/mE5 or freeze mBERT) or rewrite the claims as statements about the specific models and untuned settings. With that change, the paper would be acceptable. I would not require a full hyperparameter search, but the authors should acknowledge the confound and the collapsed runs in the text."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this for the recipe, not for the broad claim. The paper is a solid SemEval system report: it tests many encoder-classifier combinations on BRIGHTER, reports per-language dev scores, and finds that frozen bge-multilingual-gemma2 embeddings with per-emotion prompts plus a CatBoost head give the best F1-macro (55.40 dev, 56.58 test ensemble), beating their fine-tuned mBERT/XLMR baselines and the Qwen2.5-72B reference from the BRIGHTER paper. The per-emotion prompt ablation (BGE V2) is a nice, simple result, and the ensemble weighting is sensible.\n\nThe honest parts: prompts and hyperparameters are in the appendix, the tables are extensive, and the external Qwen comparison on the same test set is a reasonable datapoint. This is above-average transparency for a shared-task paper.\n\nThe soft spots are real and concentrated in the interpretation. The headline 'classifier-only significantly outperforms fully fine-tuned transformers' is not controlled: the frozen models are BGE/mE5/Jina, while the fine-tuned models are mBERT/XLMR. Different scale, different pretraining. The paper's own held-encoder comparison goes the other way: frozen XLMR-CB-ALL is 38.48, fine-tuned XLMR-BR-LANG is 45.61. So the claim should be 'these particular embeddings, used as feature extractors, work very well here,' not 'classifier-only beats fine-tuning.' Second, the fine-tuning baselines are untuned (one lr, 4 epochs, seed 42), and several runs collapse to near zero. A tuned fine-tuned model could close part of the 8-point dev gap, though the margin is large enough that the practical recipe likely survives. Third, there are small accuracy issues: the abstract's 56.58 does not obviously match Table 1's average, the prompt-comparison line mentions 'two samples,' and the Mann-Whitney test treats all configurations as independent, which is loose. No code, no error bars.\n\nMy take: the data and the math are fine, the citations are appropriate, and the practical result is credible. What needs fixing is the framing and a missing controlled ablation. For a shared-task report, that is a minor-to-moderate revision, not a rejection. Send it to review, ask for the broader claim to be narrowed or backed by a fine-tuned BGE / frozen mBERT condition, and for corrected numbers and code release.","headline":"A solid shared-task recipe paper whose practical result (BGE + per-emotion prompts + CatBoost) is credible, but whose 'classifier-only beats fine-tuning' claim is confounded with encoder choice and needs re-framing.","tokens_in":22250,"tokens_out":5662,"would_cite":true,"duration_ms":43709,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Frozen prompt-based embedding encoders with a tree classifier beat fully fine-tuned multilingual transformers on a 28-language emotion detection task.","keywords":["multi-label emotion classification","multilingual emotion detection","prompt-based embeddings","BGE","CatBoost","frozen encoders","SemEval-2025 Task 11","F1-macro"],"falsifier":"Re-run the mBERT and XLM-R fine-tuning baselines on the same BRIGHTER development splits with a small hyperparameter search, for example learning rates from 1e-5 to 5e-5, 3 to 6 epochs, and multiple seeds, and check whether any configuration reaches or exceeds the 55.40 development F1-macro of BGEV2-CB-ALL; if one does, the paper's central superiority claim over fine-tuning would not survive.","tokens_in":21096,"feed_emoji":"🎭","tokens_out":9091,"duration_ms":62475,"temperature":0.7,"pith_summary":"This paper is a system description for SemEval 2025 Task 11 Track A, multi-label emotion classification in 28 languages. It tries to establish that a surprisingly simple pipeline—frozen, prompt-based embedding encoders with a shallow tree classifier—outperforms the standard practice of fully fine-tuning multilingual transformer models. On the development set, the best BGE-with-CatBoost configuration reaches 55.40 F1-macro, compared with 47.10 for the best fully fine-tuned model, and the gap is statistically significant. On the official test set, the team's ensemble of four BGE models with weighted voting reaches 56.58 average F1-macro, which the paper reports as significantly surpassing the strongest decoder-based baseline from earlier work. If this holds, it means high-quality pretrained embeddings plus a cheap classifier can be a better use of resources than fine-tuning for multilingual emotion detection.","feed_headline":"Frozen embeddings beat fine-tuning on 28-language emotion task","feed_subtitle":"BGE with one prompt per emotion and a CatBoost classifier scored 56.58 average F1-macro on the official test set.","key_machinery":"The load-bearing mechanism is the combination of a frozen, prompt-conditioned embedding encoder and a shallow tree classifier. BGE (bge-multilingual-gemma2) and mE5 are prompt-based multilingual encoders: each input is prefixed with an instruction, and in the best configuration the model is prompted separately for each emotion, using a query like 'Represent this text for identifying the presence of the emotion {{EMOTION}}' to produce one embedding per emotion. CatBoost, a gradient-boosted tree model, is trained on these frozen embeddings with class weights to handle label imbalance. For the final submission, four BGE-plus-CatBoost models are combined by weighted voting, where each model's vote is signed and weighted by development-set F1-macro. The paper's argument is that these encoders already encode multilingual semantics well enough that fine-tuning the transformer is unnecessary, and that targeted prompts sharpen the representation for each emotion.","core_discovery":"On the paper's own terms, the central discovery is that classifier-only training on frozen embedding encoders, especially BGE with one prompt per emotion, beats end-to-end fine-tuning for multilingual multi-label emotion classification. The authors report a Mann-Whitney U test with U = 456 and p < 0.001 separating prompt-based encoder models from the rest, while the best single BGE configuration (BGEV2-CB-ALL) scores 55.40 on development and the best fine-tuned model (mBERT-MO-ALL-AL) scores 47.10. The submitted ensemble, combining BGE models trained on all languages or per language with CatBoost and weighted voting, achieves 56.58 average F1-macro on the official test set and outperforms the Qwen2.5-72B decoder baseline from the BRIGHTER paper with W = 285.0 and p < 0.001. The paper also finds that emotion-specific prompts improve over a single general prompt and that training on all languages does not significantly beat language-specific training.","pith_inferences":["A reader might infer from this result that, for low-resource multilingual emotion tasks, the marginal compute is better spent on stronger embedding models and prompt design than on fine-tuning hyperparameters.","The comparison would be more persuasive if the fine-tuning baselines received the same hyperparameter budget; a modest search over learning rate, epochs, and seeds could close part of the six-point development gap.","The same frozen-embedding-plus-tree-classifier recipe may transfer to other multilabel text tasks, such as hate-speech or stance detection, where label imbalance and low-resource languages are common.","The paper's own stated limitation is the lack of qualitative analysis, so an error-analysis pass on the weakest languages, such as VMW and YOR, could show whether the measured gap reflects true model quality or artifacts of the evaluation."],"forward_implications":["Fully fine-tuning multilingual transformers is not required to do well on multi-label emotion detection across many languages; frozen prompt-based embeddings plus CatBoost produce the best scores in this paper.","Emotion-specific prompting is a cheap and effective intervention: BGEV2 with one prompt per emotion outperforms BGEV1 with a single general prompt on development.","Multilingual joint training yields no significant benefit over language-specific training, so per-language models can be preferred when compute is limited.","Ensembling several embedding-classifier models with weighted voting improves test performance over the best single model in 25 of 28 languages.","Because the winning pipeline uses only frozen encoders and a tree classifier, strong results are achievable without large-scale fine-tuning compute."],"supporting_citations":[{"why":"Supplies the BRIGHTER dataset and the Qwen2.5-72B decoder baseline that the paper's test-set ensemble is compared against.","marker":"Muhammad et al., 2025a"},{"why":"Defines the SemEval-2025 Task 11 Track A setting, data splits, and evaluation that the experiments follow.","marker":"Muhammad et al., 2025b"},{"why":"Provides the BGE encoder, whose frozen prompt-conditioned embeddings power the best classifier-only models.","marker":"Chen et al., 2024"},{"why":"Provides the mE5 encoder, the second prompt-based embedding model in the winning family.","marker":"Wang et al., 2024"},{"why":"Provides the JinaV3 encoder used in the classifier-only comparisons across encoders.","marker":"Sturua et al., 2024"},{"why":"Provides CatBoost, the gradient-boosted classifier trained on frozen embeddings in the best-performing configurations.","marker":"Prokhorenkova et al., 2018"},{"why":"Provides mBERT, the multilingual BERT model used as one of the fully fine-tuned baselines.","marker":"Devlin et al., 2019"},{"why":"Provides XLM-R, the other fully fine-tuned multilingual transformer baseline.","marker":"Conneau et al., 2020"},{"why":"Supplies the MMTEB multilingual embedding benchmark results cited as evidence that BGE and mE5 have stronger multilingual representations.","marker":"Enevoldsen et al., 2025"},{"why":"Provides the non-parametric significance test the paper uses to claim prompt-based encoders outperform the other model families.","marker":"Mann and Whitney, 1947"}],"fun_headline_variants":["Frozen encoders beat fine-tuning in 28-language emotion AI","Classifier-only training wins on multilingual emotion task","BGE embeddings top fine-tuned models in SemEval emotion test","No fine-tuning needed: BGE outperforms XLMR for emotions","CatBoost on BGE surpasses fine-tuned XLMR in emotion task"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion that frozen prompt-based encoders are better than fine-tuning rests on the assumption that the fine-tuning baselines were given a fair comparison, since they were run with one fixed hyperparameter setting and no search.","fun_headline_variants_meta":{"raw":{"variants":["Frozen encoders beat fine-tuning in 28-language emotion AI","Classifier-only training wins on multilingual emotion task","BGE embeddings top fine-tuned models in SemEval emotion test","No fine-tuning needed: BGE outperforms XLMR for emotions","CatBoost on BGE surpasses fine-tuned XLMR in emotion task"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000219,"raw_usage":{"total_tokens":1422,"prompt_tokens":904,"completion_tokens":518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":520,"completion_tokens_details":{"reasoning_tokens":429}},"tokens_in":520,"tokens_out":518,"duration_ms":4091,"temperature":1.0,"reasoning_tokens":429,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:00:37.939892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the mBERT and XLM-R fine-tuning baselines on the same BRIGHTER development splits with a small hyperparameter search, for example learning rates from 1e-5 to 5e-5, 3 to 6 epochs, and multiple seeds, and check whether any configuration reaches or exceeds the 55.40 development F1-macro of BGEV2-CB-ALL; if one does, the paper's central superiority claim over fine-tuning would not survive.","supporting_citations":[],"review_version":1}