{"id":"73adba7b-34d9-45ac-bcff-edac081dca61","arxiv_id":"2505.06149","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Prompted LLMs beat fine-tuned encoders on HateCheck functional tests but trail them on real-world test sets across eight non-English languages.","lead":"Researchers tested four multilingual instruction-tuned LLMs for hate speech detection across eight non-English languages using zero-shot and few-shot prompts, and compared them to fine-tuned encoder models. They found prompting lags behind fine-tuning on real-world data but generalizes better on controlled functional tests, and that the best prompt varies by language.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2's prompting scores are oracle-selected over many prompts on the test set; without a held-out prompt-selection split, the functional-test advantage over encoders is overstated, and the size of the overstatement is unquantified.","rationale":"The paper's central claim is a comparative split: fine-tuned encoders win on real-world distributions, prompted LLMs win on HateCheck functional tests. The weakest link in the evidence for that split is the construction of the prompting baseline. Table 2 is explicitly built from the best zero- and few-shot results, and Appendix D shows the best prompt varying per language and per evaluation condition. Since no validation-based selection step is reported, the prompting numbers are best-of-N estimates on the test set, which is a form of selection bias. This matters most for the functional-test side of the claim, where the spread across prompts is large. The concern is real and checkable, and the reader already identified it. I do not think it overturns the qualitative split, because the functional-test gaps are large and consistent across all eight languages, but it does mean the exact quantitative claim in the abstract is not yet supported by a fair evaluation protocol. The paper also lacks error bars and significance tests, which reinforces the need for a held-out selection check. The authors provide detailed appendices and a clear description of the prompt grid, which makes the proposed re-analysis straightforward; this is a strength, not a weakness. Overall, a conditional verdict is appropriate until the selection protocol is clarified or re-run.","tokens_in":14588,"tokens_out":11762,"duration_ms":123661,"concrete_test":"Re-run the evaluation with a held-out prompt-selection protocol. For each language, hold out a prompt-development split from the real-world training data (separate from the few-shot example pool), select the single best prompt per model/language on that development split, then evaluate only the selected prompts on the real-world test set and the HateCheck functional test set. Compare the resulting F1-macro scores with Table 2. If functional-test scores drop by more than a few points per language or fall below the encoder scores in several languages, the reported prompting advantage is materially inflated; if the selected prompts remain clearly above encoder scores, the oracle-selection concern does not overturn the qualitative claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central comparison in Table 2 ('best zero- and few-shot prompting results') is obtained by taking, for each language, the maximum F1-macro across four LLMs and a large grid of prompts (vanilla, classification, CoT, NLI, role-play, translate, definition, distinction, multilingual variants, and their 1/3/5-shot combinations). No validation-based prompt-selection procedure is described anywhere in the paper. Tables 6-7 in Appendix D show that the chosen prompt differs by language and by test condition (real-world vs. functional), and Section 6 presents these selected scores as the prompting result. Selecting the maximum on the same test set used for evaluation is an oracle selection, and it inflates the prompting numbers — most on functional tests, where prompt sensitivity is high (e.g., Spanish Llama3 vanilla scores 86.37 while CoT scores 33.59 in Table 6). The headline claim that prompting 'achieve[s] better generalization on functional tests' rests on these maxima. The large observed gaps (e.g., Hindi 65.93 vs. 24.91) suggest the qualitative split would probably survive a realistic selection penalty, but the paper does not quantify that penalty, does not report error bars or significance tests, and does not describe any held-out prompt-selection protocol, so the precise magnitude of the prompting advantage is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports a comparative study of zero-shot and few-shot prompting of four multilingual instruction-tuned LLMs (LLaMA-3.1-8B-Instruct, Qwen2.5-7B-Instruct, Aya-101, BloomZ-7B1) against fine-tuned encoder classifiers (mDeBERTa, XLM-T) for hate speech detection in eight non-English languages. Using many prompt templates and 1/3/5-shot variants, it evaluates on real-world tweet test splits and the multilingual HateCheck functional benchmark. The central claim is that fine-tuned encoders generally outperform prompted LLMs on real-world data, while prompting (especially few-shot) achieves higher F1-macro on functional tests, and that prompt design is language-dependent.","tokens_in":14849,"tokens_out":5484,"duration_ms":55094,"significance":"The paper has strong practical scope: it covers four LLMs, eight languages, a wide prompt grid, and both naturalistic and functional test sets, and it includes a data-curve analysis that is relevant for low-resource deployment. If the comparative claim survives closer scrutiny, it offers a useful trade-off rule for practitioners. The reproducibility-oriented release of code and prompts is a further strength. The main weakness is that the headline comparison is based on per-language best prompts selected on the test set, so the quantitative size of the prompting advantage on functional tests is not yet established.","major_comments":[{"comment":"The prompting scores in Table 2 are 'best zero- and few-shot prompting results', obtained by taking the maximum F1-macro over four LLMs and the full prompt grid on the same test sets used for evaluation. No held-out validation-based prompt selection is described. Appendix D shows that the selected prompt changes by language and by condition (e.g., Spanish functional Llama3: vanilla 86.37 vs. CoT 33.59), so the selection penalty can be large. This makes the exact magnitude of the claimed functional-test advantage over encoders uncertain. Please either use a fixed prompt per condition chosen on a validation split, report distributions over prompts, or report the sensitivity of the headline comparison to the selection procedure.","section":"Section 6, Table 2 and Appendix C"},{"comment":"The data-curve comparison, including the statements that prompting becomes competitive with 100–200 examples in Spanish, 300–400 in Hindi, and 600–700 in German, compares an oracle-selected prompting score against fine-tuned XLM-T without error bars or significance tests. The crossover points are therefore not established with any stated uncertainty. Please add confidence intervals or seed-level variability, and state explicitly whether the same best-prompt selection criterion is applied on the encoder side or on a separate validation split.","section":"Section 6, Figure 1 and Appendix C"},{"comment":"The wording 'stronger generalization ability' is stronger than what is measured: the LLMs are not trained, and the functional-test advantage is a performance comparison after test-set prompt selection, not a controlled measure of generalization. Please qualify the claim (for example, 'on held-out functional test suites, prompted LLMs achieve higher F1-macro') and avoid implying a general capability ordering beyond the evaluated conditions.","section":"Section 6 and Conclusion"}],"minor_comments":[{"comment":"The model header 'Qwan' is a typo and should read 'Qwen'.","section":"Table 1"},{"comment":"The prompt labels 'general' and 'distiction' appear in Table 1 but do not match any template in Tables 3–5; please add the missing templates or rename the labels for consistency.","section":"Table 1 and Appendix B"},{"comment":"Full per-prompt results are shown only for Spanish and Portuguese; since the headline comparison depends on the full prompt grid, please make complete tables available for all languages in the supplementary material.","section":"Appendix D"},{"comment":"The description of random test-set construction should state whether the split is stratified by class and whether a fixed random seed is used, for reproducibility.","section":"Section 3"},{"comment":"The few-shot naming convention is slightly confusing: a 'five-shot' condition includes ten examples (five per class). Please clarify this at first use.","section":"Section 5 and Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the empirical breadth is a real strength. The central barrier is methodological: the prompting results are oracle-selected on the test set, which likely inflates the functional-test advantage. I do not see circularity in the sense of a fitted model defining the result; the issue is evaluation protocol. If the authors add a validation-based prompt-selection procedure or otherwise quantify the selection effect, the qualitative direction of the claim is likely to survive, though the margins may shrink."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read. The paper's useful contribution is breadth: it puts four instruction-tuned LLMs against two fine-tuned encoders across eight non-English languages, using both real-world test sets and HateCheck-style functional tests, with a wide prompt taxonomy (vanilla, CoT, NLI, role-play, translate, definition, distinction, and few-shot combos). That is more systematic than prior multilingual prompting evaluations, which mostly used one or two prompt styles. The data-scaling curves in Appendix C are practical: they show roughly when fine-tuning starts to beat prompting per language. The appendices are genuinely detailed; prompt templates are given in full, and the Spanish/Portuguese per-prompt tables let you see the variance. The core finding, that prompted LLMs trail fine-tuned encoders on real-world distributions but lead on functional tests, is plausible and probably robust in direction. The functional margins are large in most languages (Hindi 65.9 vs 24.9; Arabic 71.9 vs 25.5). I would not bet against that split. The soft spot is exactly the one flagged in the stress-test note. Table 2 reports \"best zero-/few-shot prompting results,\" and those best numbers are selected per language and per test condition over four LLMs and a large grid of prompts, on the same test set used for evaluation. There is no validation split for prompt selection and no sensitivity analysis. That is oracle selection, and it inflates the prompting numbers—most on functional tests, where prompt sensitivity is enormous. Spanish Llama3 goes from 86.4 with vanilla to 33.6 with CoT. Some of the high functional scores will come down under a realistic selection rule. My guess is the qualitative split survives, because the margins are big, but the paper does not quantify the penalty, so the precise magnitude of the prompting advantage is not established. I also would have liked error bars or bootstrap confidence intervals; some real-world comparisons are close (French 55.6 vs 52.2). And the code is promised but not released. These are fixable. Overall: this is a solid, practical empirical paper aimed at people building multilingual moderation systems. It deserves a serious referee. My advice: send it out, but ask for a held-out prompt-selection procedure or a bound on selection bias, and for code/data release.","headline":"A broad, useful prompt-taxonomy sweep for multilingual hate speech detection whose headline split is probably real, but whose Table 2 numbers are picked on the test set and need a held-out selection protocol before being quoted.","tokens_in":15353,"tokens_out":2724,"would_cite":true,"duration_ms":28113,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompted multilingual language models are worse at recognizing hate speech in real-world social media posts than fine-tuned encoders, yet they generalize better on controlled functional tests of hate speech detection across eight…","keywords":["hate speech detection","multilingual NLP","zero-shot prompting","few-shot prompting","instruction-tuned LLMs","HateCheck","functional tests","low-resource languages"],"falsifier":"Re-run the same evaluation but select each language's prompt on a held-out validation set (or on a separate development portion of HateCheck) and then apply the chosen prompts to the real-world test sets and the held-out functional tests; if prompted LLMs no longer beat fine-tuned encoders on the functional tests, the reported generalization advantage is an artifact of test-set prompt selection.","tokens_in":14405,"feed_emoji":"🛡️","tokens_out":14531,"duration_ms":127681,"temperature":0.7,"pith_summary":"This paper asks whether instruction-tuned multilingual large language models can detect hate speech in eight non-English languages without any fine-tuning, using zero-shot and few-shot prompting. Its central finding is a split: on real-world test sets sampled from social media, fine-tuned encoder models generally score higher, but on HateCheck functional tests—controlled cases that probe implicit hate, negation, and non-hateful slurs—prompted LLMs, especially with few-shot examples, score higher in every language. The paper also shows that the best prompt differs by language and model, so prompt design, not just model choice, drives results. The upshot is a practical division of labor: prompting is competitive when labeled data is scarce, while fine-tuning encoders wins when enough training data exists.","feed_headline":"Prompted LLMs win stress tests, lose on real-world hate speech","feed_subtitle":"Eight languages: few-shot LLMs win HateCheck tests; fine-tuned encoders still win on real tweets.","key_machinery":"The argument is carried by two evaluation instruments paired with a wide prompt-design search. The real-world test sets are 2,000-sample subsets of eight existing hate speech datasets (1,000 for Arabic, 1,500 for French) drawn from actual social media conversation. The functional instrument is the HateCheck benchmark and its multilingual extension, which supply controlled test cases for capabilities such as detecting implicit hate, handling negation, and not flagging non-hateful uses of slurs. The other mechanism is the prompt zoo: nine or more zero-shot templates (vanilla, classification, definition, chain-of-thought, NLI, role-play, multilingual, translate, distinction) plus few-shot variants with 1, 3, or 5 examples per class, applied to four instruction-tuned LLMs. The study's key comparison is the best per-language prompting result against two fine-tuned encoders, XLM-T and mDeBERTa, on both test types.","core_discovery":"On the paper's own terms, the discovery is a performance split that depends on the evaluation distribution, not on which approach is universally better. On real-world datasets, the best prompted LLM result trails the best fine-tuned encoder in most languages—for example, Turkish real-world F1 is 81.76 for few-shot prompting versus 88.32 for XLM-T, and German 77.55 versus 79.18—though prompting already beats one or both encoders in languages like Arabic and French. On HateCheck functional tests, the ordering reverses across all eight languages: Hindi few-shot prompting reaches 65.93 versus 23.26 for XLM-T, Arabic 71.88 versus 25.47, Spanish 87.40 versus 67.93. The paper attributes the reversal to stronger generalization by instruction-tuned LLMs in controlled settings and shows that few-shot examples (typically five) with language-appropriate prompt templates raise functional-test scores further. The conclusion is not that prompting replaces fine-tuning, but that the choice should be guided by data availability and by whether the target is real-world distribution or functional robustness.","pith_inferences":["Because the best-prompt numbers are selected on the test set, the magnitude of the prompting advantage on HateCheck may be overstated; a held-out prompt-selection protocol could shrink the gap.","HateCheck cases are generated from templates, so part of the LLMs' functional-test edge may come from having seen similar template patterns during pretraining; an adversarially perturbed functional set could narrow the gap.","The paper's language-specific prompt rankings suggest a testable controller: use a small labeled validation set to pick among prompt templates per language, then measure real-world and functional F1; this could turn prompt search into a deployable decision rule.","The data-threshold points (roughly 100-200 examples for Spanish, 300-400 for Hindi, 600-700 for German) imply a practical annotation-budget rule: below these budgets, prompt; above them, fine-tune—though the paper presents these numbers as observations, not as a rule."],"forward_implications":["In low-resource settings, a team with only a few hundred labeled examples can use zero- or few-shot prompting to match or exceed a fine-tuned encoder, lowering the data threshold for deploying hate speech detection in a new language.","For robustness evaluations, prompted LLMs are a better starting point than fine-tuned encoders, since they generalize better on controlled functional tests such as HateCheck.","Few-shot prompting, usually five examples per class, should be part of functional-test deployments because it consistently raises HateCheck scores in most languages.","Because the winning prompt varies by language, model, and test type, practical systems should search over prompt templates rather than assume one prompt transfers.","With abundant training data, fine-tuning an encoder remains the more effective route for real-world distributions; prompting is not a substitute there."],"supporting_citations":[{"why":"Supplies the HateCheck functional test suite, the benchmark on which prompted LLMs outperform fine-tuned encoders.","marker":"Röttger et al. (2021)"},{"why":"Provides the multilingual HateCheck extension used as functional test sets across the eight languages.","marker":"Röttger et al. (2022b)"},{"why":"Defines the XLM-T encoder baseline, a multilingual Twitter-tuned model built on XLM-R.","marker":"Barbieri et al. (2022); Conneau et al. (2020)"},{"why":"Defines the mDeBERTa encoder baseline used as the second fine-tuned comparison model.","marker":"He et al. (2021)"},{"why":"Supplies the Arabic and French real-world tweet datasets used for evaluation.","marker":"Ousidhoum et al. (2019)"},{"why":"Supplies the Spanish SemEval-2019 real-world dataset used for evaluation.","marker":"Basile et al. (2019)"},{"why":"Supplies the Portuguese real-world hate speech dataset.","marker":"Fortuna et al. (2019)"},{"why":"Supplies the Hindi HASOC 2021 real-world dataset.","marker":"Modha et al. (2021)"},{"why":"Supplies the German GAHD adversarial dataset used as a real-world test set.","marker":"Goldzycher et al. (2024)"},{"why":"Supplies the Turkish cross-domain real-world hate speech dataset.","marker":"Toraman et al. (2022)"}],"fun_headline_variants":["Across 8 languages, prompts win stress tests but lose real data","Prompted LLMs: robust on HateCheck, weak on real-world hate","Zero-shot prompting beats encoders only on functional tests","For hate speech, few-shot prompts generalize on synthetic tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that choosing each language's best prompt from the test results themselves, as Table 2 does, gives a fair estimate of prompting performance rather than an optimistically selected one.","fun_headline_variants_meta":{"raw":{"variants":["Across 8 languages, prompts win stress tests but lose real data","Prompted LLMs: robust on HateCheck, weak on real-world hate","Zero-shot prompting beats encoders only on functional tests","For hate speech, few-shot prompts generalize on synthetic tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000739,"raw_usage":{"total_tokens":3284,"prompt_tokens":913,"completion_tokens":2371,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2299}},"tokens_in":529,"tokens_out":2371,"duration_ms":16087,"temperature":1.0,"reasoning_tokens":2299,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:46:46.967311+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same evaluation but select each language's prompt on a held-out validation set (or on a separate development portion of HateCheck) and then apply the chosen prompts to the real-world test sets and the held-out functional tests; if prompted LLMs no longer beat fine-tuned encoders on the functional tests, the reported generalization advantage is an artifact of test-set prompt selection.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the XLM-T encoder baseline, a multilingual Twitter-tuned model built on XLM-R."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Hindi HASOC 2021 real-world dataset."}],"review_version":1}