{"id":"0630a079-042f-477d-9134-5aec0e187b14","arxiv_id":"2505.14272","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Cross-lingual retrieval-augmented fine-tuning beats target-only training on average in eight hate speech detection languages, with peak performance near 2,000 retrieved instances.","lead":"This paper tests whether adding similar labeled examples retrieved from a large multilingual hate speech pool to a small target-language training set improves hate speech detection. Across eight languages it finds consistent average gains over training on target data alone, with the best results typically around 2,000 retrieved instances.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No per-seed variance or significance testing undercuts 'consistently outperforms'; the paper's own Table 5 contradicts the claim that 20 retrieved instances always beat Mono.","rationale":"I focus on the missing variance and significance testing rather than the reader's designated weakest assumption (embedding-space relevance) because the headline claim is explicitly about consistent outperformance. The absence of per-seed statistics directly threatens that factual claim, and the paper's own Table 5 already contradicts the strong per-configuration wording in Section 5. A random-retrieval baseline would be important for mechanistic attribution, but it would not falsify the headline claim if the augmentation effect is real and reliable; the variance issue is more load-bearing for the claim as written. I credit the paper for otherwise clean evaluation hygiene: fixed held-out test sets, target-language instances excluded from the retrieval pool, and a priori hyperparameters. Those choices reduce circularity, but they do not compensate for the lack of uncertainty quantification over five seeds. The reader's CONDITIONAL verdict is appropriate: the method may well work, but the central claim is not yet supported at the claimed level of consistency. My recommendation is therefore UNCHANGED rather than a move to a different verdict.","tokens_in":24118,"tokens_out":5845,"duration_ms":59582,"concrete_test":"Request the authors' per-seed F1-macro scores for every cell in Table 5 (8 languages x 12 train sizes x 4 retrieval sizes x 5 seeds), or rerun with the promised code. Compute paired differences (retrieved minus Mono) with 95% bootstrap confidence intervals, and perform a sign test over languages and train sizes. Also count the number of (language, train size) cells where retrieved-20 is at least Mono. If most differences fall within the 5-seed noise, or if the sign reverses across seeds for a substantial fraction of cells, the 'consistently outperforms' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that retrieval-augmented fine-tuning 'consistently outperforms' target-only training. All reported F1-macro numbers are means over 5 seeds, but the paper never reports standard deviations, confidence intervals, or significance tests. In low-resource settings with 10-50 training examples, seed variance is typically large, so the consistency and size of the improvements are not established. This is not merely a reporting gap: Section 5 states that 'In all languages, retrieving as few as 20 instances for fine-tuning already outperforms the Mono setting,' but Table 5 contains direct counterexamples. For Ous19 fr with train size 20, Mono is 47.26 and retrieved-20 is 47.21. For San20 it with train size 100, Mono is 70.96 and retrieved-20 is 70.01. The abstract's 'consistently outperforms' is supported only at the level of language-level averages, and even those averages have no uncertainty attached. Without per-seed results, the comparison to Rottger et al. (2022) could also reflect noise rather than a reliable gain. The claim as stated therefore rests on an unverified statistical foundation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a cross-lingual retrieval-augmented fine-tuning method for hate speech detection in low-resource target languages. The approach embeds a large multilingual pool of labeled hate speech datasets, retrieves nearest neighbors of a small target-language training set in bge-m3 embedding space, and fine-tunes XLM-T on the union of retrieved and target-language instances. The method is evaluated on eight languages with training sizes from 10 to 2,000 and five seeds, reporting F1-macro against a target-only baseline (Mono) and against Röttger et al. (2022). The central claim is that retrieval-augmented fine-tuning consistently outperforms Mono and, in most cases, the state of the art, even with as few as 20 retrieved instances.","tokens_in":24190,"tokens_out":4943,"duration_ms":47781,"significance":"If the empirical claim holds, the method is practically useful: it is simple, scalable, avoids translation or intermediate-task selection, and shows large gains in very low-resource settings. The evaluation design has notable strengths: same-language instances are excluded from retrieval, related translated/English sets are excluded for German and Turkish, test sets are fixed and unseen, and ablations compare English-only versus multilingual retrieval and MMR. However, the statistical foundation is currently incomplete: no variance or significance information is reported, and the paper's own tables contain direct counterexamples to the 'consistently outperforms' statement.","major_comments":[{"comment":"The statement 'In all languages, retrieving as few as 20 instances for fine-tuning already outperforms the Mono setting' is directly contradicted by Table 5. For San20 it at training size 100, Mono is 70.96 and retrieved-20 is 70.01; for Ous19 fr at training size 20, Mono is 47.26 and retrieved-20 is 47.21; for Gahd24 de at training size 50, Mono is 57.85 and retrieved-20 is 54.53; for For19 pt at training size 50, Mono is 60.25 and retrieved-20 is 59.26. The abstract's 'consistently outperforms' is therefore too strong as stated. Please revise the claim to refer to average performance, or to specific retrieval/training sizes, or explicitly report the counterexamples.","section":"Section 5 (Table 5)"},{"comment":"All F1-macro scores are means over 5 seeds, but no standard deviations, confidence intervals, or significance tests are reported anywhere in the paper. In the 10-50 training-example regime, seed variance is typically large, so the reader cannot tell whether the reported improvements over Mono, or the differences against Röttger et al. (2022), are reliable or within noise. This is load-bearing for the central empirical claim. Please provide per-seed results, or at least standard deviations plus a paired significance test across the five seeds (e.g., Wilcoxon signed-rank) for the main comparisons.","section":"Section 4.3 (Evaluation Details) and Table 5"},{"comment":"The comparison between English-only and multilingual retrieval is used to conclude that 'incorporating even a small amount of multilingual data can be beneficial,' but the observed differences are often tiny and no variance is reported. For example, Bas19 es AVG at 20 retrieved is identical (67.27) in both settings, and Ous19 fr AVG at 20 retrieved differs by only 0.56. Without per-seed statistics or a significance test, this secondary conclusion is not supported and should be either substantiated or softened.","section":"Section 5.2 (Table 2)"}],"minor_comments":[{"comment":"The abstract says 'retrieving as small as 200 instances' while the body repeatedly emphasizes 'as few as 20 instances'; please harmonize the wording.","section":"Abstract and Section 1"},{"comment":"The language column contains the typo 'Purtegues' for Portuguese; please fix.","section":"Table 4"},{"comment":"The symbol R is used both for the desired number of retrieved instances and for the retrieved set; please use distinct notation, for example R for the count and \\mathcal{R} for the set.","section":"Section 3.2"},{"comment":"Table 7 is hard to parse because the language row labels are not cleanly aligned with the sub-tables; please reformat so that each language block is clearly separated and the with/without MMR columns are visually distinct.","section":"Appendix G (Table 7)"},{"comment":"The statement that 'All models and datasets used in this study are licensed for academic research purposes' is imprecise, since some licenses (e.g., CC BY-NC-SA) restrict commercial use; consider rewording to avoid overstating licensing compatibility.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable with a major revision. The experimental design is good in its leakage controls and scope, but the authors must add per-seed statistics or significance tests and reconcile the Section 5/Table 5 contradiction before the central claim can be accepted. The 'consistently outperforms' language should be softened or qualified even after adding statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a legitimate extension of retrieval-augmented fine-tuning (Ivison et al., Lin et al.) to multilingual hate speech, with a held-out evaluation that is mostly clean. The new thing is not the mechanism but the setup: a 265k multilingual pool, eight target languages, training sizes from 10 to 2,000, and the practical finding that retrieval peaks around 2,000 instances. The average rows in Table 5 support the central claim in all eight languages, and same-language instances (plus related translated sets) are excluded from retrieval, so there is no obvious leakage. Hyperparameters are set a priori. That is real credit.\n\nWhere it gets soft: (1) They report only means over 5 seeds, with no standard deviations, confidence intervals, or significance tests. In the 10-50 training-example regime, seed variance is typically large, so \"consistently outperforms\" is not actually established at the individual setting level. The abstract's claim is fair only as a language-average statement. (2) Section 5 says \"in all languages, retrieving as few as 20 instances ... already outperforms the Mono setting.\" Table 5 contradicts this directly: Ous19 fr at train size 20 is 47.26 Mono vs 47.21 retrieved-20, and San20 it at train size 100 is 70.96 vs 70.01. Those are small differences, but the sentence is false as written. (3) There is no random-retrieval baseline. The consistent gains could come from adding cross-lingual data rather than from nearest-neighbor selection. That does not sink the practical contribution, but it weakens the \"relevance matters\" interpretation.\n\nThe comparison to Rottger et al. is if anything conservative, since the baseline gets the best of three intermediate English tasks. The MMR part is honestly reported as helping only some languages, and the limitations section is candid about pool imbalance and licensing.\n\nWho this is for: people working on low-resource hate speech detection and cross-lingual transfer. It deserves a serious referee. I would accept with required revisions: report per-seed variance, fix the 20-instance overclaim, and add a random-retrieval ablation if feasible. Code release should be enforced as part of the review.","headline":"A clean, useful empirical study of cross-lingual retrieval-augmented fine-tuning for hate speech; the main claim is credible at the average level, but the paper overstates one result and skips variance reporting, so \"consistently outperforms\" needs qualification.","tokens_in":24876,"tokens_out":1858,"would_cite":true,"duration_ms":17517,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Cross-lingual nearest-neighbor retrieval from a large multilingual pool makes hate speech detection data-efficient: with as few as 20 labeled target examples, fine-tuning on the retrieved neighbors outperforms target-language-only…","keywords":["hate speech detection","cross-lingual transfer","nearest neighbor retrieval","low-resource NLP","data-efficient fine-tuning","multilingual embeddings","F1-macro evaluation","retrieval-augmented training"],"falsifier":"If random selection of the same number of cross-lingual examples, with the same hate/non-hate label balance, matches the nearest-neighbor selection's F1-macro, then similarity-based retrieval is not the source of the gains and the central claim would be falsified.","tokens_in":23785,"feed_emoji":"🛡️","tokens_out":13036,"duration_ms":99448,"temperature":0.7,"pith_summary":"The paper claims that a multilingual nearest-neighbor retrieval step can make hate speech detection work well with only a handful of labeled examples in the target language. The authors build a pool of about 265,000 labeled instances from fourteen hate speech datasets, then for each low-resource target language retrieve the closest pool instances to the small labeled training set using Euclidean distance in a multilingual embedding space. Fine-tuning a multilingual classifier on the target set plus the retrieved neighbors consistently beats fine-tuning on the target set alone across eight languages, and in most cases matches or beats the strongest existing data-efficient baseline. The most striking empirical finding is that as few as 20 retrieved instances already help, around 2,000 is near-optimal, and the largest gains appear when the target language has fewer than 50 labels.","feed_headline":"20 retrieved examples boost hate speech detection in 8 languages","feed_subtitle":"Cross-lingual retrieval from a 265k-example pool beats target-only training, even with fewer than 50 labels.","key_machinery":"The engine is cross-lingual nearest-neighbor retrieval over a shared embedding space. Every instance in a pool of about 265,000 labeled hate speech examples from fourteen datasets is embedded with the bge-m3 multilingual encoder; for each target-language training instance, the method retrieves the top $k$ pool examples by Euclidean distance using a hierarchical navigable small world (HNSW) index. The retrieved texts are deduplicated and combined with the target training set to fine-tune XLM-T with binary cross-entropy loss, and because source and target share the same label space (hate vs. non-hate), joint fine-tuning is well-defined. This machinery carries the argument because it turns the choice of which source languages or tasks to transfer from into an automatic instance-level similarity search, and the experiments vary the number of retrieved instances from 10 to 100,000 to show where the signal saturates.","core_discovery":"The central claim is that fine-tuning a multilingual hate speech classifier on the union of a small target-language training set and the nearest-neighbor instances retrieved from a large multilingual pool gives higher F1-macro than fine-tuning on the target-language data alone. The authors report this consistently across eight languages, with as few as 20 retrieved instances already improving performance and with gains above 10 percentage points in some extreme low-resource settings with fewer than 50 labels. In most languages the retrieved-neighbor model also matches or beats the strongest existing data-efficient baseline, which needs 20,000 instances for intermediate English fine-tuning; the proposed method achieves that with roughly 200 to 2,000 retrieved examples. The authors also find that about 2,000 retrieved instances is near-optimal and that adding more can degrade performance, and that an MMR-based diversity filter helps in some languages when the retrieval budget is small.","pith_inferences":["Because the paper does not inspect whether retrieved neighbors are topically relevant or correctly labeled, a direct test would be to measure label agreement and topic overlap of the retrieved set; this would show whether the gains come from semantic relevance or simply from extra training data.","The same retrieve-and-fine-tune recipe should transfer to other cross-lingual classification tasks that share a label space, such as sentiment or toxicity detection, though the paper evaluates only hate speech.","The pool is about two-thirds English, so rebalancing it toward target languages is an untested way to change both the size of the gains and the optimal retrieval count.","The rise-then-fall pattern in retrieval count (peak near 2,000) hints at overfitting to a narrow manifold of similar examples; the paper reports the trend but does not explain the mechanism."],"forward_implications":["With as few as 20 labeled target examples, appending retrieved cross-lingual neighbors improves F1-macro over target-only fine-tuning in every language tested.","Retrieving around 2,000 instances is the best operating point on average; adding more retrieved data tends to degrade performance.","The method removes the need to manually choose source languages or intermediate tasks, since the retrieval itself selects relevant instances, including culturally or linguistically related ones.","The pool is extensible without retraining, so new datasets and languages can be added and immediately used for future retrievals.","MMR-based deduplication improves some languages when the retrieval budget is small, by favoring diversity alongside relevance."],"supporting_citations":[{"why":"Provides the bge-m3 multilingual embedding model that defines the similarity space for retrieval.","marker":"Chen et al. (2024)"},{"why":"Provides the HNSW algorithm used for the approximate nearest-neighbor search that builds the retrieved set.","marker":"Malkov and Yashunin (2020)"},{"why":"Provides the FAISS library used to index the embedding pool and run the similarity search.","marker":"Douze et al. (2024)"},{"why":"The strongest prior data-efficient baseline (intermediate English fine-tuning) the method is compared against and claims to surpass in most languages.","marker":"Röttger et al. (2022)"},{"why":"Provides XLM-T, the multilingual Twitter-pretrained classifier fine-tuned on the combined target-plus-retrieved data.","marker":"Barbieri et al. (2022)"},{"why":"Provides Maximum Marginal Relevance, used in the diversity-filtering experiments that improve some languages.","marker":"Carbonell and Goldstein (1998)"}],"fun_headline_variants":["20 neighbors enough to beat full training in hate speech detection","Cross-lingual retrieval needs only 200 examples to beat target-only","Retrieval from 265k pool boosts detection with fewer than 50 labels","20-200 retrieved examples match or beat 20,000-instance baselines","Nearest-neighbor retrieval: better hate speech detection with 1% of labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a small labeled set of target-language examples exists and that Euclidean distance in the multilingual embedding space is a trustworthy guide to which retrieved cross-lingual examples will actually help, a relevance signal the paper does not directly validate.","fun_headline_variants_meta":{"raw":{"variants":["20 neighbors enough to beat full training in hate speech detection","Cross-lingual retrieval needs only 200 examples to beat target-only","Retrieval from 265k pool boosts detection with fewer than 50 labels","20-200 retrieved examples match or beat 20,000-instance baselines","Nearest-neighbor retrieval: better hate speech detection with 1% of labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000406,"raw_usage":{"total_tokens":2112,"prompt_tokens":947,"completion_tokens":1165,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":1066}},"tokens_in":563,"tokens_out":1165,"duration_ms":9559,"temperature":1.0,"reasoning_tokens":1066,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:37:10.277458+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If random selection of the same number of cross-lingual examples, with the same hate/non-hate label balance, matches the nearest-neighbor selection's F1-macro, then similarity-based retrieval is not the source of the gains and the central claim would be falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides XLM-T, the multilingual Twitter-pretrained classifier fine-tuned on the combined target-plus-retrieved data."}],"review_version":1}