{"id":"ef3b9b5f-d843-4183-87ac-b8b603b8c720","arxiv_id":"2608.08090","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Around 50% of translated multilingual training data is enough to reach near-optimal figurative proverb identification, and culture-specific proverbs show the largest gains from multilingual supervision.","lead":"This paper tests how much translated multilingual data is needed to spot figurative proverbs across seven languages, and finds that about half of the translated data is enough to reach near-best performance. It also proposes a four-part description scheme for proverbs and reports that culture-specific proverbs improve the most when more languages are added.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Weakly supervised, source-propagated test labels make the supervision curves and Culture-Specific gains a measure of agreement with the labeler, not with human judgment.","rationale":"The reader's weakest assumption—that the silver labels generated by XLM-R Base and E5+LR define both training and evaluation—is the correct point of maximum leverage. I agree with the reader's conditional verdict. My stress-test adds one sharpening: the issue is not just label noise but the propagation of source labels across all translations. Because every translation of a proverb inherits the source's label, the test set does not provide language-specific ground truth. The task is then implicitly defined as 'does this translation carry the same figurative status as its source?' rather than 'is this translation figurative in its own language?' This makes the multilingual gains ambiguous: a model could improve simply by learning source- concept patterns or by mimicking the weak labeler's biases. The Culture-Specific result is where this concern lands hardest, since E5+LR's fine-grained Macro-F1 is only 0.36 and the Culture-Specific class is rare, making its subgroup F1 especially sensitive to labeler bias. I also note the instance-selection ablation confound (Table 4: 'All target' vs. 600-instance conditions), but that affects only the 'combining forms' claim, whereas the label issue undermines the 50% saturation claim, the Culture-Specific claim, and the overall multilingual-supervision conclusion. The proposed concrete test—human per-language annotation on a multilingual subset—directly settles whether the curves are real. Until that is done, CONDITIONAL remains the right verdict.","tokens_in":16139,"tokens_out":4403,"duration_ms":48492,"concrete_test":"Re-annotate a held-out sample of translated proverbs—for example, the 83 concepts available in all seven target languages (581 instances)—with per-language human labels for binary figurative status and the four fine-grained forms, using the same annotation guideline. Recompute the progressive-supervision curves (Figure 2) and subgroup F1 for each figurative form (Figure 4) on this human-labeled test set, or as a human-labeled test subset, for the 0%, 10%, 50%, and 100% conditions. If the 50% saturation pattern and the Culture-Specific gains do not reproduce on human labels, the central claims are artifacts of the weak labeler and its propagation; if they persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims—50% saturation, Culture-Specific showing the largest gains, and multilingual supervision improving figurative identification—are all evaluated on test labels generated by the same weak-supervision pipeline used to label the training data (Section 3, 'Weak Supervision Labeling across Languages'; Appendix A). The binary test labels come from XLM-R Base, and the fine-grained form labels come from E5+LR, which reaches only 0.3640 Macro-F1 (Table 11). The paper's own Ethical Considerations states these 'should not be regarded as human gold annotations.' More specifically, labels are propagated from the source proverb to every translation sharing the same source ID. This assumes all translations of a concept are figuratively equivalent, yet the paper's own qualitative analysis (Table 16) shows that translations can differ in cultural imagery, and the Discussion explicitly says categorical consistency does not establish instance-level preservation. Consequently, the test set contains no per-language human ground truth. The reported performance curves therefore measure how well models reproduce the weak labeler's decisions—not how well they identify figurative language in translated proverbs. The Culture-Specific effect is especially suspect: the E5+LR labeler is least reliable on this rare form, and the gain may reflect the model learning the labeler's bias as more silver-labeled multilingual data is added. The claim that 'combining all figurative forms is best' is additionally confounded by data volume (Table 4, 'All target' uses the full training set vs. 600 instances per form), but the label validity issue is more load-bearing because it threatens all three headline findings.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies the effect of progressively adding translated multilingual training data on binary figurative-language identification in proverbs. It introduces a four-way fine-grained annotation framework (Metaphorical, Moral/Advisory, Cause–Effect, Culture-Specific) and evaluates five model families across seven languages at 0%, 10%, 50%, and 100% multilingual supervision. The authors report that performance saturates around 50% of the translated data, that combining all figurative forms gives the strongest supervision signal, that the least frequent form (Culture-Specific) shows the largest gains, and that the proposed forms are distributionally stable across languages. A controlled instance-selection ablation isolates the contribution of each fine-grained form. The dataset and annotations are to be released upon publication.","tokens_in":16333,"tokens_out":4029,"duration_ms":39067,"significance":"If the reported trends held on human-validated labels, the 50% saturation result and the Culture-Specific gains would be practically valuable for multilingual data selection and annotation. The proposed multidimensional framework is a reasonable diagnostic lens that goes beyond metaphor-centric taxonomies, and the paper includes a commendable breadth of models, controlled supervision levels, and statistical significance testing. However, the significance is heavily conditional because the evaluation relies on weakly supervised silver labels generated by the same pipeline that labels the training data, and the fine-grained labeler has only 0.36 Macro-F1 on the human-validated seed. The claimed central findings are therefore not yet established.","major_comments":[{"comment":"The test set used for all reported experiments is labeled by the weak-supervision pipeline rather than by human annotation. Binary labels come from XLM-R Base, and fine-grained form labels come from E5+LR, which reaches only 0.3640 Macro-F1 on the human-validated seed (Table 11). The paper's own Ethical Considerations state that these labels 'should not be regarded as human gold annotations.' Consequently, Figures 2 and 4 and Tables 7-8 measure agreement with the weak labeler, not figurative-language identification accuracy. The central claims about 50% saturation and Culture-Specific gains are unsupported until the same analyses are shown on a human-labeled test set.","section":"Section 3, Weak Supervision Labeling across Languages"},{"comment":"The pipeline propagates each source proverb's label to all translations sharing the same source ID. This assumes instance-level figurative equivalence across languages, yet the paper's own Table 16 and Discussion acknowledge that translations of the same concept can use different cultural imagery and that categorical consistency does not establish instance-level preservation. Because the test set contains no per-language human ground truth, the per-language results in Figure 3 and Table 12 are confounded by propagated label noise, and the cross-lingual transfer conclusion is not separately validated.","section":"Appendix A, label propagation"},{"comment":"The fine-grained instance-selection ablation uses the same E5+LR labels both to select training subsets and to define the test subgroups. This creates a shared-bias confound: when the labeler is biased for a form, training on silver-labeled instances of that form can improve agreement with the same labeler on the test subgroup without improving genuine figurative identification. The Culture-Specific result is especially vulnerable because the fine-grained labeler has low Macro-F1 overall (0.3640) and this rare form is acknowledged as the least prevalent (Table 2). The authors should report the human-validated 83-concept subset separately to demonstrate that the same trends hold.","section":"Section 4.2 and Figure 4"},{"comment":"The statement that performance 'generally stabilizes after incorporating approximately 50% of the translated multilingual training data' is not formally tested. The McNemar analyses in Table 13 compare only 0% versus 100% supervision; no test compares 50% versus 100% or quantifies the gain from 50% to 100% across models. Without such a comparison, the saturation claim is an informal reading of Figure 2 rather than a statistically supported finding.","section":"Section 5, RQ1 saturation claim"}],"minor_comments":[{"comment":"The word 'languges' is a typo and should be 'languages.'","section":"Section 1, Contributions"},{"comment":"The phrase 'biary figritve detection' contains two typos; it should be 'binary figurative detection.'","section":"Section 4.2"},{"comment":"'Macnammer significant test' should be 'McNemar significance test.'","section":"Section 5"},{"comment":"The annotation guideline is internally inconsistent: the Output row says 'A single-select label is assigned' while the Decision rule says annotators 'choose tall representative labels' and labels may overlap. The multi-label nature should be stated consistently.","section":"Table 5"},{"comment":"'recrutment' is a typo for 'recruitment.'","section":"Ethical Considerations"},{"comment":"Several references use the placeholder style 'and 1 others' (e.g., Benamara et al., Chuang et al., Ghosh et al., Lai et al., Liu et al.); these should be replaced with the full author lists.","section":"References"},{"comment":"The four subplots lack axis labels, and the caption references (a)-(d) but the displayed figure does not label the subplots clearly.","section":"Figure 4"},{"comment":"Table 12 is not cited in the main text; it should be referenced where per-language results are discussed.","section":"Section 5"}],"recommendation":"major_revision","confidential_remarks":"The central problem is the silver-label test set. The authors should be asked to re-run their main experiments on the human-validated subset or provide human labels for the test languages; without that, the reported curves cannot support the paper's conclusions. The framework and dataset construction have merit, but the evaluation must be reworked. I recommend major revision rather than rejection because the core idea is sound and the required fix is clearly scoped, though it will require substantial additional annotation or re-analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take before you read it: this paper brings a genuinely new idea—a four-way annotation of proverbs (Metaphorical, Moral/Advisory, Cause-Effect, Culture-Specific)—and a careful set of experiments on multilingual supervision. But the evaluation runs on labels the authors' own weak-supervision pipeline produced, so the headline findings are really measuring agreement with that labeler, not with human judgment.\n\nWhat's actually new and worth credit: the annotation framework is novel, grounded in Relevance Theory, and moves beyond metaphor-centric taxonomies. The authors annotate 83 concepts by hand, report inter-annotator agreement, and release data. They test five model families, use McNemar with Holm correction, and are refreshingly explicit about limitations, including that the silver labels 'should not be regarded as human gold annotations.'\n\nWhere it falls down: the central claims sit on weak labels. Binary test labels come from XLM-R Base; fine-grained subgroup membership comes from E5+LR, which reaches 0.36 Macro-F1 on the human seed. Labels are propagated from source to all translations, which the authors themselves admit doesn't guarantee instance-level preservation. So the 50% saturation point and the large Culture-Specific gains could be properties of the labeler, not of proverbs. The 'combining all forms is best' claim is also confounded: the All-target condition uses the full training set, while the per-form conditions use 600 instances. And 50% vs 100% is never statistically compared; only 0% vs 100% is.\n\nWho should read it: anyone designing multilingual figurative benchmarks or working on proverb understanding. It's a useful framework and a transparent failure mode of weak supervision. It deserves a serious referee—the issues are fixable with a human-labeled test set or per-language validation. But as it stands, I'd treat the empirical findings as hypotheses, not results.","headline":"Genuinely new annotation framework for proverbs, but the central findings are measured against labels the authors' own weak-supervision models produced, so the curves describe agreement with the labeler.","tokens_in":16960,"tokens_out":2398,"would_cite":false,"duration_ms":23970,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding translated proverb data improves figurative-language identification, with performance plateauing near 50% of the multilingual training set.","keywords":["figurative language identification","multilingual supervision","proverbs","multidimensional annotation framework","culture-specific figurative meaning","weak supervision","cross-lingual transfer","Macro-F1 evaluation"],"falsifier":"Have human annotators label a held-out sample of the test proverbs in all seven languages with the same four-form guideline, then re-run the 0%, 10%, 50%, and 100% supervision curves on those human labels; the central claim would be wrong if the plateau is not near 50% or if Culture-Specific is not the largest gainer.","tokens_in":15919,"feed_emoji":"🌍","tokens_out":13076,"duration_ms":113655,"temperature":0.7,"pith_summary":"The paper asks whether translated multilingual training data helps a model recognize whether a proverb is figurative, and how much translated data is actually needed. Using 742 proverb concepts across 6,787 instances in Arabic, English, French, German, Russian, Japanese, and Spanish, it evaluates five models (logistic regression, multilingual BERT, XLM-R Large, and two instruction-tuned LLMs) at 0%, 10%, 50%, and 100% multilingual supervision, measuring Macro-F1. The central claim is that about 50% of the translated training data is enough to reach near-optimal figurative-identification performance, and that the rarest figurative form, Culture-Specific, gains the most from multilingual supervision. The paper also introduces a four-part annotation framework for proverbs: Metaphorical, Moral/Advisory, Cause-Effect, and Culture-Specific. If correct, this would mean multilingual figurative understanding should be trained and evaluated per figurative form rather than treated as one metaphor-centric task.","feed_headline":"Half the translated data hits near-optimal proverb detection","feed_subtitle":"Adding translated proverbs lifts figurative detection; rare culture-specific sayings gain the most.","key_machinery":"The machinery is a concept-level, multidimensional annotation framework grounded in Relevance Theory, which characterizes each proverb by four non-exclusive figurative forms: Metaphorical, Moral/Advisory, Cause-Effect, and Culture-Specific. A weak-supervision pipeline scales a human-validated seed of 83 proverb concepts to the full 6,787-instance dataset: XLM-R Base produces the binary figurative/literal labels, E5 embeddings with logistic regression produce the four fine-grained labels (reaching a Macro-F1 of 0.36 on that task), and labels are propagated from each source proverb to all of its translations. The framework carries the argument by making it possible to compute supervision curves per figurative form rather than only for a single figurative-versus-literal label, and the Jaccard-overlap analysis shows Culture-Specific is the most distinct form, which is the precondition for interpreting its large gains as a distinct effect.","core_discovery":"The paper's central discovery is that multilingual supervision does not help all figurative meanings equally, and its signature result is the rarest form benefiting most. In a controlled comparison, adding translated multilingual training instances to a fixed source-language training set improves binary figurative-identification Macro-F1 for every model, and the curve flattens around 50% of the translated data: moving from 50% to 100% produces smaller gains than moving from 0% to 50%. Subgroup analyses show Culture-Specific proverbs, the least frequent form and the one with the lowest overlap with the other three, post the largest F1 gains under multilingual supervision, for example from 0.481 to 0.808 for logistic regression and from 0.278 to 0.796 for XLM-R Large, while the Literal/None subgroup sometimes declines. The paper further claims that training on all four figurative forms together gives the strongest overall performance, and that Moral/Advisory and Culture-Specific forms contribute most to instruction-tuned LLMs. It emphasizes that these results hold at the distributional level: the prevalence of the four forms is preserved across languages, but the paper does not claim every individual translation preserves the same cultural interpretation.","pith_inferences":["Editorial inference: If the Culture-Specific gain is driven by its low overlap with the other forms, then multilingual supervision may help most for any rare, semantically distinct figurative category; a testable prediction is that irony, allusion, or historically grounded sayings would show similar gain curves.","Editorial inference: The paper's distributional-preservation result does not establish that individual translations keep the same cultural meaning, so a direct concept-level comparison of aligned translations with human judgments would be the natural next experiment to confirm the culture-specific effect is real grounding rather than label propagation.","Editorial inference: The 50% plateau gives a practical data-collection target for proverb-style figurative tasks: collecting roughly half of a translated training set may capture most of the benefit, letting future benchmarks allocate annotation effort to more languages or forms instead."],"forward_implications":["Adding translated multilingual training data significantly improves binary figurative identification in proverbs, with all five models fixing more test errors than they introduce when supervision moves from 0% to 100%.","About 50% of the translated training data is enough to approach the performance ceiling; doubling to 100% adds only small gains, so the multilingual benefit is not simply proportional to data volume.","The four figurative forms are not equally transferable: Metaphorical gives the strongest gains for mBERT and Aya, Moral/Advisory for Qwen2.5-3B, and Cause-Effect for XLM-R Large, but combining all forms yields the best overall performance.","Culture-Specific, the rarest and most distinct form, gains the most from multilingual supervision, suggesting that multilingual data differentially helps figurative knowledge that is hardest to learn from shared cues."],"supporting_citations":[{"why":"Supplies the Relevance Theory view of proverb comprehension as inferential, grounding the four-form annotation framework.","marker":"(Sperber and Wilson, 1986)"},{"why":"Provides the aligned multilingual proverb translation dataset from which the 742-concept, 6,787-instance benchmark is built.","marker":"(Alshaalan et al., 2026)"},{"why":"Contributes the MMFLD multilingual multi-figure sentences used to train the binary weak-supervision labeler.","marker":"(Lai et al., 2023b)"},{"why":"Contributes the MAPS short-proverb sentences across six languages to the weak-supervision training pool.","marker":"(Liu et al., 2023)"},{"why":"Contributes Arabic idiom examples that close the Arabic coverage gap in weak-supervision training.","marker":"(Benamara et al., 2023)"},{"why":"XLM-R is both the model family whose Base variant generates binary silver labels and whose Large variant is evaluated.","marker":"(Conneau et al., 2019)"},{"why":"Defines mBERT, one of the five evaluated architectures in the supervision-curve experiments.","marker":"(Devlin et al., 2018)"},{"why":"Defines Aya Expanse 8B, the instruction-tuned LLM whose form-level contributions are analyzed.","marker":"(Dang et al., 2024)"},{"why":"Defines Qwen2.5-3B, the instruction-tuned LLM that reaches the highest Macro-F1 on four of the seven languages.","marker":"(Qwen Team, 2024)"}],"fun_headline_variants":["Half the translations reach near-optimal proverb detection","Multilingual training: rarest proverb forms benefit most","50% of translated data matches full proverb performance","Culture-specific proverbs gain the most from multilingual data","Diminishing returns after half the translated proverb data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated labels used for training and testing are accurate enough: most of the dataset is labeled by a weak-supervision model, the fine-grained labeler reaches only 0.36 Macro-F1, and every translation of a source proverb receives the same label, so a systematic labeling bias would make the 50% plateau and the culture-specific gains artifacts of the labelers rather than properties of proverbs.","fun_headline_variants_meta":{"raw":{"variants":["Half the translations reach near-optimal proverb detection","Multilingual training: rarest proverb forms benefit most","50% of translated data matches full proverb performance","Culture-specific proverbs gain the most from multilingual data","Diminishing returns after half the translated proverb data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000941,"raw_usage":{"total_tokens":4043,"prompt_tokens":987,"completion_tokens":3056,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":2982}},"tokens_in":603,"tokens_out":3056,"duration_ms":24285,"temperature":1.0,"reasoning_tokens":2982,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:26:36.419351+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators label a held-out sample of the test proverbs in all seven languages with the same four-form guideline, then re-run the 0%, 10%, 50%, and 100% supervision curves on those human labels; the central claim would be wrong if the plateau is not near 50% or if Culture-Specific is not the largest gainer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Relevance Theory view of proverb comprehension as inferential, grounding the four-form annotation framework."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the aligned multilingual proverb translation dataset from which the 742-concept, 6,787-instance benchmark is built."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes Arabic idiom examples that close the Arabic coverage gap in weak-supervision training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Aya Expanse 8B, the instruction-tuned LLM whose form-level contributions are analyzed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Qwen2.5-3B, the instruction-tuned LLM that reaches the highest Macro-F1 on four of the seven languages."}],"review_version":1}