{"id":"663eddcd-bef2-4896-81b1-f5fd2edd3032","arxiv_id":"2608.09766","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Cultivar is a locale-localised FLORES benchmark whose paired contrastive instances reveal that translation-specialised models are less robust to localised content, two models may be overfit to FLORES, and US-grounded content is translated better regardless of source language.","lead":"This paper introduces Cultivar, a translation benchmark that rewrites FLORES sentences to reflect 27 different locales, pairing each localised sentence with its original counterpart. It evaluates 32 open-weight models and finds that translation-specialised models are less robust to localised content, some models may be overfit to FLORES itself, and models translate US-themed content better than content from other regions, even in the same language.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The contrastive delta metric confounds localisation robustness with reference difficulty, so the contamination and US-bias findings require a reference-difficulty control before they can stand.","rationale":"The Reader's weakest assumption identifies exactly the load-bearing concern of this paper: DeltaBLEU compares scores against different reference sentences, so the metric cannot distinguish localisation robustness from reference difficulty. This concern is central because it underlies all three headline findings: MT-specialised models being less robust, potential FLORES overfitting, and US-centric bias. The paper's own Section 4.4 explicitly acknowledges that locale difficulty may reflect data artifacts, and Figure 3's US-bias conclusion rests on only two languages, both of which have US localisations that are structurally closest to the original FLORES references. The proposed perplexity-control test provides a direct, quantitative way to check whether per-locale delta variation is explained by reference predictability. If the correlation is strong, the empirical findings would need to be reframed as reference-effects rather than model behaviour, though the released benchmark and the ranking-consistency results in Table 2 would remain valuable. Because the resource is useful and the weakness is addressable, a conditional verdict remains appropriate pending this control; no change to the Reader's verdict is needed.","tokens_in":18454,"tokens_out":8563,"duration_ms":79109,"concrete_test":"Compute, for each of the 27 locales, the mean per-token negative log-likelihood (perplexity) of the original FLORES English reference sentences and of the localised Cultivar English reference sentences under a fixed strong language model (e.g., Llama-3.3-70B or GPT-5.5 with greedy decoding). Define Delta_ppl = mean_ppl(localised refs) - mean_ppl(original refs). Then calculate Spearman's correlation across locales between Delta_ppl and the paper's mean per-locale DeltaBLEU in Table 4, and likewise between Delta_ppl and mean Delta chrF. If |rho| > 0.5 with the expected sign (easier references, i.e., lower perplexity after localisation, correspond to positive Delta), the delta metric is dominated by reference difficulty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the original FLORES and localised Cultivar test sets have equal reference difficulty, so that DeltaBLEU (Section 3.2) isolates localisation robustness. This is unverified. Because BLEU and chrF are computed against different reference sentences for the two sets, any systematic difference in the predictability or n-gram distribution of the references shifts Delta for all models. Section 4.4 concedes that locale difficulty may reflect 'data artifact relative to the original FLORES instances,' and the US-bias result in Figure 3 is drawn from only Hindi and Chinese, where the US localisations have among the smallest edit distances to FLORES (Table 1: cmn_Hans_US has 180 LLM paraphrases kept and none reverted; hin_Deva_US also shows minimal reversion). If US localised references remain closer to the original English distribution, models will score higher on them for reasons unrelated to training-data bias. Similarly, the large negative Delta values for aya-expanse-8b and Seed-X-PPO-7B could reflect that their outputs, even when high-quality, match a harder localised reference less well. Without a reference-difficulty control, the contamination and US-centric-bias conclusions are ungrounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Cultivar, a locale-oriented translation benchmark built by localising a named-entity-rich subset of the FLORES dev set into 27 language-script-location combinations through an LLM-assisted, human-post-edited pipeline. It proposes a source-contrastive evaluation paradigm: models are evaluated on localised Cultivar instances and their unlocalised FLORES counterparts, and the discrepancy, measured by ΔBLEU and ΔchrF, is interpreted as localisation robustness and, together with false-positive/false-negative word recall, as a probe for benchmark contamination. The paper reports a benchmark of 32 open-weight models and three main findings: MT-specialised models are less robust to localisation, a few models may overfit FLORES, and models tend to translate US-grounded content better than content from other locales regardless of language.","tokens_in":18629,"tokens_out":8431,"duration_ms":75879,"significance":"If the central metric were validated, Cultivar would be a valuable and reusable diagnostic resource for locale-aware MT evaluation and for detecting sensitivity to distribution shift and possible test-set contamination. The construction is careful: annotator selection criteria are explicit, human post-editing is substantial (over 98% of instances are newly localised content), a detailed annotation interface and instructions are provided, and the dataset is publicly released. The source-contrastive design is a worthwhile methodological proposal that goes beyond static single-version test sets. However, the current analysis does not yet validate the claim that ΔBLEU and ΔchrF isolate localisation robustness, because the two test sets have different reference sentences; the headline findings are therefore more preliminary than the abstract suggests. The paper also ships the evaluation results for 32 models, which is useful for the community even if some interpretative claims need revision.","major_comments":[{"comment":"The central metric ΔBLEU = BLEU_localised − BLEU_original compares scores computed against two different reference sets, so it measures a compound of localisation robustness and any systematic difference in reference difficulty. The paper provides no control for reference difficulty—e.g., human quality judgements on a sample, language-model perplexity of the two reference sets, or a calibration set of equal-difficulty pairs—and §4.4 explicitly concedes that locale difficulty 'could be attributed to ... data artifact relative to the original FLORES instances.' Without such a control, the three headline findings (MT-specialised models less robust, potential FLORES overfitting, US-centric bias) are not established. The 'random baseline' model (SmolLM-1.7B-Instruct in Table 3) does not resolve this: a garbage-producing model can have Δ≈0 simply because its outputs are about equally distant from both references, regardless of reference difficulty.","section":"§3.2, Eq. (1)–(2)"},{"comment":"The claim that models 'tend to translate US content better than that of other locales, regardless of language' rests on only two languages (Hindi and Chinese) and only into English, and for these languages the US localisations are the ones with the smallest deviation from FLORES (cmn_Hans_US: 180 LLM paraphrases kept and 0 reverted; hin_Deva_US: 113 kept and only 8 edited). The abstract's 'regardless of language' is an overgeneralization from this evidence. Moreover, since the US entries are flagged with a dagger as non-official locales, the comparison is between official-locale localisations and a deliberately extra-territorial US variant; any tendency for the US variants to be lexically closer to the original English reference would produce exactly the observed positive Δ for US without implying a training-data bias.","section":"§4.5, Figure 3 and Table 1"},{"comment":"The interpretation that aya-expanse-8b and Seed-X-PPO-7B 'potentially overfit FLORES' is not directly supported by the false-positive analysis, which shows no elevated verbatim recall for these two models (Table 3: F Pos ≈ 0.01 for both, comparable to all other models). The text itself hedges appropriately with 'could hint at,' but the abstract reports the finding as 'a few models potentially overfit FLORES.' A large negative Δ can arise from sensitivity to the lexical distribution shift or from the localised references being harder for these specific models. To make this claim load-bearing, the authors would need a positive control—for example, a model known to be trained on FLORES and a comparable model known not to be—or a demonstration that the large Δ persists after controlling for reference difficulty.","section":"§4.3, Table 3"},{"comment":"Each locale was produced by a single annotator (Table 1 lists per-locale counts with no annotator overlap information). Consequently, cross-locale comparisons, including the US versus non-US comparisons in Figure 3, entangle the locale variable with annotator style, personal bias, and cultural knowledge. The paper acknowledges individual annotator bias in §4.4 but does not quantify it. Since the goal of the benchmark is to isolate locale-driven differences, the absence of any inter-annotator agreement measure, or at least a multi-annotator reliability study on a subset, is a threat to the validity of the resource as a controlled diagnostic. This is particularly relevant to the US-bias claim, which depends on comparing localisations made by different annotators.","section":"§2.3–2.4, Table 1"}],"minor_comments":[{"comment":"The definitions of false positives and false negatives are ambiguous: 'unique terms in the original FLORES reference' could be read as all unique terms in that reference, but the intended meaning is evidently 'terms that appear only in the original reference and not in the localised reference' (and symmetrically for false negatives). Please make this set-difference explicit to avoid misreading.","section":"§3.2"},{"comment":"The dagger symbol for the four non-official-locale entries is mentioned only in the body text; please define it in the table caption as well, because the table is otherwise self-contained.","section":"Table 1 caption"},{"comment":"The two panels show ΔBLEU and ΔchrF for Hindi and Chinese across locations, but it would be clearer to use distinct line types or colors for the two languages; the current plot makes it easy to misread which points belong to which language.","section":"Figure 3 caption"},{"comment":"The statement 'If we treat these weaker models with FN>44% as random baselines' introduces an arbitrary threshold; please justify this cutoff or show that the conclusion is robust to its choice.","section":"§4.2"},{"comment":"The grouping 'MT-specialised models' in Table 3 mixes pre-trained MT models (NLLB, MADLAD) with LLM-based translation models (Hy-MT2, Seed-X-PPO-7B); consider defining the categories explicitly or reporting subgroup analyses, since the 'MT-specialised are less robust' claim is aggregated over these heterogeneous groups.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"The dataset is a useful community resource and the construction pipeline is careful, but the quantitative claims in the abstract go beyond what the current analysis supports because of the unvalidated reference-difficulty assumption in the Δ metrics. I recommend major revision with a focus on validating the contrastive metric—for example, through a reference-difficulty control or human evaluation—and on scaling back or carefully qualifying the contamination and US-bias claims. The single-annotator design is also a validity concern that should be addressed explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth taking seriously. Cultivar is a useful, carefully built resource—paired FLORES and localised instances, human post-editing, 27 locales, released on HuggingFace. The source-contrastive idea is a real step beyond language-pair evaluation, and the pipeline is described well enough to replicate. The model ranking correlations between FLORES and Cultivar are high, the false-positive analysis gives some evidence against verbatim memorization, and the manual error analysis adds qualitative support. The authors also deserve credit for acknowledging in Section 4.4 that locale difficulty may reflect data artifacts relative to FLORES. That honesty matters.\n\nThe soft spots are real but not fatal. The stress-test note is right: delta BLEU and delta chrF are computed against different reference sets, so any systematic difference in reference difficulty shifts the deltas for all models. That means the contamination and US-bias findings, as stated in the abstract, outrun the evidence. The paper itself uses hedged language in the body—\"potential overfitting,\" \"could hint\"—so the gap is mostly between the abstract/conclusion and what Section 4 actually supports. I would also flag: no significance testing on the deltas, one annotator per locale, and the US-bias generalization drawn from only Hindi and Chinese. The manual error analysis helps, but it covers one language pair and a subset of models.\n\nThe central resource does not depend on the contested interpretations. Even if the empirical findings are softened, the benchmark itself is reusable and the contrastive method is a legitimate diagnostic. The right fix is to add a reference-difficulty calibration—for example, having independent annotators rate the localised vs original references for naturalness and predictability, or using additional references for both conditions—and to narrow the claims accordingly.\n\nI'd send this to peer review. It deserves a serious referee because the resource is valuable and the method will influence how people think about multilingual evaluation. But I would expect revision: either a control for reference difficulty or substantially more cautious wording of the contamination and US-bias conclusions.","headline":"A genuinely reusable locale-contrastive benchmark with a careful construction pipeline, whose headline empirical claims need one more round of controls before they can be stated as firmly as the abstract does.","tokens_in":19319,"tokens_out":1930,"would_cite":true,"duration_ms":21241,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Swap US names for local ones in a translation test, and most models translate the US version best while some appear to have memorised the original FLORES set.","keywords":["machine translation evaluation","localisation","data contamination","FLORES","contrastive evaluation","named entities","multilingual benchmark","US-centric bias"],"falsifier":"Build a second Cultivar-like localisation of the same FLORES sentences with independent annotators for a few locales; if the model deltas and the US advantage shift or flip, the measured effects are artifacts of the particular paraphrases. Alternatively, compare deltas against human-rated translation errors on a random sample of localised outputs; if locales with negative deltas show no more human errors than the US split, the delta reflects reference difficulty rather than model robustness.","tokens_in":18233,"feed_emoji":"🌍","tokens_out":5190,"duration_ms":40193,"temperature":0.7,"pith_summary":"This paper argues that multilingual translation benchmarks should be evaluated in a source-contrastive way: the same sentence structure, but with named entities and cultural references swapped for a specific locale. It introduces Cultivar, a 27-locale subset of FLORES created by LLM paraphrase plus human post-editing, and pairs it with the original FLORES to measure translation performance deltas. Benchmarking 32 open-weight models, it finds that MT-specialised and smaller models drop more on localised content, that two models show large deltas consistent with FLORES overfitting, and that models generally translate US-grounded content better than content from other locales of the same language. If correct, the contrastive design turns a static test set into a reusable diagnostic for localisation robustness and data contamination.","feed_headline":"Localised names expose US bias in 32 translation models","feed_subtitle":"Cultivar re-grounds FLORES sentences in 27 locales; MT-specialised models and a few flagship LLMs drop most.","key_machinery":"The central mechanism is the contrastive delta, Δ = score_localised − score_original, computed with spBLEU and chrF via sacrebleu; a negative delta means a model degrades when the source content is grounded in a specific locale. The Cultivar construction pipeline—named-entity selection with spaCy, LLM paraphrasing that preserves sentence structure, and human post-editing by a locale-native annotator—produces the paired test sets. A false-positive/false-negative word-recall analysis further checks whether models regurgitate the original FLORES wording after localisation.","core_discovery":"The paper claims that translation evaluation should be source-contrastive: instead of one fixed test instance per language, a benchmark should contain paired instances that share sentence structure but vary locale-specific named entities and cultural references. It constructs Cultivar by localising 200 FLORES dev-set sentences that contain at least two named entities into 27 language-script-location combinations, using an LLM to paraphrase and a human annotator who lives in the locale to post-edit. On this data, the paper reports three findings: MT-specialised models are less robust to localisation than general LLMs, as measured by mean pairwise ΔBLEU and ΔchrF; two models (aya-expanse-8b and Seed-X-PPO-7B) show large negative deltas that the authors interpret as probable FLORES overfitting; and for Hindi and Chinese localised to multiple regions, models consistently translate US-grounded content better than content from other locales of the same language. The paper positions Cultivar as a diagnostic tool that separates localisation sensitivity from general translation ability.","pith_inferences":["If independently re-localised counterparts were built, the US advantage could be separated into a model bias and a benchmark artifact; the paper's single-annotator-per-locale design leaves that confound open.","The same contrastive design could probe locale robustness in other generation tasks, such as summarisation or instruction following, by pairing a global and a localised version of the same prompt.","In production, a deployment team using one of the flagged MT-specialised models should expect disproportionately higher error rates when source content mentions locally specific people, places, or units."],"forward_implications":["Model rankings on FLORES are highly correlated with rankings on Cultivar (average Spearman 0.98, Kendall 0.89), so the contrastive set can be used without disrupting standard leaderboards.","After training, a model with a large negative Δ on Cultivar should be treated as memorising the FLORES style even if its word-level false-positive rate is low.","The source-contrastive pipeline can be applied to other existing benchmarks, not just FLORES, to build locale-specific variants for more regions.","For Chinese and Hindi, the positive US deltas imply that multilingual models are better at translating content originally written for a US audience than for other regions of the same language."],"supporting_citations":[{"why":"FLORES is the base test set whose English dev instances Cultivar localises.","marker":"Goyal et al., 2022"},{"why":"The FLORES-200 extension supplies the multilingual original test sets and the tokenizer used for spBLEU.","marker":"Team et al., 2024"},{"why":"NTREX is an alternative multilingual MT test suite, used here to frame the problem.","marker":"Federmann et al., 2022"},{"why":"BOUQuET is the alternative benchmark that starts from native texts, against which the paper contrasts its approach.","marker":"Andrews et al., 2025"},{"why":"Establishes data contamination from memorisation as a threat to benchmark validity.","marker":"Magar and Schwartz, 2022"},{"why":"Argues that benchmarks need contamination measurement, motivating the paper's diagnostic purpose.","marker":"Sainz et al., 2023"},{"why":"CODET provides a prior contrastive-dialectal evaluation method that this work extends to locales.","marker":"Alam et al., 2024"},{"why":"Defines BLEU, one of the two metrics used for the contrastive delta.","marker":"Papineni et al., 2002"},{"why":"Defines chrF, the character-level metric used alongside BLEU.","marker":"Popović, 2015"},{"why":"Sacrebleu provides the standard implementation of BLEU and chrF with the FLORES tokenizer.","marker":"Post, 2018"}],"fun_headline_variants":["Translation models favour US content over other locales","Locale-focused benchmark exposes US bias in 32 models","MT-specialised models less robust to localisation shifts","Cultivar reveals translation models' blind spots for non-US locales","New translation benchmark highlights US-centric model bias"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that swapping named entities into another locale leaves a sentence neither easier nor harder to translate, so any score gap is caused by the model's handling of localised content rather than by the reference itself.","fun_headline_variants_meta":{"raw":{"variants":["Translation models favour US content over other locales","Locale-focused benchmark exposes US bias in 32 models","MT-specialised models less robust to localisation shifts","Cultivar reveals translation models' blind spots for non-US locales","New translation benchmark highlights US-centric model bias"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000704,"raw_usage":{"total_tokens":3145,"prompt_tokens":885,"completion_tokens":2260,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2184}},"tokens_in":501,"tokens_out":2260,"duration_ms":14737,"temperature":1.0,"reasoning_tokens":2184,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:09:42.808978+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build a second Cultivar-like localisation of the same FLORES sentences with independent annotators for a few locales; if the model deltas and the US advantage shift or flip, the measured effects are artifacts of the particular paraphrases. Alternatively, compare deltas against human-rated translation errors on a random sample of localised outputs; if locales with negative deltas show no more human errors than the US split, the delta reflects reference difficulty rather than model robustness.","supporting_citations":[],"review_version":1}