{"id":"5f3994ed-b7c7-4288-ab78-6e5bce34e394","arxiv_id":"2505.06010","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new multilingual evaluation dataset and model comparison show that neural MT systems vary widely in how faithfully they transfer URLs, emails, IBANs, and emojis, with EuroLLM 9B leading and emojis remaining a hard failure mode.","lead":"The authors build a 36,000-sentence synthetic benchmark in four languages to test whether machine translation models copy entities like emails, URLs, and emojis without changing them. Across eight models, they find large quality gaps, with EuroLLM 9B performing best and the OPUS model family worst.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's ground-truth regexes are internally defective: Appendix A has no emoji pattern, and the phone regex drops a leading '+' via its \\b anchor; if these change Table 2, the reported rankings are not reliable.","rationale":"The reader's weakest assumption points to the validity of the hand-crafted regular expressions as ground truth and to the representativeness of the synthetic sentences. My concern lands squarely on the regex validity part, but more specifically: the manuscript itself fails to provide an emoji regex, and the phone regex's word-boundary placement silently drops the leading '+', which is semantically part of many international phone numbers. These are concrete internal defects, so they are the most load-bearing threat to the exact numbers in the strongest claim (EuroLLM 95.89, Google 93.32, OPUS 45.68, and the emoji failure cluster). I do not emphasize the synthetic-representativeness concern because the paper explicitly labels the evaluation as a sanity check and a controlled benchmark; internal measurement validity must be settled first. The proposed test is a re-run of Table 2 with two targeted regex fixes; if the scores are stable, the concern does not land and the conditional verdict stands. If the scores shift materially, the rankings need re-reporting with corrected extraction, keeping the paper conditional rather than rejectable, since the flaw is fixable and the qualitative finding of model variability may survive. This does not move the reader's CONDITIONAL verdict, so I recommend UNCHANGED.","tokens_in":16569,"tokens_out":11684,"duration_ms":123465,"concrete_test":"Recompute Table 2 with corrected extraction: add an explicit emoji detector using Unicode Extended_Pictographic or an emoji library, and rewrite the phone regex to capture an optional leading '+' (for example, \\b\\+?[0-9][0-9\\s\\-]{2,}[0-9]\\b). Then compare per-category and macro-average scores for all eight models. If any model's macro-average shifts by more than about 1 point, or the sub-5.5% emoji cluster changes membership, the headline numbers are artifacts of the regex ground truth rather than stable model abilities.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's accuracy numbers are computed entirely by the hand-crafted regular expressions in Appendix A (Section 5.3), but that ground truth is incomplete in at least two concrete ways. First, Appendix A lists regexes for alphanumeric, email, IBAN, IP, ISBN, phone, social, and URL categories, but has no emoji regex, even though emoji is one of the nine dataset categories and the source of the paper's most striking finding (Table 2: five models below 5.5%). Section 3.2 says each sentence was filtered to contain exactly one entity of the expected category 'using regular expressions listed in Appendix A'; without an emoji pattern, the emoji accuracy column and the related no-match analysis are not reproducible from the manuscript. Second, the phone regex begins with \\b before a character class that includes '+'. Because '+' is a non-word character, \\b is false at the start of '+49...', so the match begins at the following digit and the leading '+' is never captured. The extraction therefore compares '49 030 1234567890' rather than '+49 030 1234567890', and a model that deletes the '+' is counted as correct. This systematically inflates phone-transfer accuracy for every model; since phone scores are among the highest per-category numbers in Table 2, the macro-averaged rankings can be distorted. These are internal measurement-pipeline defects, not merely external representativeness concerns, so they threaten the exact numeric claims and the reliability of the dataset as a measuring instrument.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript constructs a new synthetic multilingual benchmark for evaluating whether machine translation systems preserve no-translate entities. The dataset consists of 36,000 sentences in English, German, Polish, and Ukrainian, covering nine categories (alphanumeric sequences, emails, emojis, IBANs, IPs, ISBNs, phone numbers, social handles, URLs). Eight translation systems (OPUS, mBART, NLLB, M2M100, EuroLLM, MADLAD, SeamlessM4T, Google Translate) are evaluated across all 12 translation directions. Correct transfer is measured by extracting entities with hand-crafted regular expressions and comparing source and target sequences via Levenshtein distance. The main results (Table 2) show EuroLLM 9B as the best (95.89% macro-average), Google Translate second (93.32%), and OPUS worst (45.68%), with emoji being the most difficult category for most models. The authors also analyze error types, prompt effects for EuroLLM, and correlations with entity length and sentence length.","tokens_in":16850,"tokens_out":7973,"duration_ms":73184,"significance":"If the evaluation pipeline can be corrected and the numbers reproduced, the paper would provide a useful, larger-scale benchmark than existing entity-transfer sets such as ACES, with 1,000 examples per category/language cell and publicly released data and translations. The breadth is a clear strength: eight models, twelve directions, nine categories, and an error analysis that goes beyond binary accuracy. The emoji finding, however, rests on a detection mechanism that is not documented in the manuscript, and the phone-number regex is objectively incorrect for international prefixes. Until these issues are fixed, the headline rankings cannot be trusted. The CometKiwi sanity checks and the correlation analyses are informative additions, and the authors deserve credit for transparently reporting their generation and sampling procedures.","major_comments":[{"comment":"The emoji accuracy numbers are not reproducible. Section 3.2 states that each sentence was filtered to contain exactly one entity of the expected category 'using regular expressions listed in Appendix A', and Section 5.3 states that the same expressions are used for source-target comparison. However, Appendix A lists patterns for only eight categories and contains no emoji pattern, even though emoji is one of the nine dataset categories and drives the most striking result in Table 2 (five models below 5.5%). Without a documented emoji detection rule, the emoji column, the no-match error fractions attributed to emojis in Section 6, and the corresponding Figure 4 results cannot be verified.","section":"3.2, 5.3, Appendix A, Table 2"},{"comment":"The phone-number expression is defective for international numbers. The pattern starts with \\b before a character class that includes '+'; because '+' is a non-word character, \\b is false before '+49...', so the regex matches only from the first digit onward and never captures the leading '+'. Table 10 confirms this: srcEntity is '49 030 1234567890' while the source sentence contains '+49 030 1234567890'. A model that deletes the '+' is therefore counted as correct. Since phone numbers are among the highest-scoring categories in Table 2, this systematically inflates the phone accuracy for every model and distorts the macro-averages and the claim that IP, phone, and ISBN are the three easiest categories.","section":"Appendix A (Phone regex), Table 10"},{"comment":"The EuroLLM scores in the main comparison are obtained with a prompt chosen by evaluating two prompt variants on the same test set used for the headline evaluation. This is a form of test-set selection; the other models were evaluated only with their default configurations. The reported 95.89% for EuroLLM 9B is therefore not directly comparable to the other systems. The authors should either report both prompt conditions in the main table or select prompts on a held-out development set before computing the final rankings.","section":"Section 6, 'Prompt selection for EuroLLM', Tables 2 and 5"}],"minor_comments":[{"comment":"The list of best directions repeats 'de→ pl' twice; one of the two entries should presumably be 'pl→ de'.","section":"Table 3"},{"comment":"The caption says 'Average change when using large models and smaller ones', but the table contains absolute accuracy values, not changes; the caption should be revised.","section":"Table 5"},{"comment":"Entries such as '1,225 (352)' and '1 (11)' are unexplained; please clarify what the parenthesized numbers denote.","section":"Table 8"},{"comment":"The text uses 'Spearmans's' where 'Spearman's' is intended.","section":"Section 6"},{"comment":"Appendix C acknowledges the synthetic nature of the test set, but Section 3.2 calls the resulting data a 'high-quality dataset' without reporting any human validation; the authors should state whether a manual inspection of a sample was performed.","section":"Appendix C and Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper addresses a practical problem and the dataset could be valuable, but the two regex issues are internal correctness problems that must be fixed before the numerical claims can be defended. The prompt-selection concern also needs a response. I do not see these flaws as unfixable within a revision, and the underlying empirical contribution is worth preserving."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The bottom line: this is a useful benchmark paper, not a revolution. It contributes a 36k-sentence synthetic test set for no-translate entities across nine categories and four languages, and a comparative measurement of eight MT systems on it. The high-level picture — EuroLLM 9B and Google Translate lead, OPUS trails, emoji is a failure class for five models — is credible in direction and worth knowing.\n\nWhat's new and what's done well: prior work (ACES, DEMETR, SMAUG) was built to stress metrics, not systems, and had very few examples per interaction. Here 1,000 examples per category/language cell and all 12 directed pairs support statistically stronger statements. The error analysis with Levenshtein distances, CometKiwi sanity checks, and the explicit EuroLLM prompt comparison are honest and useful. Publishing the dataset and translations is a real plus.\n\nThe soft spots are real, and one of them is load-bearing. The evaluation ground truth is the regex set in Appendix A. There is no emoji regex in that appendix, even though emoji is a headline category and the section says selection and detection used these regexes. Also, the phone regex starts with \\b before a class that includes '+'; since '+' is a non-word character, the regex cannot match the leading plus. It matches from the first digit, so a model that deletes '+' is scored correct. Phone scores are among the highest per category, so this systematically inflates macro averages and can shift rankings. These internal defects are not just representativeness worries; they make the exact numbers in Table 2 unreproducible as written.\n\nThe EuroLLM prompt selection is another real issue: the focused prompt was chosen after evaluating on this test set. That is selection on the label. Other models did not get instruction tuning, so the comparison is not apple-to-apples. They should either tune prompts on a held-out set or clearly label EuroLLM as the prompt-tuned upper bound. The causal claim about byte-level tokenization is asserted with a non-significant correlation and one example; I'd cut that. The dataset is synthetic and unvalidated by humans, so how far rankings transfer to natural text remains open.\n\nMinor but worth a copyedit: Table 10 has a mislabeled direction (says en→de, source is Polish), Table 8's 'supported languages' column has implausible values, and the per-direction prose lists de→pl twice. No evaluation code or commit hash is provided.\n\nWho is this for? MT practitioners choosing systems and benchmark builders wanting a denser no-translate entity set. It deserves a serious referee, but as a major revision. I would send it out.","headline":"Useful new 36k no-translate entity benchmark, credible high-level rankings, but the regex-based ground truth has a missing emoji pattern and a phone pattern that masks '+' deletion — needs fixing before the numbers are trusted.","tokens_in":17414,"tokens_out":3373,"would_cite":true,"duration_ms":33546,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 36,000-sentence benchmark measures how often machine translators preserve URLs, IBANs, emails, and emojis, and finds a large gap between the best and worst models.","keywords":["no-translate entities","neural machine translation","entity transfer accuracy","multilingual benchmark","emoji translation","tokenization granularity","Levenshtein distance","synthetic dataset"],"falsifier":"Re-run the same evaluation on naturally occurring texts that contain the same entity types, or replace the regex-based ground truth with human annotations of whether each entity is unchanged; if the accuracy ordering of the eight systems changes substantially, the synthetic generation or the regexes are driving the result.","tokens_in":16340,"feed_emoji":"🌐","tokens_out":11135,"duration_ms":95993,"temperature":0.7,"pith_summary":"The paper asks whether popular neural machine translation models recognize text fragments that must be copied into the target language without modification—URLs, email addresses, IBANs, phone numbers, IP addresses, ISBNs, social media handles, alphanumeric codes, and emojis. To answer this, the authors generate a new synthetic dataset of 36,000 sentences in English, German, Polish, and Ukrainian, each containing exactly one such entity, and measure how often eight systems reproduce the entity character-for-character. The central finding is a wide quality gap: the best system, EuroLLM 9B, transfers entities correctly 95.89% of the time on a macro average, Google Translate 93.32%, and the OPUS family only 45.68%. The biggest failure class is emojis, where five of the eight systems score below 5.5%. If the dataset is representative, the paper supplies a working instrument for ranking models on this specific failure mode and for debugging why it happens.","feed_headline":"Best model copies untranslatable entities 95.9% of time; worst 45.7%","feed_subtitle":"A new 36,000-sentence benchmark reveals which machine translators garble URLs, IBANs, emails, and emojis.","key_machinery":"The load-bearing instrument is a new synthetic multilingual benchmark: 36,000 sentences, with 1,000 examples for each of nine entity categories in each of four languages, and each sentence containing exactly one entity detectable by hand-crafted regular expressions. The measurement mechanism is character-exact entity matching: a transfer counts as correct only when a regular expression for the expected category finds the identical character sequence in the source and target sentences, and mismatches are quantified by Levenshtein distance. The generation pipeline combines instruction-following sentence generation with filtering by language detection, length bucketing, grammar checking, and regex validation, which together make the dataset a controlled testbed for entity transfer.","core_discovery":"The paper establishes that no-translate entity transfer is a separable, measurable quality axis in neural machine translation, not fully predicted by overall translation quality. Across all twelve translation directions among English, German, Polish, and Ukrainian, macro-average entity-transfer accuracy ranges from 95.89% for EuroLLM 9B to 45.68% for the OPUS family, with a global average of 78.65%. Numerical sequences such as IP addresses, phone numbers, and ISBNs are transferred well by nearly all models, while emojis are the main systemic failure; models without byte-level tokenization support almost always drop them. Error analysis shows partial translation, character substitution, repetition, and omission, with one-character edits the most common failure size, and longer entities more error-prone for some models but not others. The paper also demonstrates that a focused prompt instructing the model to keep non-translatable elements unchanged improves EuroLLM both on entity transfer and on CometKiwi translation quality.","pith_inferences":["A testable extension of the prompt result: applying the same focused-instruction recipe to other instruction-tuned translation LLMs should improve their entity-transfer accuracy if the effect reflects a general property of explicit instructions rather than something specific to EuroLLM's training.","A consequence the paper leaves implicit is that production translation systems handling identifiers, bank details, or handles would benefit from a guardrail that detects these entities in the source and verifies their presence in the output, since even the best model still fails on about 4% of cases.","Because the ground truth is regex-based, the dataset measures 'string copied exactly' rather than 'entity preserved semantically'; a phone number reformatted with different spacing, or an emoji rendered as its text description, would count as wrong even if a human reader accepts the translation.","A concrete fix suggested by the emoji results: adding byte-level Unicode coverage for emojis to tokenizers should raise transfer accuracy without retraining the translation model, and this could be tested directly on the low-scoring systems."],"forward_implications":["Entity-transfer accuracy is a distinct evaluation axis: a model can score acceptably on overall quality metrics yet still alter or drop an IBAN, URL, or email in the same sentence.","Adding one explicit instruction to keep non-translatable elements unchanged raised EuroLLM 9B's macro accuracy from 94.83% to 95.89%, so prompt design is a low-cost lever for this failure class.","Emoji transfer is a sharp diagnostic: five of the eight systems scored under 5.5% on emojis, making emoji presence enough to expose tokenizer-level limitations.","Tokenization granularity appears to matter: EuroLLM splits an example ISBN into 30 subtokens and transfers it well, while OPUS splits the same number into 12 chunks and struggles, suggesting that finer-grained tokenizers handle entities better.","The dataset gives a statistically stronger view than prior challenge sets, with 1,000 examples per language-category pair and all twelve translation directions covered rather than only translations into English."],"supporting_citations":[{"why":"Supplies the instruction-following model used to generate all candidate sentences, so it determines the dataset's linguistic surface.","marker":"(Rivière et al., 2024)"},{"why":"Introduces EuroLLM, the best-scoring system in the comparison and the target of the focused-prompt experiment.","marker":"(Martins et al., 2024)"},{"why":"Provides the OPUS-MT model family, which anchors the low end of the accuracy range and the tokenization-based explanation.","marker":"(Tiedemann et al., 2024)"},{"why":"Introduces MADLAD, a high-scoring multilingual model whose 3B and 7B versions are compared to test the effect of scale.","marker":"(Kudugunta et al., 2023)"},{"why":"Defines the prior ACES challenge set for no-translate entities, against which this dataset is positioned and compared.","marker":"(Amrhein et al., 2022)"},{"why":"Introduces NLLB, one of the eight evaluated systems and a representative massively multilingual translation model.","marker":"(Costa-jussà et al., 2022)"},{"why":"Provides CometKiwi, the quality-estimation metric used to confirm that entity-transfer results are not just overall quality scores.","marker":"(Rei et al., 2022b)"}],"fun_headline_variants":["95.9% vs 45.7%: NMT entity-transfer accuracy gap","Emojis stump NMTs: new multilingual entity benchmark","New 36K test: best NMT keeps 95.9% entities, worst 45.7%","MT models drop emojis: 36K-sentence benchmark exposes gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The rankings depend on the synthetic sentences generated by the language model being representative of how these entities actually appear in real texts, and on the regular expressions being an exact definition of an unchanged entity; if either fails, the model rankings may not transfer to production use.","fun_headline_variants_meta":{"raw":{"variants":["95.9% vs 45.7%: NMT entity-transfer accuracy gap","Emojis stump NMTs: new multilingual entity benchmark","New 36K test: best NMT keeps 95.9% entities, worst 45.7%","MT models drop emojis: 36K-sentence benchmark exposes gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001894,"raw_usage":{"total_tokens":7413,"prompt_tokens":919,"completion_tokens":6494,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":6404}},"tokens_in":535,"tokens_out":6494,"duration_ms":45519,"temperature":1.0,"reasoning_tokens":6404,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:50:19.940652+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same evaluation on naturally occurring texts that contain the same entity types, or replace the regex-based ground truth with human annotations of whether each entity is unchanged; if the accuracy ordering of the eight systems changes substantially, the synthetic generation or the regexes are driving the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the prior ACES challenge set for no-translate entities, against which this dataset is positioned and compared."}],"review_version":1}