{"id":"210e3e08-5d48-4e10-8da8-a3232ee2e4c3","arxiv_id":"2505.17784","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A multilingual extension of the CUTE benchmark shows that LLM token-manipulation performance varies by language and script, with surprisingly strong results on low-resource languages and weak results on sub-character understanding.","lead":"EXECUTE is a new multilingual benchmark that asks large language models to spell, insert, delete, substitute, and swap characters and words in eight languages with different scripts. It finds that models often score higher on low-resource languages and that they understand sub-character components in Chinese, Japanese, and Korean poorly.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The inverse-resource claim conflates language knowledge with tokenization granularity, and Table 3 already contradicts it with Qwen 2.5's Hindi result.","rationale":"The reader's weakest assumption—that English prompts with generated examples are understood equally across languages—is a genuine threat to cross-language comparability, but I find a more load-bearing confound for the headline claim: the inverse-resource result is identified from languages that are also byte-tokenized, and knowledge is never measured. The paper's own CWT analysis makes this confound visible: high characters-per-token predicts difficulty on character-level tasks, and low-resource languages have very low c/t. The claimed causal story ('less knowledge weakens the language bias') is therefore not separable from a purely tokenization-driven explanation. Table 3 additionally provides a within-paper counterexample (Qwen 2.5 Hindi), which makes the inverse-resource generalization internally fragile. I do not argue the benchmark is worthless: the released framework, the multi-model results, and the sub-character tasks are valuable, and the paper honestly lists limitations. The concern is specifically that the central inverse-resource claim requires an additional measurement step. Since the reader's verdict was already CONDITIONAL, my recommendation is to keep that verdict rather than strengthen or weaken it. I do not agree fully with the reader because the prompt-comprehension issue is secondary to the knowledge/tokenization confound for the headline claim.","tokens_in":15057,"tokens_out":8455,"duration_ms":95661,"concrete_test":"Compute, for each of the 11 model checkpoints and each of the 8 languages, an independent knowledge proxy: mean per-token negative log-likelihood on a held-out sample of native text in that language, using the same models and decoding settings. Then fit an item-level mixed-effects model predicting EXECUTE accuracy from this log-likelihood and from characters-per-token (c/t) from Table 1, with model and language random effects. If the coefficient on log-likelihood is not significantly negative after controlling for c/t—or if Qwen 2.5's Hindi log-likelihood is low while its EXECUTE accuracy is high—the inverse-resource claim fails. A minimal first check is to report Qwen 2.5's Hindi and Amharic perplexities and verify whether the claimed inverse ordering holds within that model.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('Surprisingly, the less an LLM knows a language, the better it performs on EXECUTE') is inferred from a confounded contrast rather than from any measured quantity. The decisive evidence is Amharic, Tamazight, and Santali, which are simultaneously the least-resourced languages and the most byte-level tokenized, with characters-per-token of 0.48, 0.49, and 0.42 respectively. Their near-perfect scores are therefore exactly what the paper's own Table 1 CWT statistics predict, without needing any assumption about the model 'knowing' the language. The cipher and byte-level English controls do not separate tokenization from recognition: both interventions change the tokenizer's behavior, and byte-level English remains bad on word tasks, so token granularity alone cannot explain the pattern. Moreover, the claim is internally contradicted by Table 3: for the best overall model, Qwen 2.5 32B, Hindi—a relatively high-resource language—is the highest-scoring language at 86.2, while Amharic scores only 74.4, close to Arabic and below English, Chinese, Korean, Japanese, and Hindi. No model-specific language-knowledge proxy (e.g., perplexity or token log-likelihood) is reported; 'knows a language' is simply assumed from corpus size. The inverse-resource conclusion therefore rests on three byte-level low-resource languages, and the causal attribution to language knowledge rather than to token granularity is unsupported. The benchmark itself remains a useful resource, but the headline finding needs an explicit test against a knowledge measure and CWT controls before it can be treated as established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EXECUTE, a multilingual extension of the CUTE benchmark for testing LLM token understanding. It covers eight main languages spanning diverse scripts and writing systems, plus two additional low-resource languages (Tamazight, Santali) and cipher/byte-level English controls. The benchmark includes character- and word-level composition and manipulation tasks, with additional sub-character tasks for Chinese, Japanese, and Korean (radicals and Jamo). Experiments across 11 open-source LLMs yield four headline findings: non-English results often differ from English, performance correlates with character-word-token statistics, lower-resourced languages sometimes achieve higher scores, and LLMs struggle with sub-character components. The paper also provides an expandable framework and releases code.","tokens_in":15368,"tokens_out":4531,"duration_ms":36764,"significance":"If the central claims were fully supported, the paper would be a valuable resource for the field: it extends token-understanding evaluation to a much broader typological range than CUTE, adds sub-character probing for CJK languages, and offers an easily extensible framework with public code. The multi-model, multi-language result tables are a useful empirical contribution, and the authors explicitly release the benchmark, which should facilitate future work. The main scientific claims, however, are currently stronger than the evidence: the inverse-resource conclusion is confounded by tokenization granularity and contradicted by part of the reported data, and the claimed comparability to CUTE is not established by the appendix. With revision of these load-bearing claims, the benchmark itself remains a solid and reusable contribution.","major_comments":[{"comment":"The claim 'Surprisingly, the less an LLM knows a language, the better it performs on EXECUTE' is not supported by the full results. In Table 3, the best overall model (Qwen 2.5 32B) scores Hindi at 86.2, the highest of any language, while Amharic scores only 74.4; the near-perfect Amharic numbers in Table 2 come from Llama 3.3 only. The conclusion in §5 repeats the generalization, but the data show that the inverse-resource pattern is model-dependent and cannot be stated as a general property of LLMs. Please either restrict the claim to the models and languages where it holds, or provide a per-model language-knowledge proxy (e.g., perplexity or token log-likelihood) that is measured rather than assumed.","section":"Abstract, §4.1, Table 3"},{"comment":"The inverse-resource claim is confounded with tokenization granularity. Amharic, Tamazight, and Santali are simultaneously the lowest-resource languages and the most byte-level tokenized (c/t = 0.48, 0.49, 0.42 in Table 1), so their high scores are exactly what the paper's own CWT statistics predict. The cipher and byte-level English controls do not separate tokenization from language recognition: ciphered English changes both the tokenizer's behavior and recognizability, while byte-level English changes tokenization but retains recognition and remains poor on several word-level tasks. Moreover, Table 2 shows Cipher Inv Spell = 0.0, directly contradicting the text that 'Llama achieves near-perfect results in the low-resource languages, as well as the ciphered English.' This claim needs to be corrected, and the tokenization confound needs to be addressed explicitly, for example by holding tokenization granularity fixed across languages or by reporting results conditioned on c/t.","section":"§4.1, Table 1, Table 2"},{"comment":"The assertion that 'the scores from CUTE and EXECUTE are still largely comparable' is not supported by the comparison in Table 5. For Russian, Inverse Spelling drops from 74.1 in CUTE to 37.9 in EXECUTE (a 36.2-point gap), while Swap Word rises from 33.3 to 46.5; English Insert Word also differs by about 14 points. These are not minor differences, and they undermine the claim that the changed data source, vocabulary, examples, and prompt format left results 'largely comparable.' Since the paper explicitly cites users of the English and Russian datasets to CUTE, this comparability statement is load-bearing and should be replaced by a quantified account of which tasks shift and by how much.","section":"Appendix A, Table 5; §3.1"},{"comment":"Cross-language comparisons assume that the English prompt texts with language-specific examples are understood equally well in all eight languages. Section 3.1 states that this design 'ensures that the LLMs understand the task consistently across languages,' but no evidence is provided for equal comprehension. If, for example, the automatically generated examples are harder or more ambiguous in some languages, every cross-language comparison, including the inverse-resource claim, would be invalidated. A concrete control would be to run a subset of models on fully translated prompts for a few languages, or to measure per-language example difficulty (e.g., by scoring the model's ability to follow the few-shot format on trivial items). Without such a check, the cross-language conclusions should be framed more cautiously.","section":"§3.1, §4"},{"comment":"The claim that 'LLMs struggle with understanding sub-character components' is too broad. Table 4 shows that Char-to-Rad and Rad-to-Char are near zero for Chinese and Japanese, which does support a compositional-decomposition failure. However, the Contains Rad / Contains Jamo tasks are much higher (up to 96.6 for Korean), showing that models can often recognize a component when asked whether it appears in a character. The abstract and conclusion should be qualified to say that LLMs struggle with composing and decomposing characters into components, not with understanding sub-character components in general.","section":"Abstract, §4.4, Table 4"},{"comment":"The claim that 'performance is inversely correlated' with training-data size across model families is not a controlled comparison. The 7-9B models compared in that paragraph differ in tokenizer, architecture, training distribution, and training duration; training-token count alone is not a causal variable. This is a minor overinterpretation, but since it is used to bolster the inverse-resource narrative, it should either be removed or reframed as an uncontrolled observation.","section":"§4.3"}],"minor_comments":[{"comment":"The caption says 'Llama 3.3 on low-resource languages' but the table includes English, Cipher, Byte, and Reg columns; please clarify that the table reports multiple English-variant conditions and specify what 'Reg' denotes.","section":"Table 2"},{"comment":"The figure has no legend explaining the two bar colors; please add a legend or a caption note that distinguishes character-level and word-level tasks.","section":"Figure 2"},{"comment":"The caption lists 'Hindi, Korean, Japanese, and Russian' but the table columns are ordered Hindi, Japanese, Korean, Russian; the caption order should match the table order.","section":"Table 9"},{"comment":"The statement that 'Byte-level English improves character tasks but fails on word tasks' is too strong: Del Word is 96.5 and Cont Word 96.7 in the Byte column. Please specify which word tasks fail rather than saying word tasks generally fail.","section":"§4.1, Table 2"},{"comment":"The Japanese spelling example shows an answer identical to the input ('かわいい' → 'かわいい'), which may confuse readers because Japanese words are not written with spaces between letters; please add a note explaining how spelling is defined for logographic and syllabic scripts.","section":"Figure 4"},{"comment":"The comparison in Table 5 would be more informative with standard deviations or per-example counts, since several of the differences (e.g., Russian Inv Spell) are large enough to affect the paper's central conclusions.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and released code are useful, and the paper is likely publishable after the load-bearing claims are reworked. The most important changes are: (1) correct the false statement about ciphered English in Table 2, (2) remove or strongly qualify the inverse-resource claim given the Table 3 contradiction and the tokenization confound, (3) replace the 'largely comparable' CUTE claim with a quantified account of task-level differences, and (4) add a control for cross-language prompt comprehension. With those revisions, the paper would make a solid contribution; as is, the headline claims outrun the evidence."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading as a benchmark contribution, not as proof of the inverse-resource claim. What's new: an 8-language extension of CUTE with diverse scripts and writing systems, CJK sub-character tasks, and low-resource languages that are byte-level tokenized. The benchmark is simple to extend, and they release it. That's a real resource.\n\nThey also did genuine work: reported all results per model and task, added cipher and byte-level English controls, and analyzed CWT statistics. The sub-character results in Table 4 are a useful addition; the finding that models cannot reliably compose or decompose radicals and Jamo is credible and new for these languages.\n\nThe soft spot is the central claim. \"The less an LLM knows a language, the better it performs\" is not supported by the evidence. The three star performers (Amharic, Tamazight, Santali) are exactly the languages tokenized at byte level, with c/t around 0.4–0.5. Their near-perfect scores are what the paper's own Table 1 predicts from token granularity. The controls do not cleanly separate tokenization from recognition: byte-level English helps character tasks but fails on word tasks, and ciphered English changes the tokenizer too. More importantly, Table 3 contradicts the simple story: on Qwen 2.5 32B, Hindi is the highest-scoring language at 86.2, while Amharic scores 74.4, below English, Chinese, Japanese, and Korean. The inverse-resource trend is model-dependent and likely driven by a few byte-level languages on some models. The authors acknowledge that Amharic's near-perfect score may decline with more training data, which is honest, but it does not fix the abstract's overgeneralization.\n\nSecond, the CUTE comparability claim does not hold. Appendix A shows Russian Inverse Spelling at 37.9 in EXECUTE versus 74.1 in CUTE—a 36-point gap. With new data, new examples, and a different vocabulary, absolute scores are not \"largely comparable.\" They should say the benchmarks measure similar constructs but scores are not directly comparable.\n\nThird, they assume English prompts with language-specific examples are understood equally across languages, but they never measure per-language prompt comprehension. If models misunderstand instructions in some languages, cross-language comparisons are confounded. That is a validity gap for the comparative findings.\n\nThe sub-character ground truth is also a bit under-specified: Kangxi radical decompositions are not unique, and they do not state how they resolved conflicts. Minor, but worth clarifying.\n\nVerdict: solid benchmark, honest effort, but the headline finding needs rework. I would send it to peer review—it deserves referee time—but I would expect major revision on the inverse-resource claim and the CUTE comparison. The benchmark itself and the sub-character data are the durable parts.","headline":"A useful multilingual benchmark with honest measurements, but the headline inverse-resource finding is confounded with token granularity and contradicted by the paper's own Table 3.","tokens_in":15873,"tokens_out":2423,"would_cite":true,"duration_ms":19628,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The less an LLM knows a language, the better it performs on token-manipulation tasks, and it cannot reliably decompose Chinese, Japanese, or Korean characters into their components.","keywords":["LLM token understanding","multilingual benchmark","character manipulation","sub-character components","CWT statistics","low-resource languages","Kangxi radicals","Hangul Jamo"],"falsifier":"Translate the full prompts (not just the examples) into each of the eight target languages and re-run EXECUTE. If the inverse-resource pattern disappears—if English and Russian no longer score below Amharic, Tamazight, and Santali—then the central claim is an artifact of uneven comprehension of English prompts rather than of token-manipulation skill. For the sub-character claim, a complementary check is to test characters with unambiguous radical decompositions; near-zero scores on that subset would confirm the absence of compositional understanding, while strong scores would point to annotation ambiguity rather than missing knowledge.","tokens_in":14878,"feed_emoji":"🔤","tokens_out":24669,"duration_ms":183769,"temperature":0.7,"pith_summary":"EXECUTE is a multilingual extension of the CUTE benchmark, built to test whether large language models can manipulate the characters and words of eight very different writing systems. The paper reports two patterns: difficulties are not always at the character level—Arabic and Russian resemble English, Hindi and Korean show a weaker version of the English pattern, Chinese and Japanese are easier at the character level than at the word level, and Amharic is nearly perfect—and performance is inversely related to how well the model knows the language. The offered explanation is that strong language knowledge biases a model toward producing real, grammatical words, so it resists the literal string manipulations the tasks demand, while weak knowledge of a low-resource language removes that pressure. The paper also introduces sub-character tasks and finds that LLMs cannot reliably decompose Chinese and Japanese characters into Kangxi radicals (the conventional component parts of Chinese characters) or Hangul syllables into their component letters (Jamo). If these claims hold, token-manipulation skill must be measured separately from language proficiency, and models lack compositional knowledge of script components.","feed_headline":"Knowing less about a language boosts LLM token-manipulation scores","feed_subtitle":"Weaker language knowledge means higher accuracy on character and word tasks across eight scripts.","key_machinery":"The load-bearing object is the EXECUTE benchmark itself: twelve task types (spelling, inverse spelling, contains, insertion, deletion, substitution, and swapping, each at word and character level) applied to translated TinyStories sentences in eight languages, with an expandable pipeline that requires only translation to add a language. The paper's explanation rests on two mechanisms. The first is the language's character-word-token (CWT) statistics—average characters per word, tokens per word, and characters per token—which predict the clustering of languages by task profile. The second is the model's real-word bias: when a model knows a language well, it favors existing words and grammatical forms, making it reluctant to output the ungrammatical manipulated strings the tasks require; low-resource languages, especially byte-level tokenized ones, escape this pressure. The sub-character tasks (composition and decomposition of Kangxi radicals, the conventional component parts of Chinese characters, and of Hangul Jamo, the component letters of Korean syllables) expose the absence of compositional script knowledge.","core_discovery":"The paper's central claim is that an LLM's ability to manipulate the characters and words of a language is governed less by how well it knows the language than by the language's writing-system geometry and the model's prior bias toward real words. Evidence comes from eight languages with very different scripts and writing systems: Russian and Arabic scores track English; Hindi and Korean follow the English direction with a smaller gap between character-level and word-level scores; Chinese and Japanese are easier at the character level because a character is usually a word; and Amharic, the lowest-resource language, approaches perfect scores on the best model. The paper interprets the Amharic result as a release from linguistic bias: when the model barely knows the language, it no longer prefers grammatical real words and can emit the literal manipulated string. The sub-character experiments extend the claim one level down: converting characters to and from their Kangxi radicals (the conventional component parts of Chinese characters) is near zero for most models, converting Hangul to and from its Jamo letters is better but still weak, and only the simpler 'does this character contain this component' task reaches moderate accuracy.","pith_inferences":["Beyond the paper: if the real-word-bias mechanism is correct, then a decoding-time intervention that penalizes completions that are real words, or a prompt that asks the model to 'forget' the language, should raise EXECUTE scores for English and Russian; this is testable with the released tasks and is not claimed by the paper.","Beyond the paper: the CWT correlation is computed at the language level, but the same logic applies per tokenizer; recomputing tokens-per-word for each model's vocabulary and correlating within a single language would separate tokenization geometry from the model's linguistic priors.","Beyond the paper: the Amharic result suggests that byte-level tokenization is not inherently harmful for exact string manipulation; if so, the binding constraint in high-resource languages is the model's own language prior, and changing tokenizer granularity alone will not fix CUTE-style failures.","Beyond the paper: extending the sub-character tasks to other scripts with explicit components, such as Cherokee syllables or Ethiopic fidels, would show whether the missing compositional knowledge is a general property of LLMs or specific to Chinese, Japanese, and Korean."],"forward_implications":["If the inverse-resource result holds, EXECUTE scores cannot be interpreted as a measure of language proficiency: a model that knows a language better can appear worse at manipulating it.","If the CWT account is right, a newly added language's task profile can be predicted from its characters-per-word, tokens-per-word, and characters-per-token before any model is run.","A model that is told to temporarily 'forget' the target language, or that is debiased against producing grammatical output, should improve on EXECUTE in high-resource languages; the paper states this debiasing need explicitly.","Because the benchmark is expandable by translation alone, the same protocol can produce comparable token-understanding scores for any language, including ones not in the current eight."],"supporting_citations":[{"why":"Defines the CUTE benchmark whose composition and manipulation tasks EXECUTE extends, and reports the real-word bias the paper builds on.","marker":"Edman et al., 2024"},{"why":"Supplies the TinyStories stories that are translated into the benchmark's languages.","marker":"Eldan and Li, 2023"},{"why":"Provides the radical-decomposition analysis that the Chinese and Japanese character-to-radical tasks are modeled on.","marker":"Wu et al., 2025"},{"why":"Documents the label-frequency bias in few-shot prompts that motivates the balanced 'yes'/'no' examples in the contains tasks.","marker":"Zhao et al., 2021"}],"fun_headline_variants":["LLMs ace token games in scripts they don't know well","Why LLMs are better at token tricks in unfamiliar scripts","Low-resource Amharic stumps LLM language bias","Language ignorance boosts LLM character accuracy","Script geometry beats language knowledge for LLM token tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's cross-language comparisons stand or fall on the assumption that an English prompt with language-specific examples is understood consistently across all eight languages, so score differences reflect how models process characters and words rather than uneven comprehension of the task.","fun_headline_variants_meta":{"raw":{"variants":["LLMs ace token games in scripts they don't know well","Why LLMs are better at token tricks in unfamiliar scripts","Low-resource Amharic stumps LLM language bias","Language ignorance boosts LLM character accuracy","Script geometry beats language knowledge for LLM token tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1503,"prompt_tokens":840,"completion_tokens":663,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":585}},"tokens_in":456,"tokens_out":663,"duration_ms":5363,"temperature":1.0,"reasoning_tokens":585,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:40:12.442074+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Translate the full prompts (not just the examples) into each of the eight target languages and re-run EXECUTE. If the inverse-resource pattern disappears—if English and Russian no longer score below Amharic, Tamazight, and Santali—then the central claim is an artifact of uneven comprehension of English prompts rather than of token-manipulation skill. For the sub-character claim, a complementary check is to test characters with unambiguous radical decompositions; near-zero scores on that subset would confirm the absence of compositional understanding, while strong scores would point to annotation ambiguity rather than missing knowledge.","supporting_citations":[],"review_version":1}