{"id":"6e5d5212-8c9a-438b-b463-20b67cb4f531","arxiv_id":"2508.11260","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"LLMs solve linguistics puzzles worse when the puzzle language is morphologically complex or distant from English, and pre-splitting words into morphemes improves their scores.","lead":"This paper tests large language models on 629 Linguistics Olympiad puzzles in 41 low-resource languages and finds that models solve fewer puzzles as morphological complexity rises, and more puzzles when the target features resemble English. The likely takeaway for builders is that morpheme-level tokenization, splitting words into meaningful pieces before the model reads them, improves solve rates, which points to a concrete fix for low-resource language systems.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Feature labels (morphological complexity, English overlap) are observational and may be confounded by puzzle length, script, or answer format; the paper's causal reading needs controls.","rationale":"The reader's weakest assumption identifies both contamination and feature-label confounds. I agree, and I particularly emphasize the feature-label attribution because the abstract makes causal claims about morphological complexity and English overlap on the basis of observational data. The concrete paired-analysis test would directly address this. Since only the abstract was available, I cannot verify whether the full paper already includes such controls; therefore I do not move the verdict away from UNVERDICTED. An honest non-finding is not appropriate here because the abstract alone genuinely leaves the attribution question open, and the test is essential to settling it.","tokens_in":759,"tokens_out":1629,"duration_ms":19175,"concrete_test":"Run a matched-pair analysis on the 629 puzzles: group puzzles by script, word category, answer format, and length bands, then compare accuracy within pairs/groups that differ only in morphological complexity or English overlap. If the effect size drops to zero within matches, the headline claims are confounded. For contamination, perform a membership test: hold out a random subset of puzzles, prompt the models with the puzzle statements, and check for verbatim or near-verbatim recall of answers; high recall would falsify the 'minimal contamination' premise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claims are built on correlating LLM accuracy with author-assigned puzzle features across 629 problems. Puzzles are not randomized, so 'higher morphological complexity' and 'English overlap' may co-vary with unmeasured puzzle attributes such as answer length, number of examples, script unfamiliarity, or answer format. If those confounds are not controlled, the abstract's causal phrasing ('LLMs struggle with...', 'perform better on...') is not warranted. The morpheme-splitting result is stronger because it is an intervention, but it still depends on the tokenizer baseline: if the baseline and the morpheme-split tokenizer differ in vocabulary size, normalization, or language-specific preprocessing, the accuracy gain may reflect those differences rather than morpheme awareness. Additionally, 'minimal contamination environment' is asserted but not demonstrated; if puzzle texts or answers appear in training corpora, memorization could produce the same patterns without any linguistic reasoning. Without access to the full methods—feature definitions, regression or matching design, tokenizer implementation, and contamination checks—these attribution threats remain unresolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports an analysis of LLM performance on 629 Linguistics Olympiad puzzles from 41 low-resource languages. The abstract claims three findings: (1) LLMs do worse on puzzles with higher morphological complexity, (2) LLMs do better on puzzles whose linguistic features overlap with English, and (3) pre-splitting words into morphemes improves solvability. The authors interpret these results as evidence that tokenization and language-specific representation, rather than reasoning alone, bottleneck LLM performance on linguistic reasoning in low-resource languages. The full text was not available for this review; only the abstract and the reader's summary were examined.","tokens_in":980,"tokens_out":1635,"duration_ms":20336,"significance":"If the claims are supported by rigorous controls, this would be a useful contribution to the study of LLM linguistic reasoning. A 629-problem, 41-language stimulus set with linguistically informed feature labels could provide a valuable benchmark, and the morpheme-splitting intervention is a concrete, actionable direction for tokenizer design. The main significance depends on whether the observational feature correlations survive confound control and whether the contamination claim can be substantiated.","major_comments":[{"comment":"The phrase 'minimal contamination environment' is load-bearing: the entire empirical strategy assumes the 629 puzzles and their answers are absent from LLM training corpora. No evidence for this is given in the abstract, and the full text is unavailable. Without contamination checks (e.g., probing or membership tests), all reported performance differences could be memorization artifacts rather than evidence about linguistic reasoning.","section":"Abstract, paragraph 1"},{"comment":"The claims that 'LLMs struggle with puzzles involving higher morphological complexity' and 'perform better on puzzles involving linguistic features that are also found in English' are framed causally, but the features appear to be correlational properties of non-randomized puzzles. Morphological complexity and English overlap may co-vary with puzzle length, script unfamiliarity, answer format, number of examples, or other difficulty factors. The abstract reports no regression, matching, or other controls, so the causal reading is not currently warranted.","section":"Abstract, paragraph 2"},{"comment":"The morpheme-splitting result is an intervention and therefore stronger, but its validity depends on an apples-to-apples comparison with the raw-text tokenizer. If the baseline and morpheme-split tokenizers differ in vocabulary size, normalization, handling of unknown tokens, or language-specific preprocessing, the accuracy gain may reflect these implementation differences rather than morpheme awareness. The abstract provides no tokenizer details; the full methods must specify this comparison to support the stated conclusion.","section":"Abstract, paragraph 2"},{"comment":"This review is based only on the abstract because the full text was not available. As a consequence, the dataset construction, feature definitions, inter-annotator reliability (if any), statistical model, baseline models, and significance testing cannot be checked. These are central to the paper's claims; without them, the reported findings are unverifiable.","section":"Abstract (overall)"}],"minor_comments":[{"comment":"The term 'minimal contamination environment' should be defined operationally; the reader cannot tell whether it means low risk, verified absence, or only an assumption. A sentence on how contamination was assessed would help.","section":"Abstract, paragraph 1"},{"comment":"The abstract does not state which LLMs were evaluated, how many runs were averaged, or whether the reported differences are statistically significant. Adding these numbers would make the findings more interpretable.","section":"Abstract, paragraph 2"},{"comment":"'UNVEILING' is in all caps, which is a style choice rather than a substantive issue; if the journal prefers sentence case, this could be adjusted.","section":"Title"}],"recommendation":"uncertain","confidential_remarks":"This review was written without access to the full text. The abstract reports interesting results, but the central empirical claims—contamination-freeness, confound control for observational features, and tokenizer comparability—cannot be assessed from the abstract alone. Before making a decision, the editor should obtain the full manuscript and send it to a referee. The recommendation of 'uncertain' reflects the missing information, not a judgment about the eventual merits of the work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read the abstract and the reader's take. My own verdict is also unverified, but for a different reason: I'm willing to say the paper deserves peer review even though I can't check the methods.\n\nWhat is actually new: a 629-problem, 41-language benchmark with linguistically informed feature labels, plus a preprocessing intervention (morpheme splitting). That is more than a pure benchmark sweep. The morpheme-splitting result, if it holds up, is a concrete, actionable lead for tokenizer design in low-resource languages. The finding that English overlap helps and morphological complexity hurts is consistent with transfer-learning intuition, but that doesn't make it trivial; it's useful to have it documented on a controlled puzzle set.\n\nWhere I'm cautious: the abstract alone cannot support the causal phrasing. 'LLMs struggle with higher morphological complexity' and 'perform better with English overlap' are correlations over observational puzzle features. Puzzles vary in length, answer format, script unfamiliarity, and number of examples, and none of those are visible in the abstract. The author-authored labels could be entangled with those confounds. The stress-test note is fair on that point. The morpheme-splitting result is stronger because it's an intervention, but I'd need to know the tokenizer baselines were matched on vocabulary size and normalization before I'd trust the gain.\n\nThe 'minimal contamination environment' claim is asserted, not demonstrated. If any of those problems leaked into training corpora, memorization could produce the same pattern. That's a real threat, and the authors need to show contamination checks.\n\nI disagree with the reader's weakest-assumption phrasing in one way: the feature labels are not necessarily 'author-authored' in a suspicious sense—they're linguistically motivated annotations. The circularity burden is low. The concern is just whether the taxonomy was designed after seeing performance differences. That's a legitimate worry but not a disqualifier.\n\nWho is this for? Anyone working on multilingual LLM evaluation or tokenization. If the full paper has clean feature definitions, controls for puzzle difficulty, and tokenizer ablation details, it will be a solid contribution. If not, it's a nice dataset with overstated claims.\n\nRecommendation: send it to peer review. The dataset alone is useful, and the morpheme-splitting intervention is testable. A good referee can force the authors to tighten the causal language and add controls.","headline":"Abstract-only, so I can't sign off on the claims, but the dataset and the tokenizer intervention are worth a referee's time.","tokens_in":1512,"tokens_out":853,"would_cite":false,"duration_ms":11488,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that LLM performance on Linguistics Olympiad puzzles is systematically shaped by morphological complexity and overlap with English, and that morpheme-splitting preprocessing improves solvability.","keywords":["linguistics olympiad","low-resource languages","morphological complexity","tokenization","linguistic reasoning","large language models","minimal contamination benchmark","morpheme splitting"],"falsifier":"Search for near-verbatim versions of the 629 puzzle statements or their official answers in public LLM training corpora and correlate per-puzzle retrieval overlap with model accuracy; if accuracy rises with overlap, the 'minimal contamination' premise fails and morphological/English-overlap effects could be memorization artifacts. A complementary check would be to rerun the analysis on matched puzzles that vary morphology but hold length, answer format, and exemplar count constant.","tokens_in":681,"feed_emoji":"🧩","tokens_out":4209,"duration_ms":46546,"temperature":0.7,"pith_summary":"The paper tries to establish that LLM failures on Linguistics Olympiad puzzles are not random: performance drops when a puzzle's language is morphologically more complex and rises when the puzzle's linguistic features resemble English. It analyses 629 puzzles from 41 low-resource languages, each labelled with linguistically informed features, and shows that pre-splitting words into morphemes improves solvability. A sympathetic reader should care because these puzzles are treated as a minimal contamination environment, so the measured differences look like genuine linguistic reasoning, not memorized answers. If the claim is right, a large part of LLM difficulty with low-resource language reasoning sits in tokenization and representation rather than in the reasoning itself.","feed_headline":"LLMs fail high-morphology language puzzles; splitting words helps","feed_subtitle":"Analysis of 629 Linguistics Olympiad puzzles shows English-like features boost LLM scores and morphology drags them down.","key_machinery":"The central object is a labelled puzzle corpus: 629 Linguistics Olympiad problems across 41 low-resource languages, annotated with linguistically informed features such as morphological complexity and the degree of overlap with English features. The argument runs through two linked mechanisms: feature labels let the analysis attribute performance differences to specific linguistic structures, and the morpheme-splitting preprocessing intervention changes only the tokenization of input text, isolating representation from reasoning.","core_discovery":"Using 629 Linguistics Olympiad problems drawn from 41 low-resource languages, the paper claims that model accuracy is systematically lower on puzzles with high morphological complexity and systematically higher on puzzles whose target features also appear in English. The central discovery is the morpheme-splitting result: when puzzle words are split into morphemes before being fed to the model, solvability improves, suggesting that the models fail partly because standard tokenizers are poorly matched to morphologically rich languages. The authors frame the bridge as a call for informed, language-specific tokenizers, with the Olympiad corpus serving as a nearly contamination-free arena for st","pith_inferences":["If the minimal-contamination premise holds, a direct testable extension is to vary tokenizer granularity (morpheme, syllable, character) on the same 629 puzzles and check whether accuracy tracks granularity, which would confirm representation as the bottleneck.","Because the feature labels are author-assigned, an editorially added caution is that matched control puzzles (same length, answer format, number of examples) are needed to rule out confounds before attributing effects purely to morphology or English overlap.","An implication the authors leave implicit is that standard subword tokenizers may be a hidden confound in any low-resource-language benchmark, not just puzzle-solving.","The morpheme-splitting gain, if real, plausibly extends beyond Olympiad puzzles to morphology-heavy applications such as translation and information extraction in agglutinative languages, though the paper itself does not claim this."],"forward_implications":["High-morphology languages should be expected to yield systematically lower LLM scores on Olympiad-style reasoning, not noise.","English-overlap features inflate apparent ability, so cross-lingual reasoning claims need to control for shared features.","Morpheme-splitting preprocessing improves solvability, making tokenizer design a direct lever on reasoning performance.","The 629-puzzle corpus can serve as a contamination-controlled benchmark for studying linguistic reasoning in low-resource languages.","Language-specific, morphology-aware tokenizers are a concrete path toward fairer LLM performance on low-resource languages."],"supporting_citations":[],"fun_headline_variants":["Splitting words into morphemes boosts LLM puzzle scores","Language puzzles: LLMs stumped by morphology, saved by splitting","English-like features help LLMs on linguistics puzzles, morphology hurts","Morpheme splitting improves LLMs on 629 language puzzles","LLMs do better on puzzles with English-like features; splitting helps"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the 629 Olympiad puzzles and their answers are absent from LLM training corpora, so the performance gaps reflect reasoning about language structure rather than memorization.","fun_headline_variants_meta":{"raw":{"variants":["Splitting words into morphemes boosts LLM puzzle scores","Language puzzles: LLMs stumped by morphology, saved by splitting","English-like features help LLMs on linguistics puzzles, morphology hurts","Morpheme splitting improves LLMs on 629 language puzzles","LLMs do better on puzzles with English-like features; splitting helps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000785,"raw_usage":{"total_tokens":3257,"prompt_tokens":659,"completion_tokens":2598,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":403,"completion_tokens_details":{"reasoning_tokens":2517}},"tokens_in":403,"tokens_out":2598,"duration_ms":19568,"temperature":1.0,"reasoning_tokens":2517,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T20:01:10.676605+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Search for near-verbatim versions of the 629 puzzle statements or their official answers in public LLM training corpora and correlate per-puzzle retrieval overlap with model accuracy; if accuracy rises with overlap, the 'minimal contamination' premise fails and morphological/English-overlap effects could be memorization artifacts. A complementary check would be to rerun the analysis on matched puzzles that vary morphology but hold length, answer format, and exemplar count constant.","supporting_citations":[],"review_version":1}