{"id":"93574622-941f-4abb-a328-fada06a81184","arxiv_id":"2506.14012","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Code-switching hurts LLM comprehension when non-English tokens enter English text, but inserting English into other languages often improves accuracy; fine-tuning mitigates losses more reliably than prompting.","lead":"This paper tests how well eight large language models understand text that mixes English with Arabic, German, French, or Chinese. It finds models lose accuracy when foreign words appear in English sentences, but often gain accuracy when English words appear in other languages.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The asymmetric claim depends on unverified generation fidelity: if the LLM-Centric pipeline selectively switches only high-frequency or context-free nouns (or the GPT-4o preference reflects fluency artifacts), the measured deltas are artifacts. A targeted re-annotation test is needed.","rationale":"The reader identified the same weakest assumption: generated CSW fidelity is unverified. I agree with the CONDITIONAL verdict. The paper's main monotonic degradation result (EN-matrix) is robust across models and methods, so a REJECT is too strong. However, the headline asymmetry—especially the facilitative EL-matrix effect—is exactly where unvalidated generation quality bites hardest. If Claude's noun selection is biased toward simple, recoverable insertions, the EN→AR degradation could be artificially small (because only easy nouns are switched) or artificially large (because inserted nouns are out-of-vocabulary for the model); neither case supports the structural-bias interpretation. Similarly, the AR→EN facilitation (Mistral-7B 0.35→0.48, Δ=+0.13) could be explained by inserting high-frequency English nouns that act as retrieval cues, rather than by a general English-centric processing bias. The Appendix B numbers show the selection process itself was not strongly discriminative: human preference for Claude over GPT-4o was only 52–62%, and the method choice was decided by GPT-4o without human adjudication. This is the single load-bearing gap. My proposed test is concrete: it checks whether the generated switch distribution matches natural CSW and whether the headline deltas survive human-validated instances. If the test passes, the paper's claims are much stronger; if it fails, the central asymmetry is an artifact. Thus the verdict remains CONDITIONAL, pending this verification.","tokens_in":16450,"tokens_out":1344,"duration_ms":14299,"concrete_test":"Sample 100 generated noun-token instances per matrix-embedded pair (e.g., EN→AR and AR→EN) and have bilingual annotators independently (1) judge naturalness/acceptability, (2) tag each switched noun for frequency, concreteness, and syntactic function, and (3) compare the distribution of switch positions against a reference corpus of natural CSW (e.g., Egyptian or Moroccan Arabic CSW data). If the generated switches are biased toward high-frequency, isolated nouns (or if naturalness judgments fall below, say, 70% acceptability), recompute the headline deltas using only human-validated instances; if the asymmetry vanishes, the central claim fails.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central asymmetry—degradation for EN-matrix CSW versus facilitation for EL-matrix CSW—rests on the assumption that the generated CSW stimuli faithfully represent natural code-switching under ECT/MLF. The manuscript's own Appendix B reveals that (i) only 100 examples per language pair were human-checked for LLM selection (Claude vs. GPT-4o), with preferences as low as 52%; (ii) the decisive method comparison (LLM-Centric vs. Alignment-Based) was judged solely by GPT-4o, with no human verification; and (iii) the LLM-Centric prompt instructs Claude to identify nouns that 'would serve as appropriate code-switching points' and 'ensure that nouns in common expressions are not code-switched.' If Claude systematically selects short, high-frequency, or semantically transparent nouns—or avoids nouns whose translation would require inflectional changes—the generated stimuli are systematically easier than natural CSW. This would inflate the EN-matrix degradation (because the inserted tokens are still unfamiliar) and could trivially produce the EL-matrix facilitation effect if the inserted English words are the most predictable or recoverable content words. The paper even acknowledges in its Limitations that only 'one of the fundamental forms' of CSW is tested. The lack of per-instance human verification, combined with small preference margins, means the measured deltas (e.g., Mistral-7B Arabic 0.35→0.48) may reflect generation-selection artifacts rather than genuine comprehension structure. This is internally consistent but externally unvalidated; the central claim needs direct evidence that the generated CSW matches naturalistic switch distributions.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces a pipeline for generating code-switched (CSW) versions of three established benchmarks (Belebele, MMLU, XNLI) using two strategies: a linguistically motivated noun-token method grounded in the Equivalence Constraint Theory (ECT) and Matrix Language Frame (MLF) model, and a heuristic ratio-token method that substitutes roughly 20% of aligned tokens at random. The authors evaluate eight LLMs across five languages and report two main findings: (i) embedding non-English tokens into English text degrades performance, and (ii) embedding English tokens into non-English text frequently improves comprehension, especially for models with weaker proficiency in the matrix language. They also test prompt-based and fine-tuning-based mitigation strategies, concluding that fine-tuning yields more stable recovery. The central claim is an asymmetric, English-centric structural bias in LLM processing of code-switched input.","tokens_in":16744,"tokens_out":5978,"duration_ms":65595,"significance":"If the asymmetric effect is genuine and attributable to properties of code-switching rather than to artifacts of the generation procedure, the paper would make a valuable contribution to multilingual NLP evaluation and to the study of English-centric biases in LLMs. The authors release code and data, evaluate a diverse set of models and languages, and ground their generation in established linguistic theories, which are all concrete strengths. However, the headline claims currently rest on small test sets without statistical support and on a generation pipeline whose fidelity is only weakly validated, so the significance of the finding is conditional on addressing these issues.","major_comments":[{"comment":"The test set comprises only 300 samples per embedded language, and the reported deltas (e.g., LLaMA-70B on EN→AR: 0.70→0.66, Δ≈−0.04) correspond to roughly 12 examples on 300 instances. No confidence intervals, bootstrap estimates, or significance tests are reported anywhere in the paper. Given the number of models (8) × languages (4) × tasks (3) comparisons, many of the headline differences are within the noise band of a 300-sample evaluation. The authors should report uncertainty intervals and, if appropriate, apply multiple-testing corrections, or otherwise justify that the deltas are not chance-level fluctuations.","section":"Section 3.4 and Tables 1–3"},{"comment":"The noun-token and ratio-token methods are compared to isolate the effect of linguistic constraints, but the two conditions differ in more than constraints: noun-token replaces all identified nouns, while ratio-token replaces ~20% of aligned tokens at random, which may include function words and content words of varying categories, at a different substitution rate. Therefore, the Discussion's inference (Section 8) that comparable degradation under both methods reveals a 'structural vulnerability' beyond token-level unfamiliarity is confounded. A matched comparison (e.g., same proportion of random nouns, or same token category with and without ECT/MLF constraint) is necessary before this claim can be drawn.","section":"Section 3.2 and Tables 1–2"},{"comment":"The validity of the central asymmetric result—particularly the facilitation observed when English is embedded into non-English matrix languages (Table 3)—depends on the generated CSW faithfully representing natural code-switching. However, the generation-selection evidence is thin: only 100 examples per language were human-checked for the choice of generation model (preferences as low as 52% for Arabic in Table 6), and the decisive comparison between the LLM-Centric and Alignment-Based pipelines was judged solely by GPT-4o (Table 8), with no human verification and preferences as low as 53.4% for German. Furthermore, the LLM-Centric prompt (Figures 4–5) explicitly instructs the model to avoid switching nouns in common expressions and to adjust inflections, potentially biasing the inserted nouns toward short, high-frequency, semantically transparent items. If so, the observed EL-matrix improvement (e.g., Mistral-7B on Arabic: 0.35→0.48) could be an artifact of the predictability or recoverability of the inserted English words rather than a genuine CSW comprehension effect. The authors should provide per-instance human evaluation of the final benchmark instances (naturalness, switch-point grammaticality, and the representativeness of the chosen nouns) and report inter-annotator agreement.","section":"Appendix B.1/B.4, Tables 6/8, and Section 3.3"},{"comment":"When English is the embedded language, the inserted English words are drawn from the parallel English source sentence (the original benchmark text), while the baseline is the non-English translation. For models that are English-centric and have limited proficiency in the matrix language, replacing selected nouns with the original English nouns may simply make the instance easier because those exact English words are from the source-side text and are highly accessible in the model's pretraining distribution. This confound could trivially produce the facilitation effect reported in Table 3. The authors should control for this by, for example, inserting English nouns that are not taken from the parallel source sentence, or by comparing against insertion of matrix-language nouns matched for frequency and semantic content; otherwise the 'English embedding improves comprehension' claim is not separable from a source-leakage artifact.","section":"Section 3.3 and Section 6.1 (Table 3)"}],"minor_comments":[{"comment":"Several references are malformed: 'Li Nguyen and. 2018' is missing the author's given name, 'Pranjal Khanuja et al. 2020' should be 'Khanuja et al.', and 'Genta Winata et al. 2021a' is inconsistently formatted; the reference list should be checked for completeness and consistency.","section":"References"},{"comment":"The caption of Figure 7 says the prompt was given to Claude 3.5 Sonnet for choosing the best summary, but the text in Appendix B.2/B.4 states that GPT-4o was used as the judge; the caption also refers to 'summaries' rather than code-switched sentences. Please correct the model attribution and the description.","section":"Figure 7"},{"comment":"In Table 5, the entry for Llama 3B on EN→AR contains an extraneous ':a' after the accuracy value; please fix the typo.","section":"Table 5"},{"comment":"The abstract states that 'fine-tuning offers a more stable path', but the fine-tuning experiment is performed on a single model (Llama 8B) with gains of +0.03–0.04; the claim should be tempered to reflect the limited evidence, or additional models should be fine-tuned.","section":"Abstract and Section 7.2"},{"comment":"The notation 'B = {Bp}P p=1' is typeset incorrectly; it should read B = {B_p}_{p=1}^P. Please also define the indexing and the product set explicitly.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the released code and data are valuable assets. However, the central asymmetry claim is currently under-supported by three load-bearing issues: (1) the absence of any statistical uncertainty quantification on 300-sample test sets; (2) the confound between token category/rate and linguistic constraint in the noun-token vs. ratio-token comparison; and (3) the reliance on a generation pipeline whose fidelity is validated only weakly and whose output the paper itself does not verify per instance. These issues are fixable with additional human evaluation, matched control conditions, and proper error reporting, which is why I recommend major revision rather than rejection. I would also recommend that the editor ask the authors to clarify the provenance of the non-English MMLU/XNLI baselines, as this bears directly on the English-embedding facilitation result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a well-executed empirical study with one genuinely new result, but the result that matters most—English-embedded CSW improving comprehension—is the least supported one. I'd send it to review, but referees should push hard on statistics and generation fidelity.\n\nWhat's actually new: nobody has evaluated decoder-only LLMs on reasoning and comprehension benchmarks code-switched under ECT/MLF constraints at this scale. The directional asymmetry (EN-matrix degrades, EN-embedded helps) is absent from the cited literature. The paper does several things right: eight models, four embedded languages, three benchmarks, two generation pipelines, and public code and data. The EN-matrix degradation is consistent and believable—every model drops on every language pair, and the drop is smaller under linguistically constrained switching, though that comparison is confounded.\n\nThe soft spots are real but proportionate. The headline asymmetry—especially Mistral 7B on Arabic going 0.35→0.48—could easily be a generation artifact. The LLM-centric prompt tells Claude to switch 'appropriate' nouns and avoid common expressions; if that selects high-frequency, easily recoverable nouns, the facilitation is trivial. Only 100 examples per language pair were human-checked for model selection, with preference rates as low as 52%, and the decisive method comparison was judged by GPT-4o alone, with no human verification. The paper admits in its Limitations that only one fundamental form of CSW is tested. There are also no confidence intervals or significance tests anywhere; on a 300-sample test set, deltas of 0.01–0.04 are within plausible noise. The noun-token vs ratio-token comparison confounds switch rate with linguistic constraint, so the claim that linguistic constraints don't help is not actually supported.\n\nThat said, the stress-test concern is not fatal. The EN-matrix degradation is robust across all models and languages; it's unlikely to be pure artifact. The facilitation finding needs a targeted re-annotation test: human judges should rate the naturalness of generated CSW and check whether switched nouns are systematically shorter or higher-frequency than natural CSW.\n\nWho's this for? Anyone working on multilingual evaluation or English-centric bias in LLMs. The dataset alone is a contribution. It deserves a serious referee, but I'd require the authors to add significance testing, release generation samples for inspection, and provide human validation of the LLM-centric pipeline before the facilitation claim is convincing. Worth bringing to a reading group.","headline":"Solid and genuinely new empirical study, but the headline asymmetry rests on thin statistics and unverified generation fidelity; deserves review with pressure on the facilitation claim.","tokens_in":17278,"tokens_out":3564,"would_cite":true,"duration_ms":34755,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Foreign words in English confuse LLMs; English words abroad help","keywords":["code-switching","large language models","multilingual comprehension","benchmark generation","Equivalence Constraint Theory","Matrix Language Frame","LLM evaluation","natural language inference"],"falsifier":"If bilingual human annotators rated a sample of the generated code-switched items as unnatural or as switching only easy, high-frequency nouns, and if the accuracy deltas on those items disappeared or reversed, the central asymmetry would be an artifact of the generation pipeline rather than a property of LLM comprehension.","tokens_in":16251,"feed_emoji":"🔀","tokens_out":5498,"duration_ms":48034,"temperature":0.7,"pith_summary":"This paper asks whether large language models truly understand code-switched text—sentences that alternate between two languages—rather than merely recognizing it. To find out, the authors generate code-switched versions of three established reasoning benchmarks (Belebele, MMLU, XNLI) using switches constrained by two linguistic theories of where code-switching is grammatical. They find an asymmetry: replacing nouns in English sentences with Arabic, German, French, or Chinese words consistently lowers model accuracy, even when the switches follow linguistic constraints, while replacing nouns in those languages with English words often raises accuracy, especially when the model is weak in the matrix language. Prompting to expect code-switching helps some models and hurts others, whereas fine-tuning on code-switched data yields steadier, though partial, recovery. The paper's central claim, if correct, is that LLMs harbor an English-centric structural bias in mixed-language processing, not just a token-level vocabulary gap.","feed_headline":"Foreign words in English confuse LLMs; English words abroad help","feed_subtitle":"An English-centric bias: foreign tokens degrade English reasoning, English tokens boost other languages.","key_machinery":"The load-bearing object is the code-switched benchmark generation pipeline itself: a two-step LLM-centric approach in which Claude 3.5 Sonnet first marks candidate switch points (nouns for the linguistically grounded variant, random tokens for the heuristic variant) and then fills the placeholders with words from a parallel sentence under instructions to respect the Equivalence Constraint Theory and the Matrix Language Frame model. An alignment-based pipeline using the AWESOME aligner with LaBSE embeddings and Stanza POS tagging serves as the controlled comparison, and GPT-4o acts as judge to select the more natural outputs. The evaluation machinery is the accuracy delta, the difference between accuracy on a code-switched benchmark and its monolingual baseline, which makes the asymmetry visible: negative deltas when foreign tokens enter English, positive deltas when English enters other languages.","core_discovery":"Using a two-step LLM pipeline built on the Equivalence Constraint Theory and the Matrix Language Frame model, the paper replaces nouns in English benchmark items with translations in Arabic, German, French, and Chinese (noun-token CSW), and in a second experiment replaces roughly 20% of random tokens (ratio-token CSW). On weighted average accuracy across Belebele, MMLU, and XNLI, LLaMA-70B drops from 0.70 in English to 0.66 on EN→AR and EN→DE; Qwen-7B falls from 0.61 to 0.52–0.57. Reversing the roles, Mistral-7B on Arabic rises from 0.35 to 0.48 when English nouns are embedded, and gains appear for other low-proficiency matrix languages as well. Extreme mixing with four embedded languages does not always cause the largest drops, and script type (Latin vs. non-Latin) has no uniform effect. The results are taken as evidence that the degradation is structural, not merely lexical unfamiliarity.","pith_inferences":["The paper's noun-token design means the measured asymmetry concerns noun substitutions only; a natural extension would switch verbs, adjectives, or function words, which the Equivalence Constraint Theory treats differently, and which may produce larger or opposite effects.","An implicit testable claim is that the English-as-embedded-language boost should shrink as models' non-English proficiency improves; one could verify this by correlating per-model accuracy gains with per-language pretraining token counts.","The finding that prompting helps only some model families suggests that instruction-following behavior interacts with code-switching robustness in ways that could be probed by varying prompt position and phrasing systematically; the paper does not do that.","If correctness holds, downstream applications such as multilingual search, social-media moderation, and LLM training-data recycling should treat code-switched text as a distinct domain, not a minor variant of monolingual text."],"forward_implications":["If the asymmetry is real, LLM deployment in multilingual communities will systematically under-serve users who code-switch into a local language from English, while over-crediting comprehension when English words appear in local-language input.","Benchmarks built on this pipeline can be reused to track whether future models reduce the English-centric bias, by measuring the same accuracy deltas over time.","Fine-tuning on synthetic code-switched data, demonstrated on LLaMA-8B, offers a practical route to recover a few points of accuracy for models that do not respond to prompting.","The absence of a uniform Latin/non-Latin penalty in the extreme-mixing experiment implies that robustness to code-switching is not simply a function of script or number of languages mixed, but of model-specific language representations.","Because gains from embedding English appear mainly when the model is weak in the matrix language, English may act as a crutch that offsets gaps in multilingual proficiency rather than evidence of genuine cross-lingual comprehension."],"supporting_citations":[{"why":"Supplies the Equivalence Constraint Theory that defines where noun-token switches are grammatically permissible.","marker":"Poplack (1978)"},{"why":"Supplies the Matrix Language Frame model distinguishing matrix-language grammar from embedded-language content, the basis for the generation prompts.","marker":"Myers-Scotton (1993)"},{"why":"Provides the noun-token versus ratio-token substitution methods that the paper adapts and compares.","marker":"Chan et al. (2024)"},{"why":"Prior equivalence-constrained LLM generation that the paper extends with its two-step placeholder pipeline.","marker":"Kuwanto et al. (2024)"},{"why":"Shows LLMs can generate fluent code-switched text, motivating the LLM-centric generation approach.","marker":"Potter and Yuan (2024)"},{"why":"Source of the Belebele reading-comprehension benchmark whose passages and questions are code-switched.","marker":"Bandarkar et al. (2023)"},{"why":"Source of the MMLU multi-domain reasoning benchmark used for code-switched question evaluation.","marker":"Hendrycks et al. (2020)"},{"why":"Source of the XNLI natural-language-inference benchmark used as the third task.","marker":"Conneau et al. (2018)"}],"fun_headline_variants":["LLMs falter on foreign words in English, gain from English words abroad","Code-switching reveals English-centric bias in large language models","Why mixing languages confuses LLMs: English bias emerges","Foreign tokens hurt English reasoning, English tokens help others","LLMs show structural bias in code-switched text, fine-tuning helps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All conclusions rest on the assumption that the generated code-switched sentences faithfully represent natural code-switching; only 100 examples per language pair were human-checked, and the rest of the generation and quality judgments were handled by Claude 3.5 Sonnet and GPT-4o without per-instance human verification.","fun_headline_variants_meta":{"raw":{"variants":["LLMs falter on foreign words in English, gain from English words abroad","Code-switching reveals English-centric bias in large language models","Why mixing languages confuses LLMs: English bias emerges","Foreign tokens hurt English reasoning, English tokens help others","LLMs show structural bias in code-switched text, fine-tuning helps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000661,"raw_usage":{"total_tokens":3004,"prompt_tokens":909,"completion_tokens":2095,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":2008}},"tokens_in":525,"tokens_out":2095,"duration_ms":13949,"temperature":1.0,"reasoning_tokens":2008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:24:27.349534+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If bilingual human annotators rated a sample of the generated code-switched items as unnatural or as switching only easy, high-frequency nouns, and if the accuracy deltas on those items disappeared or reversed, the central asymmetry would be an artifact of the generation pipeline rather than a property of LLM comprehension.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Equivalence Constraint Theory that defines where noun-token switches are grammatically permissible."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows LLMs can generate fluent code-switched text, motivating the LLM-centric generation approach."}],"review_version":1}