Pith. sign in

REVIEW 4 major objections 5 minor 53 references

Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Foreign words in English confuse LLMs; English words abroad help

desk verdict Solid and genuinely new empirical study, but the headline asymmetry rests on thin statistics and unverified generation fidelity; deserves review with pressure on the facilitation claim. read the letter →

arxiv 2506.14012 v1 pith:VSZXNJTF submitted 2025-06-16 cs.CL

Amr Mohamed , Yang Zhang , Michalis Vazirgiannis , Guokan Shang This is my paper
classification cs.CL
keywords code-switchinglargelanguagemodelsmultilingualcomprehensionbenchmarkgenerationEquivalenceConstraintTheoryMatrixFrameLLMevaluationnaturalinference
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models truly understand code-switched text—sentences that alternate between two languages—rather than merely recognizing it. To find out, the authors generate code-switched versions of three established reasoning benchmarks (Belebele, MMLU, XNLI) using switches constrained by two linguistic theories of where code-switching is grammatical. They find an asymmetry: replacing nouns in English sentences with Arabic, German, French, or Chinese words consistently lowers model accuracy, even when the switches follow linguistic constraints, while replacing nouns in those languages with English words often raises accuracy, especially when the model is weak in the matrix language. Prompting to expect code-switching helps some models and hurts others, whereas fine-tuning on code-switched data yields steadier, though partial, recovery. The paper's central claim, if correct, is that LLMs harbor an English-centric structural bias in mixed-language processing, not just a token-level vocabulary gap.

What carries the argument

The load-bearing object is the code-switched benchmark generation pipeline itself: a two-step LLM-centric approach in which Claude 3.5 Sonnet first marks candidate switch points (nouns for the linguistically grounded variant, random tokens for the heuristic variant) and then fills the placeholders with words from a parallel sentence under instructions to respect the Equivalence Constraint Theory and the Matrix Language Frame model. An alignment-based pipeline using the AWESOME aligner with LaBSE embeddings and Stanza POS tagging serves as the controlled comparison, and GPT-4o acts as judge to select the more natural outputs. The evaluation machinery is the accuracy delta, the difference between accuracy on a code-switched benchmark and its monolingual baseline, which makes the asymmetry visible: negative deltas when foreign tokens enter English, positive deltas when English enters other languages.

What would settle it

If bilingual human annotators rated a sample of the generated code-switched items as unnatural or as switching only easy, high-frequency nouns, and if the accuracy deltas on those items disappeared or reversed, the central asymmetry would be an artifact of the generation pipeline rather than a property of LLM comprehension.

Watch

Extended reading notes

Core claim

Using a two-step LLM pipeline built on the Equivalence Constraint Theory and the Matrix Language Frame model, the paper replaces nouns in English benchmark items with translations in Arabic, German, French, and Chinese (noun-token CSW), and in a second experiment replaces roughly 20% of random tokens (ratio-token CSW). On weighted average accuracy across Belebele, MMLU, and XNLI, LLaMA-70B drops from 0.70 in English to 0.66 on EN→AR and EN→DE; Qwen-7B falls from 0.61 to 0.52–0.57. Reversing the roles, Mistral-7B on Arabic rises from 0.35 to 0.48 when English nouns are embedded, and gains appear for other low-proficiency matrix languages as well. Extreme mixing with four embedded languages does not always cause the largest drops, and script type (Latin vs. non-Latin) has no uniform effect. The results are taken as evidence that the degradation is structural, not merely lexical unfamiliarity.

Load-bearing premise

All conclusions rest on the assumption that the generated code-switched sentences faithfully represent natural code-switching; only 100 examples per language pair were human-checked, and the rest of the generation and quality judgments were handled by Claude 3.5 Sonnet and GPT-4o without per-instance human verification.

Editorial extensions

If this is right

  • If the asymmetry is real, LLM deployment in multilingual communities will systematically under-serve users who code-switch into a local language from English, while over-crediting comprehension when English words appear in local-language input.
  • Benchmarks built on this pipeline can be reused to track whether future models reduce the English-centric bias, by measuring the same accuracy deltas over time.
  • Fine-tuning on synthetic code-switched data, demonstrated on LLaMA-8B, offers a practical route to recover a few points of accuracy for models that do not respond to prompting.
  • The absence of a uniform Latin/non-Latin penalty in the extreme-mixing experiment implies that robustness to code-switching is not simply a function of script or number of languages mixed, but of model-specific language representations.
  • Because gains from embedding English appear mainly when the model is weak in the matrix language, English may act as a crutch that offsets gaps in multilingual proficiency rather than evidence of genuine cross-lingual comprehension.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's noun-token design means the measured asymmetry concerns noun substitutions only; a natural extension would switch verbs, adjectives, or function words, which the Equivalence Constraint Theory treats differently, and which may produce larger or opposite effects.
  • An implicit testable claim is that the English-as-embedded-language boost should shrink as models' non-English proficiency improves; one could verify this by correlating per-model accuracy gains with per-language pretraining token counts.
  • The finding that prompting helps only some model families suggests that instruction-following behavior interacts with code-switching robustness in ways that could be probed by varying prompt position and phrasing systematically; the paper does not do that.
  • If correctness holds, downstream applications such as multilingual search, social-media moderation, and LLM training-data recycling should treat code-switched text as a distinct domain, not a minor variant of monolingual text.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces a pipeline for generating code-switched (CSW) versions of three established benchmarks (Belebele, MMLU, XNLI) using two strategies: a linguistically motivated noun-token method grounded in the Equivalence Constraint Theory (ECT) and Matrix Language Frame (MLF) model, and a heuristic ratio-token method that substitutes roughly 20% of aligned tokens at random. The authors evaluate eight LLMs across five languages and report two main findings: (i) embedding non-English tokens into English text degrades performance, and (ii) embedding English tokens into non-English text frequently improves comprehension, especially for models with weaker proficiency in the matrix language. They also test prompt-based and fine-tuning-based mitigation strategies, concluding that fine-tuning yields more stable recovery. The central claim is an asymmetric, English-centric structural bias in LLM processing of code-switched input.

Significance. If the asymmetric effect is genuine and attributable to properties of code-switching rather than to artifacts of the generation procedure, the paper would make a valuable contribution to multilingual NLP evaluation and to the study of English-centric biases in LLMs. The authors release code and data, evaluate a diverse set of models and languages, and ground their generation in established linguistic theories, which are all concrete strengths. However, the headline claims currently rest on small test sets without statistical support and on a generation pipeline whose fidelity is only weakly validated, so the significance of the finding is conditional on addressing these issues.

major comments (4)
  1. [Section 3.4 and Tables 1–3] The test set comprises only 300 samples per embedded language, and the reported deltas (e.g., LLaMA-70B on EN→AR: 0.70→0.66, Δ≈−0.04) correspond to roughly 12 examples on 300 instances. No confidence intervals, bootstrap estimates, or significance tests are reported anywhere in the paper. Given the number of models (8) × languages (4) × tasks (3) comparisons, many of the headline differences are within the noise band of a 300-sample evaluation. The authors should report uncertainty intervals and, if appropriate, apply multiple-testing corrections, or otherwise justify that the deltas are not chance-level fluctuations.
  2. [Section 3.2 and Tables 1–2] The noun-token and ratio-token methods are compared to isolate the effect of linguistic constraints, but the two conditions differ in more than constraints: noun-token replaces all identified nouns, while ratio-token replaces ~20% of aligned tokens at random, which may include function words and content words of varying categories, at a different substitution rate. Therefore, the Discussion's inference (Section 8) that comparable degradation under both methods reveals a 'structural vulnerability' beyond token-level unfamiliarity is confounded. A matched comparison (e.g., same proportion of random nouns, or same token category with and without ECT/MLF constraint) is necessary before this claim can be drawn.
  3. [Appendix B.1/B.4, Tables 6/8, and Section 3.3] The validity of the central asymmetric result—particularly the facilitation observed when English is embedded into non-English matrix languages (Table 3)—depends on the generated CSW faithfully representing natural code-switching. However, the generation-selection evidence is thin: only 100 examples per language were human-checked for the choice of generation model (preferences as low as 52% for Arabic in Table 6), and the decisive comparison between the LLM-Centric and Alignment-Based pipelines was judged solely by GPT-4o (Table 8), with no human verification and preferences as low as 53.4% for German. Furthermore, the LLM-Centric prompt (Figures 4–5) explicitly instructs the model to avoid switching nouns in common expressions and to adjust inflections, potentially biasing the inserted nouns toward short, high-frequency, semantically transparent items. If so, the observed EL-matrix improvement (e.g., Mistral-7B on Arabic: 0.35→0.48) could be an artifact of the predictability or recoverability of the inserted English words rather than a genuine CSW comprehension effect. The authors should provide per-instance human evaluation of the final benchmark instances (naturalness, switch-point grammaticality, and the representativeness of the chosen nouns) and report inter-annotator agreement.
  4. [Section 3.3 and Section 6.1 (Table 3)] When English is the embedded language, the inserted English words are drawn from the parallel English source sentence (the original benchmark text), while the baseline is the non-English translation. For models that are English-centric and have limited proficiency in the matrix language, replacing selected nouns with the original English nouns may simply make the instance easier because those exact English words are from the source-side text and are highly accessible in the model's pretraining distribution. This confound could trivially produce the facilitation effect reported in Table 3. The authors should control for this by, for example, inserting English nouns that are not taken from the parallel source sentence, or by comparing against insertion of matrix-language nouns matched for frequency and semantic content; otherwise the 'English embedding improves comprehension' claim is not separable from a source-leakage artifact.
minor comments (5)
  1. [References] Several references are malformed: 'Li Nguyen and. 2018' is missing the author's given name, 'Pranjal Khanuja et al. 2020' should be 'Khanuja et al.', and 'Genta Winata et al. 2021a' is inconsistently formatted; the reference list should be checked for completeness and consistency.
  2. [Figure 7] The caption of Figure 7 says the prompt was given to Claude 3.5 Sonnet for choosing the best summary, but the text in Appendix B.2/B.4 states that GPT-4o was used as the judge; the caption also refers to 'summaries' rather than code-switched sentences. Please correct the model attribution and the description.
  3. [Table 5] In Table 5, the entry for Llama 3B on EN→AR contains an extraneous ':a' after the accuracy value; please fix the typo.
  4. [Abstract and Section 7.2] The abstract states that 'fine-tuning offers a more stable path', but the fine-tuning experiment is performed on a single model (Llama 8B) with gains of +0.03–0.04; the claim should be tempered to reflect the limited evidence, or additional models should be fine-tuned.
  5. [Section 3.1] The notation 'B = {Bp}P p=1' is typeset incorrectly; it should read B = {B_p}_{p=1}^P. Please also define the indexing and the product set explicitly.

Circularity Check

0 steps flagged · score 2.0 of 10

No construction-level circularity: the evaluation is self-contained against external benchmarks, with only a peripheral self-citation preventing a score of 0.

full rationale

The paper's central claims are empirical accuracy deltas (Eq. 3) measured on fixed external benchmarks (Belebele, MMLU, XNLI) by frozen public LLMs (Llama, Qwen, Mistral, ALLaM). No parameter is fitted to the evaluation data, and no predictive quantity is defined in terms of the measured quantities. The code-switched stimuli are generated by a Claude-based pipeline and selected via GPT-4o preferences plus a small human check, but the evaluated models are independent of the generator, so the measured deltas are not forced by construction. The self-citation to Mohamed et al. (2025) in Section 4 is used only as a loose analogy for language-pair similarity effects in machine translation and is not load-bearing for any conclusion. The acknowledged limitations (noun-token switching only, fixed 20% ratio-token rate) concern scope and external validity, not circularity. Even if the generation fidelity is arguable, that is a data-quality concern, not a derivation-level circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to data for the central claim; the 20% ratio-token rate is an experimental design choice. The work introduces no new theoretical entities. The load-bearing assumptions are domain-level: that the ECT/MLF-constrained generation pipeline produces natural CSW and that LLM-as-a-judge preferences reflect human quality judgments.

free parameters (1)
  • ratio-token substitution rate = 0.20 (20%)
    Chosen following Chan et al. (2024); not fitted, but it directly controls the difficulty of the non-linguistic condition, and results may shift with this rate.
assumptions (4)
  • domain assumption Equivalence Constraint Theory and Matrix Language Frame model correctly characterize natural code-switching, and Claude's generation respects them.
    Invoked in Section 3.2 and Appendix B prompting; no per-instance validation of the final benchmarks.
  • domain assumption GPT-4o's pairwise preferences correlate with human judgments of code-switched text quality and are free of generator bias.
    Used in Section 3.4 and Appendix B.2 to select the generation approach and alignment backbone.
  • domain assumption The five languages (EN, AR, DE, FR, ZH) span enough linguistic distance to support Hypothesis 1, with no explicit distance metric.
    Section 4 states the set creates varying degrees of similarity, but distance is not quantified.
  • domain assumption Baseline and code-switched evaluations are comparable: same harness, same prompts, and same data instances in the compared conditions.
    Required by the delta-accuracy metric in Eq. (3); the paper does not report per-instance pairing explicitly.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text." pith.science (2026). https://pith.science/paper/VSZXNJTF

@misc{pith2026250614012,
  author       = {Pith},
  title        = {Pith review of: Lost in the Mix: Evaluating LLM Understanding of Code-Switched Text},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VSZXNJTF}},
  note         = {Machine review of arXiv:2506.14012}
}
abstract

Code-switching (CSW) is the act of alternating between two or more languages within a single discourse. This phenomenon is widespread in multilingual communities, and increasingly prevalent in online content, where users naturally mix languages in everyday communication. As a result, Large Language Models (LLMs), now central to content processing and generation, are frequently exposed to code-switched inputs. Given their widespread use, it is crucial to understand how LLMs process and reason about such mixed-language text. This paper presents a systematic evaluation of LLM comprehension under code-switching by generating CSW variants of established reasoning and comprehension benchmarks. While degradation is evident when foreign tokens disrupt English text$\unicode{x2013}$even under linguistic constraints$\unicode{x2013}$embedding English into other languages often improves comprehension. Though prompting yields mixed results, fine-tuning offers a more stable path to degradation mitigation.

Figures

Figures reproduced from arXiv: 2506.14012 by the authors.

Figure 1
Figure 1. An example illustrating the noun-token CSW methodology from Experiment 1. The figure demonstrates [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of LLM accuracy on monolingual English versions of [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Comparison of Llama 8B and its instruction￾tuned variant (CSW-Llama 8B) on monolingual English benchmarks (Belebele, MMLU, and XNLI) versus their noun-token code-switched counterparts. English serves as the matrix language, with Arabic, French, German, and Chinese, as embedded languages. selecting samples from the parallel corpus where the English sentence length was greater than 70 words. This filtering yielded app… view at source ↗
Figures from the paper (12 more)
Figure 4
Figure 4. Figure 4: Step 1 — Placeholder identification prompt (noun-token variant). You will be given a pair of parallel texts in English and { target_language }. Your goal is to produce a code - switched version of the English text by replacing each of the hashtag - sequences (#######) …
Figure 5
Figure 5. Figure 5: Step 2 — Placeholder filling prompt (noun-token variant). 13 [PITH_FULL_IMAGE:figures/full_fig_p013_5.png]
Figure 6
Figure 6. Figure 6: Prompt used in the ratio-token variant (random placeholder insertion). B.4 Final Generation Approach Selection We compared the Alignment-Based Pipeline and the LLM-Centric Method for generating noun-token CSW text across 100 samples per language and benchmark. Results …
Figure 7
Figure 7. Figure 7: The prompt given to Claude 3.5 Sonnet for choosing the best summary between the baseline and LLM [PITH_FULL_IMAGE:figures/full_fig_p014_7.png]
Figure 8
Figure 8. Figure 8: Instructional prompt prepended for Belebele multiple-choice QA tasks. MMLU Prompt You are an expert in understanding code - switched text . You will be given a question in code - switched English and Arabic . You have to understand it and respond to the given question …
Figure 9
Figure 9. Figure 9: Instructional prompt prepended for MMLU multiple-choice QA tasks. XNLI Prompt You are an expert in understanding code - switched text . You will be given two code - switched passages that correspond to a premise and a hypothesis in code - switched English and Arabic te…
Figure 10
Figure 10. Figure 10: Instructional prompt prepended for XNLI natural language inference tasks. D Instruction Tuning for Model-Based Mitigation We fine-tuned LLaMA-3.1-8B-Instruct to improve its comprehension of code-switched text using a targeted instruction-tuning dataset. Full-model tra…
Figure 11
Figure 11. Figure 11: Infusion-style template. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_11.png]
Figure 12
Figure 12. Figure 12: Conversion-style template. Blend English and < LANGUAGE > in the sentence below : English text : " < ENGLISH_SENTENCE > " < LANGUAGE > equivalent : " < TRANSLATION_SENTENCE > " [PITH_FULL_IMAGE:figures/full_fig_p016_12.png]
Figure 13
Figure 13. Figure 13: Blending-style template. Generate a code - switched rendition by swapping in < LANGUAGE >: English original : " < ENGLISH_SENTENCE > " < LANGUAGE > snippet : " < TRANSLATION_SENTENCE > " [PITH_FULL_IMAGE:figures/full_fig_p016_13.png]
Figure 14
Figure 14. Figure 14: Rendition-style template. Switch parts of this English sentence into < LANGUAGE >: English : " < ENGLISH_SENTENCE > " < LANGUAGE >: " < TRANSLATION_SENTENCE > " [PITH_FULL_IMAGE:figures/full_fig_p016_14.png]
Figure 15
Figure 15. Figure 15: Switching-style template. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 23 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. https://aclanthology.org/2020.lrec-1.223/ L in CE : A centralized benchmark for linguistic code-switching evaluation . In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1803--1813, Marseille, France. European Language Resources Association

    work page 2020

  4. [4]

    Q Jiang Albert, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, and Devendra Singh Chaplot. 2023. Mistral 7b. arXiv

    work page 2023

  5. [5]

    Li Nguyen and. 2018. https://doi.org/10.1080/07268602.2018.1510727 Borrowing or code-switching? traces of community norms in vietnamese-english speech . Australian Journal of Linguistics, 38(4):443--466

    work page arXiv 2018

  6. [6]

    Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023. The belebele benchmark: a parallel reading comprehension dataset in 122 language variants. arXiv preprint arXiv:2308.16884

    arXiv 2023

  7. [7]

    M Saiful Bari, Yazeed Alnumay, Norah A Alzahrani, Nouf M Alotaibi, Hisham A Alyahya, Sultan AlRashed, Faisal A Mirza, Shaykhah Z Alsubaie, Hassan A Alahmed, Ghadah Alabduljabbar, et al. 2024. Allam: Large language models for arabic and english. arXiv preprint arXiv:2407.15390

    arXiv 2024

  8. [8]

    Bullock and Almeida Jacqueline Toribio

    Barbara E. Bullock and Almeida Jacqueline Toribio. 2009. The Cambridge Handbook of Linguistic Code-switching. Cambridge Handbooks in Language and Linguistics. Cambridge University Press

    work page 2009

Show all 53 references
  1. [9]

    Kelvin Wey Han Chan, Christopher Bryant, Li Nguyen, Andrew Caines, and Zheng Yuan. 2024. https://aclanthology.org/2024.lrec-main.698/ Grammatical error correction for code-switched sentences by learners of E nglish . In Proceedings of the 2024 Joint International Conference on...

    2024

  2. [10]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \'a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://doi.org/10.18653/v1/2020.acl-main.747 Unsupervised cross-lingual representation learning ...

    2020 doi

  3. [11]

    Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. https://doi.org/10.18653/v1/D18-1269 XNLI : Evaluating cross-lingual sentence representations . In Proceedings of the 2018 Conference on Empirical Methods ...

    2018 doi

  4. [12]

    Richeek Das, Sahasra Ranjan, Shreya Pathak, and Preethi Jyothi. 2023. https://doi.org/10.18653/v1/2023.acl-long.66 Improving pretraining techniques for code-switched NLP . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon...

    2023 doi

  5. [13]

    Sourya Dipta Das, Ayan Basak, Soumil Mandal, and Dipankar Das. 2022. Advcodemix: Adversarial attack on code-mixed data. In Proceedings of the 5th Joint International Conference on Data Science & Management of Data (9th ACM IKDD CODS and 27th COMAD), pages 125--129

    2022

  6. [14]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  7. [15]

    Seza Do g ru \"o z, Sunayana Sitaram, Barbara E

    A. Seza Do g ru \"o z, Sunayana Sitaram, Barbara E. Bullock, and Almeida Jacqueline Toribio. 2021. https://doi.org/10.18653/v1/2021.acl-long.131 A survey of code-switching: Linguistic and social perspectives for language technologies . In Proceedings of the 59th Annual Meeting...

    2021 doi

  8. [16]

    Zi-Yi Dou and Graham Neubig. 2021. https://doi.org/10.18653/v1/2021.eacl-main.181 Word alignment by fine-tuning embeddings on parallel corpora . In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2...

    2021 doi

  9. [17]

    Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. https://doi.org/10.18653/v1/2022.acl-long.62 Language-agnostic BERT sentence embedding . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long...

    2022 doi

  10. [18]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  11. [19]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  12. [20]

    Nicolas Guerin, Shane Steinert-Threlkeld, and Emmanuel Chemla. 2024. The impact of syntactic and semantic proximity on machine translation with back-translation. arXiv preprint arXiv:2403.18031

    2024 arXiv

  13. [21]

    Ayushman Gupta, Akhil Bhogal, and Kripabandhu Ghosh. 2024. Code-mixer ya nahi: Novel approaches to measuring multilingual llms' code-mixing capabilities. arXiv preprint arXiv:2410.11079

    2024 arXiv

  14. [22]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300

    2020 arXiv

  15. [23]

    Maite Heredia, Gorka Labaka, Jeremy Barnes, and Aitor Soroa. 2025. Conditioning llms to generate code-switched text: A methodology grounded in naturally occurring data. arXiv preprint arXiv:2502.12924

    2025

  16. [24]

    Muhammad Huzaifah, Weihua Zheng, Nattapol Chanpaisit, and Kui Wu. 2024. https://aclanthology.org/2024.lrec-main.565/ Evaluating code-switching translation with large language models . In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Langu...

    2024

  17. [25]

    Pranjal Khanuja et al. 2020. https://aclanthology.org/2020.acl-main.338 Improving code-switched nlp using data augmentation . In Proceedings of ACL 2020, pages 1860--1871

    2020

  18. [26]

    Prashant Kodali, Anmol Goel, Likhith Asapu, Vamshi Krishna Bonagiri, Anirudh Govil, Monojit Choudhury, Manish Shrivastava, and Ponnurangam Kumaraguru. 2024. From human judgements to predictive models: Unravelling acceptability in code-mixed sentences. arXiv preprint arXiv:2405.05572

    2024 arXiv

  19. [27]

    Garry Kuwanto, Chaitanya Agarwal, Genta Indra Winata, and Derry Tanti Wijaya. 2024. Linguistics theory meets llm: Code-switched text generation via equivalence constrained large language models. arXiv preprint arXiv:2410.22660

    2024 arXiv

  20. [28]

    Zihan Liu, Genta Indra Winata, Zhaojiang Lin, Peng Xu, and Pascale Fung. 2020. https://doi.org/10.1609/aaai.v34i05.6362 Attention-informed mixed-language training for zero-shot cross-lingual task-oriented dialogue systems . Proceedings of the AAAI Conference on Artificial Inte...

    2020 doi

  21. [29]

    Amr Mohamed, Mingmeng Geng, Michalis Vazirgiannis, and Guokan Shang. 2025. Llm as a broken telephone: Iterative generation distorts information. arXiv preprint arXiv:2502.20258

    2025

  22. [30]

    Melissa G. Moyer. 2002. https://doi.org/10.1017/S004740450224405X Pieter muysken, bilingual speech: A typology of code-mixing. cambridge: Cambridge university press, 2000. pp. xvi, 306. hb 59.95. Language in Society, 31(4):621–624

    2002 doi

  23. [31]

    P. Muysken. 2000. https://books.google.fr/books?id=lJI7qrIKmokC Bilingual Speech: A Typology of Code-Mixing . Cambridge University Press

    2000

  24. [32]

    Myers-Scotton

    R. Myers-Scotton. 1993. Social Motivations for Code-Switching: Evidence from Africa. Oxford University Press

    1993

  25. [33]

    Mark Mysl \'i n. 2014. https://api.semanticscholar.org/CorpusID:272681368 Codeswitching and predictability of meaning in discourse . In Codeswitching and predictability of meaning in discourse

    2014

  26. [34]

    Lynnette Hui Xian Ng and Luo Qi Chan. 2024. What talking you?: Translating code-mixed messaging texts to english. arXiv preprint arXiv:2411.05253

    2024 arXiv

  27. [35]

    Millicent Ochieng, Varun Gumma, Sunayana Sitaram, Jindong Wang, Vishrav Chaudhary, Keshet Ronen, Kalika Bali, and Jacki O'Neill. 2024. Beyond metrics: evaluating llms' effectiveness in culturally nuanced, low-resource real-world scenarios. arXiv preprint arXiv:2406.00343

    2024 arXiv

  28. [36]

    Tanmay Parekh, Emily Ahn, Yulia Tsvetkov, and Alan W Black. 2020. https://doi.org/10.18653/v1/2020.conll-1.46 Understanding linguistic accommodation in code-switched human-machine dialogues . In Proceedings of the 24th Conference on Computational Natural Language Learning, pag...

    2020 doi

  29. [37]

    o rn Gamb \

    Parth Patwa, Gustavo Aguilar, Sudipta Kar, Suraj Pandey, Srinivas PYKL, Bj \"o rn Gamb \"a ck, Tanmoy Chakraborty, Thamar Solorio, and Amitava Das. 2020. https://doi.org/10.18653/v1/2020.semeval-1.100 S em E val-2020 task 9: Overview of sentiment analysis of code-mixed tweets ...

    2020 doi

  30. [38]

    Shana Poplack. 1988. https://doi.org/doi:10.1515/9783110849615.215 8. Contrasting patterns of codeswitching in two communities , pages 215--244. De Gruyter Mouton, Berlin, New York

    1988 doi

  31. [39]

    Susan Poplack. 1978. Sometimes i'll start a sentence in spanish y termino en español: Toward a typology of code-switching. Linguistics, 16(7-8):581--618

    1978

  32. [40]

    Tom Potter and Zheng Yuan. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.942 LLM -based code-switched text generation for grammatical error correction . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16957--16965, Miami, Fl...

    2024 doi

  33. [41]

    Adithya Pratapa, Gayatri Bhat, Monojit Choudhury, Sunayana Sitaram, Sandipan Dandapat, and Kalika Bali. 2018. https://doi.org/10.18653/v1/P18-1143 Language modeling for code-mixing: The role of linguistic theory based synthetic data . In Proceedings of the 56th Annual Meeting ...

    2018 doi

  34. [42]

    Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. https://doi.org/10.18653/v1/2020.acl-demos.14 S tanza: A python natural language processing toolkit for many human languages . In Proceedings of the 58th Annual Meeting of the Association for Co...

    2020 doi

  35. [43]

    Ye Qi, Devendra Sachan, Matthieu Felix, Sarguna Padmanabhan, and Graham Neubig. 2018. https://doi.org/10.18653/v1/N18-2084 When and why are pre-trained word embeddings useful for neural machine translation? In Proceedings of the 2018 Conference of the North A merican Chapter o...

    2018 doi

  36. [44]

    Samson Tan and Shafiq Joty. 2021. https://doi.org/10.18653/v1/2021.naacl-main.282 Code-mixing on sesame street: Dawn of the adversarial polyglots . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu...

    2021 doi

  37. [45]

    Genta Winata et al. 2021 a . https://aclanthology.org/2021.emnlp-main.190 Multilingual pretrained models are effective for code-switching nlp . In Proceedings of EMNLP 2021, pages 2345--2356

    2021

  38. [46]

    Genta Indra Winata, Andrea Madotto, Zhaojiang Lin, Rosanne Liu, Jason Yosinski, and Pascale Fung. 2021 b . https://doi.org/10.18653/v1/2021.mrl-1.1 Language models are few-shot multilingual learners . In Proceedings of the 1st Workshop on Multilingual Representation Learning, ...

    2021 doi

  39. [47]

    Genta Indra Winata, Andrea Madotto, Chien-Sheng Wu, and Pascale Fung. 2019. https://doi.org/10.18653/v1/K19-1026 Code-switched language models using neural based synthetic data from parallel sentences . In Proceedings of the 23rd Conference on Computational Natural Language Le...

    2019 doi

  40. [48]

    Anjali Yadav, Tanya Garg, Matej Klemen, Matej Ulcar, Basant Agarwal, and Marko Robnik Sikonja. 2024. Code-mixed sentiment and hate-speech prediction. arXiv preprint arXiv:2405.12929

    2024 arXiv

  41. [49]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

    2025 arXiv

  42. [50]

    Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Fikri Aji. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.774 Multilingual large language models are not (yet) code-switchers . In Proceedings of the 2023 Conference on Empirical Methods i...

    2023 doi

  43. [51]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223, 1(2)

    2023 arXiv

  44. [52]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  45. [53]

    Zhihong Zhu, Xuxin Cheng, Zhiqi Huang, Dongsheng Chen, and Yuexian Zou. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.486 Enhancing code-switching for cross-lingual SLU : A unified view of semantic and grammatical coherence . In Proceedings of the 2023 Conference on Empiri...

    2023 doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.