Minimal Pair-Based Evaluation of Code-Switching

Igor Sterner; Simone Teufel

REVIEW 3 major objections 5 minor 1 cited by

Minimal Pair-Based Evaluation of Code-Switching

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A minimal-pair benchmark built from real bilingual sentences and one-word manipulations tests whether language models share bilinguals' code-switching preferences, and finds that larger models agree more often.

desk verdict A genuinely useful code-switching benchmark that is undermined by an overclaimed scaling result and an unexamined token-length confound. read the letter →

arxiv 2506.01840 v2 pith:HDTM3GEN submitted 2025-06-02 cs.CL

Igor Sterner , Simone Teufel This is my paper

classification cs.CL

keywords code-switchingminimalpairsacceptabilityjudgmentsLLMevaluationbilingualismpart-of-speechbenchmarkmultilinguallanguagemodels

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Code-switching (using two languages in one sentence) is natural to bilinguals but hard to evaluate in language models, because existing ratings ask people to score sentences absolutely and do not compare candidates directly. This paper proposes an intervention: for each naturally occurring code-switched sentence, it translates one word at the switch point into the other language, creating a minimal pair whose only difference is that single word. Bilingual judges prefer the naturally occurring sentence in all 11 language pairs, with per-pair accuracies mostly between 75 and 88 percent. When current LLMs are scored on the resulting benchmark, larger models assign higher probability to the naturally occurring sentence more consistently than smaller models, and the best tested model still stays below the human agreement level. The authors intend the benchmark to make code-switching acceptability measurable once and reusable for any future system, replacing ad-hoc human judgments for every model.

What carries the argument

The central object is the minimal pair of code-switched sentences: one naturally occurring sentence paired with a variant created by translating exactly one word next to a switch point into the other language, with the number of switch points held constant between the two members. The construction pipeline combines token-level language identification, sentence segmentation, machine translation, word alignment, and dependency parsing to generate pairs in a language-agnostic way, and it excludes nouns, unaligned material, and non-integrative switches to keep pairs plausible. Evaluation is by accuracy: the proportion of pairs in which a model assigns higher probability to the naturally occurring sentence than to the manipulated variant, with the log-probability margin used as the measure of how large the acceptability difference is.

What would settle it

A reader could audit the assumption directly by running a larger human study on sampled pairs: if for any language pair bilinguals prefer the naturally occurring sentence no more often than chance, the gold standard fails for that pair. The paper already reports 22 German–English pairs in which all three annotators chose the manipulated variant instead, so examining those pairs systematically would show whether the failure is structured, for example when the manipulation fixes a typo in the original.

Watch

Extended reading notes

Core claim

The paper's central claim is that the acceptability of code-switching can be operationalised as a choice between two sentences that differ by one word: the sentence a bilingual actually produced and a variant produced by translating a single word adjacent to the switch point into the other language. Under the assumption that observed code-switching is acceptable and that the manipulation lowers acceptability, the naturally occurring sentence is the gold standard. The paper reports that bilinguals choose that sentence consistently across all 11 language pairs, with multi-rater agreement coefficient $\kappa = 0.57$ for German–English and accuracy ranging from about 75 to 88 percent. It then reports that LLM performance scales with size: the largest model tested is significantly better than all smaller models on most language pairs, reaching 74.8 percent accuracy on German–English, and that the probability margin between pair members is larger when the manipulated word belongs to a closed class such as auxiliaries or pronouns. This is presented as the first benchmark that directly measures LLM code-switching acceptability and permits testing of linguistic hypotheses about which word classes are more switchable.

Load-bearing premise

The benchmark's gold standard rests on the assumption that every naturally observed code-switched sentence is acceptable and that translating one word next to the switch point lowers acceptability on average; if many observed sentences are typos, mislabeled, or not genuine code-switching, the labels are wrong.

Editorial extensions

If this is right

Because human judgments are needed only once to validate how pairs are built, any new automatic system can be evaluated on the benchmark by comparing its sentence probabilities, making code-switching evaluation cheap and repeatable.
Accuracy on the benchmark rises with model size within the tested model families, and the largest tested model significantly beats all smaller ones on most language pairs, so the benchmark can track future progress in code-switching modelling.
The part-of-speech analysis provides direct empirical support for the long-standing claim that closed-class words (auxiliaries, pronouns, determiners) are more switchable than open-class words (verbs, adjectives, adverbs) in almost all language pairs.
Spoken transcription code-switching is not inherently easier or harder for LLMs than written code-switching: Turkish–German shows the same scaling trend as social-media-derived pairs, whereas the different-script Chinese–English pair shows no improvement with scale.
Even the best tested model remains below the human agreement ceiling, so current LLMs do not fully capture bilingual code-switching acceptability.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the scaling trend persists in future base models, the benchmark could act as a canary for whether pretraining corpora contain enough natural code-switched text, in the same way monolingual minimal-pair benchmarks track grammaticality.
The same one-word-swap intervention could be applied to additional language pairs, including non-Latin-script pairs, to separate the Chinese–English plateau's likely cause (no shared tokens between scripts) from genuine difficulty differences in code-switching modelling.
The closed-class result would be cleaner if tested after controlling for word frequency, word length, and tokenizer behaviour, since those confounds could create larger margins without a linguistic cause.
The 22 German–English pairs where all human judges preferred the manipulated sentence form a ready-made audit set for the assumption that the observed sentence is always acceptable; some may be cases where the manipulation accidentally corrected an error.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper builds the first algorithmically constructed minimal-pair benchmark for code-switching, covering 11 language pairs with up to 1,000 pairs each. That part is solid and a real contribution. The method - translating one word adjacent to a switch point in a naturally occurring CS sentence - is language-agnostic in principle and the human validation, even with limited annotators, shows that the manipulation usually lowers acceptability. The data and code are promised. This is worth the community's attention.

The soft spots are in the claims, not the resource. The abstract says 'the larger the model, the more consistently it assigns higher probability to the naturally occurring CS sentence.' The paper's own Table 7 shows Chinese-English going 66.2, 65.7, 71.1, 70.6, 67.7 across Llama sizes, i.e., no monotone scaling, and for five other language pairs the 405B model is statistically indistinguishable from the 70B. The conclusion should be qualified to 'often' or 'for most language pairs.' That is a straightforward fix.

The token-length issue is more substantive. The metric compares unnormalized sentence probabilities, and the manipulation changes the token count by construction. The paper cites Ueda et al. against normalization, but that does not answer whether the observed CS sentence is systematically shorter, making a length preference masquerade as CS acceptability. A simple control - accuracy when the shorter sentence always wins, or a length-matched subset - should be reported before the scaling claim is taken as established. This is not a fatal flaw; it is an absent baseline.

The human validation is thin: one annotator per pair for 10 of 11 language pairs. Accuracy around 80% means the benchmark labels are decent but noisy. That is acceptable for a benchmark, but the paper should present a per-language breakdown of the annotator's agreement with the gold standard, which it does, and acknowledge that conclusions about 'bilinguals consistently prefer' rest on a single individual for most pairs. The error analysis showing 76% fair pairs in German-English is honest, and the limitation section is unusually candid.

Who this is for: anyone building or evaluating CS generation systems. It deserves a serious referee, not a desk reject. I would accept it with major revisions: narrow the abstract, add a token-length control, and ideally add a second annotator for a subset of the other language pairs. The underlying benchmark is likely to be reused even if the current experiments are corrected.

Referee Report

3 major / 5 minor

Summary. The paper introduces ACS, a benchmark of minimal pairs for code-switching (CS) acceptability across 11 language pairs. Each pair consists of a naturally occurring CS sentence from social media or spoken corpora and a manipulated variant created by translating one word adjacent to the switch point. The authors validate the benchmark with human judgments, report inter-annotator agreement for German–English, and evaluate several open-weight LLM families using unnormalized sentence probabilities. They report three main findings: humans prefer the observed CS sentence on average; larger LLMs more often assign higher probability to the observed CS sentence; and larger absolute LLM margins occur for closed-class word manipulations, consistent with theoretical claims.

Significance. The ACS benchmark is a potentially valuable resource for CS evaluation: it is publicly released, covers an unusually wide set of language pairs, and avoids the subjectivity of absolute acceptability ratings by using controlled minimal pairs. The human validation experiments, the error analysis on a 100-pair sample, and the use of permutation tests are strengths that make the claims checkable. If the scaling claim can be stated with the proper qualifications and the token-length confound is addressed, the benchmark would be a useful tool for tracking LLM progress on code-switching. The paper is honest about several limitations, including noisy Twitter data, one-participant validation for most language pairs, and uncertainty about training-data leakage.

major comments (3)

[Abstract and §3.2] The abstract's scaling claim, 'the larger the model, the more consistently it assigns higher probability to the naturally occurring CS sentence than to the variant,' is contradicted by the paper's own results. Section 3.2 reports that for Chinese–English 'there is no improvement with model size (all comparisons have p>0.05)', and the following paragraph states that Llama-3.1 405B is statistically indistinguishable from smaller models for Danish–English, Indonesian–English, Dutch–English, Swedish–English, and Turkish–English. Appendix Table 7 confirms this: for zh-en the 405B accuracy is 67.7% while the 70B model reaches 70.6%, and for da-en the 405B is 74.5% versus 74.2% for 70B. The scaling claim therefore cannot be stated as a general result without specifying the language pairs and model families for which it holds.
[§3.1, Eq. (1) and footnote 7] The accuracy metric and margin are computed from unnormalized sentence probabilities, and the minimal-pair manipulation changes the token count by construction. The footnote cites Ueda et al. (2024) to justify not normalizing by token count, but that citation addresses a specific normalization method, not whether token-length differences are a confound in this benchmark. Because English words are often tokenized into fewer subword tokens than words in several other languages covered here, the observed CS sentence may systematically be the shorter one. A baseline that simply chooses the shorter sentence, or an analysis conditioned on token-length difference, is needed to establish that the LLM accuracies in Figures 2–4 and the margin analyses in Figures 3 and 5 reflect CS acceptability rather than a token-length preference. This is load-bearing for the scaling claim.
[§2 and §2.2 (error analysis)] The gold-standard assumption, stated in Section 2 as 'We assume that all observed CS is acceptable' and that the manipulation will on average decrease acceptability, is only partially verified. The error analysis on 100 German–English pairs found that only 76% of genuine CS sentences produced 'fair' pairs, and Section 2.3 reports 22 minimal pairs for which all three human annotators chose the manipulated sentence. The Limitations section also acknowledges that Twitter noise 'can contribute to the acceptability differential,' with the 'belive'/'glauben' example showing that the manipulation can make the variant more correct than the observed sentence. Label noise of this kind can attenuate or distort both human and LLM accuracy, so the paper should either filter out known error cases, report robustness analyses excluding them, or quantify how much the main results depend on pairs whose labels are most uncertain.

minor comments (5)

[§2.1] The statement that the treated languages 'must be written in Latin script' is inconsistent with the later inclusion of Chinese–English; please clarify that this condition applies only to the Twitter-derived language pairs and that the Chinese–English and Turkish–German data are complements from spoken corpora.
[§2.3] For the ten language pairs other than German–English, human validation rests on a single participant per pair. The abstract's phrase 'bilinguals consistently prefer' is stronger than the evidence; the paper should state that the single bilingual participant in each pair preferred the observed sentence on average, or provide additional participants for those pairs.
[Limitations] There is a typo in the Limitations section: 'we also also cannot be sure' should read 'we also cannot be sure.'
[Figure 4 and Appendix D] The caption 'Horizontal light blue lines represent the accuracy achieved by our participants' is misleading for language pairs with a single participant; consider labeling these lines as 'single-participant accuracy' or 'mean human accuracy' as appropriate.
[Table 7] The zh-en row of Table 7 shows non-monotonic accuracy across model sizes (66.2, 65.7, 71.1, 70.6, 67.7); since this row is central to the exception to the scaling claim, it should be discussed in the main text rather than only in the appendix.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: benchmark labels are external observed data and LLM scores are direct measurements, not fitted predictions.

full rationale

The benchmark labels are external observed data (naturally occurring tweets and spoken transcriptions), not outputs of the models under evaluation, and the LLM accuracy and margin scores contain no fitted parameters. The core claims—bilinguals prefer the observed CS sentence and larger LLMs assign it higher probability—are direct measurements against those external labels. The paper's reliance on its own AnE and sat-12-sm tools is supported by separate published evaluations and by an internal error analysis on 100 German–English pairs; these tools affect data construction but do not by themselves force the scaling, human-agreement, or POS-margin results. The 'all observed CS is acceptable' assumption is explicitly stated and is tested against independent human judgments rather than derived from them. The token-length difference footnote acknowledges a potential confound but does not reduce the prediction to the input; it is a validity concern, not circularity. No equation in the paper defines the model output in terms of the labels, and no fitted parameter is renamed as a prediction. The self-citations are present but not load-bearing for the central claims, so the circularity score is low.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The benchmark's claims rest on these domain assumptions. They are stated and partially validated, but not proven, and the paper's own error analysis shows the first two assumptions can fail.

free parameters (3)

Noun-exclusion rule at switch points
Section 2.2 disallows minimal pairs where a noun is adjacent to a switch point to keep complexity comparable. This shapes benchmark coverage and may affect POS-based findings.
Minimum POS group size = 10
Section 3.2 restricts POS analysis to categories with at least 10 minimal pairs, which can change which POS classes are compared.
Data filtering thresholds = 6 tokens minimum, 200 characters maximum, Levenshtein distance below 5, under 50 percent unknown words, 1,000 pair cap
Sections 2.1 and 2.2: thresholds chosen manually to reduce noise; they determine which sentences become minimal pairs and therefore what the benchmark measures.

assumptions (4)

domain assumption All observed CS is acceptable
Section 2: 'We assume that all observed CS is acceptable, and we can get a near-unlimited supply of these from social media.' The benchmark's gold standard is the observed sentence.
domain assumption Translating one word at a switch point decreases acceptability on average
Section 2: 'we propose to manipulate observed CS sentences, under the assumption that on average this manipulation will decrease acceptability.' The entire minimal-pair construction depends on this.
domain assumption Automatic NLP tools are accurate enough
Sections 2.1 and 2.2 rely on AnE LID, Lingua, MADLAD-400 MT, awesome-align, Stanza, and sat-12-sm; error analysis only audits 100 German-English pairs, with 76 percent fair pairs.
domain assumption LLM sentence probabilities can be compared across sentences without length normalization
Section 3.1 compares model sentence log-probs directly; the authors cite Ueda et al. (2024) for not normalizing, but tokenizer differences remain a risk.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Minimal Pair-Based Evaluation of Code-Switching." pith.science (2026). https://pith.science/paper/HDTM3GEN

@misc{pith2026250601840,
  author       = {Pith},
  title        = {Pith review of: Minimal Pair-Based Evaluation of Code-Switching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HDTM3GEN}},
  note         = {Machine review of arXiv:2506.01840}
}

read the original abstract

There is a lack of an evaluation methodology that estimates the extent to which large language models (LLMs) use code-switching (CS) in the same way as bilinguals. Existing methods do not have wide language coverage, fail to account for the diverse range of CS phenomena, or do not scale. We propose an intervention based on minimal pairs of CS. Each minimal pair contains one naturally occurring CS sentence and one minimally manipulated variant. We collect up to 1,000 such pairs each for 11 language pairs. Our human experiments show that, for every language pair, bilinguals consistently prefer the naturally occurring CS sentence. Meanwhile our experiments with current LLMs show that the larger the model, the more consistently it assigns higher probability to the naturally occurring CS sentence than to the variant. In accordance with theoretical claims, the largest probability differences arise in those pairs where the manipulated material consisted of closed-class words.

Figures

Figures reproduced from arXiv: 2506.01840 by the authors.

**Figure 2.** ACS (de-en) model performance. 3/3 2/3 1/3 0/3 −10 0 10 20 *** Human agreement LLM Margin [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Human agreement with the gold standard vs. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Llama-3 model performance across three other language pairs. Horizontal light blue lines represent the accuracy achieved by our participants. but in Kodali et al.’s (2024) experiment, it performs best on their related CS task. Here, it performs better than others (per…

**Figure 5.** Figure 5: Absolute LLM margin vs. part of speech of the [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]

**Figure 6.** Figure 6: Absolute LLM margin vs. part of speech of minimal pair change for tweet-derived minimal pairs. Blue [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: Absolute LLM margin vs. part of speech of minimal pair change for data derived from transcriptions of [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: Llama-3 model performance. Horizontal light blue lines represent the accuracy achieved by our participants [PITH_FULL_IMAGE:figures/full_fig_p022_8.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions
cs.CL 2026-02 reject novelty 5.0 of 10

A 126-sentence spoken code-switching benchmark, a flexible scoring metric, and an agentic LLM parser are introduced, but the paper's tables contradict its claimed accuracy gains.

Reference graph

Works this paper leans on

8 extracted references · 6 canonical work pages · cited by 1 Pith paper

[1]

For the work here, we do not include these wordlists

One of the problems in our previous corpus was Swiss German sentences, because we decided to include Swiss and Austrian wordlists in the compilation process. For the work here, we do not include these wordlists

work page
[2]

Such words are almost never German/English words

We also exclude tweets which contain words with an umlaut but not in the German wordlist. Such words are almost never German/English words

work page
[3]

We search for German words in a bilingual dictionary (dict.cc) which have identical corresponding English entries, and vice versa

We found many monolingual tweets which arise due to incorrectly identified interlingual homographs and other shared words between the languages. We search for German words in a bilingual dictionary (dict.cc) which have identical corresponding English entries, and vice versa. We remove all such words from their wordlists of each language. A second aspect o...

work page
[4]

menschgemachten

We only keep tweets with fewer than 50% unknown words. Borrowed words Borrowed words are words of foreign origin used in one language that are fully assimilated and hence considered part of that language. They form a grey area in language change, and for our purposes should not be altered in a minimal pair. We collect lists of borrowed words for each lang...

work page 2022
[2018]

InInterspeech, pages 1938–1942

The role of cognate words, pos tags and en- trainment in code-switching. InInterspeech, pages 1938–1942. Igor Sterner. 2024.Multilingual identification of En- glish code-switching. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 163–173, Mexico City, Mexico. Association for Computational ...

work page 1938
[2021]

DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing

DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing. Preprint, arXiv:2105.06681v1. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos- quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023.Findings of the BabyLM challenge: Sample-efficient ...

work page Pith review arXiv 2023
[2024]

Preprint, arXiv:2411.06096v1

ZhoBLiMP: a systematic assessment of lan- guage models with linguistic minimal pairs in chinese. Preprint, arXiv:2411.06096v1. Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Peng Xu, Yan Xu, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J. Barezi, Qifeng Chen, Xi- aojuan Ma, Bertram Shi, and Pascale Fung. 2022. ASCEND: A spontaneous Chinese-E...

arXiv 2022
[2025]

Preprint, arXiv:2504.02768v2

MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal pairs. Preprint, arXiv:2504.02768v2. Carina Kauf and Anna Ivanova. 2023.A better way to do masked language model scoring. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 925–935, Toronto, Canada. Associati...

arXiv 2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

For the work here, we do not include these wordlists

One of the problems in our previous corpus was Swiss German sentences, because we decided to include Swiss and Austrian wordlists in the compilation process. For the work here, we do not include these wordlists

work page

[2] [2]

Such words are almost never German/English words

We also exclude tweets which contain words with an umlaut but not in the German wordlist. Such words are almost never German/English words

work page

[3] [3]

We search for German words in a bilingual dictionary (dict.cc) which have identical corresponding English entries, and vice versa

We found many monolingual tweets which arise due to incorrectly identified interlingual homographs and other shared words between the languages. We search for German words in a bilingual dictionary (dict.cc) which have identical corresponding English entries, and vice versa. We remove all such words from their wordlists of each language. A second aspect o...

work page

[4] [4]

menschgemachten

We only keep tweets with fewer than 50% unknown words. Borrowed words Borrowed words are words of foreign origin used in one language that are fully assimilated and hence considered part of that language. They form a grey area in language change, and for our purposes should not be altered in a minimal pair. We collect lists of borrowed words for each lang...

work page 2022

[5] [2018]

InInterspeech, pages 1938–1942

The role of cognate words, pos tags and en- trainment in code-switching. InInterspeech, pages 1938–1942. Igor Sterner. 2024.Multilingual identification of En- glish code-switching. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), pages 163–173, Mexico City, Mexico. Association for Computational ...

work page 1938

[6] [2021]

DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing

DaLAJ - a dataset for linguistic acceptability judgments for Swedish: Format, baseline, sharing. Preprint, arXiv:2105.06681v1. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mos- quera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and Ryan Cotterell. 2023.Findings of the BabyLM challenge: Sample-efficient ...

work page Pith review arXiv 2023

[7] [2024]

Preprint, arXiv:2411.06096v1

ZhoBLiMP: a systematic assessment of lan- guage models with linguistic minimal pairs in chinese. Preprint, arXiv:2411.06096v1. Holy Lovenia, Samuel Cahyawijaya, Genta Winata, Peng Xu, Yan Xu, Zihan Liu, Rita Frieske, Tiezheng Yu, Wenliang Dai, Elham J. Barezi, Qifeng Chen, Xi- aojuan Ma, Bertram Shi, and Pascale Fung. 2022. ASCEND: A spontaneous Chinese-E...

arXiv 2022

[8] [2025]

Preprint, arXiv:2504.02768v2

MultiBLiMP 1.0: A massively multilingual benchmark of linguistic minimal pairs. Preprint, arXiv:2504.02768v2. Carina Kauf and Anna Ivanova. 2023.A better way to do masked language model scoring. In Proceed- ings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pa- pers), pages 925–935, Toronto, Canada. Associati...

arXiv 2023