Code-Switching and Syntax: A Large-Scale Experiment

Igor Sterner; Simone Teufel

REVIEW 2 major objections 5 minor 1 cited by

Code-Switching and Syntax: A Large-Scale Experiment

T0 review · 2 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a graph network fed only syntactic structure can distinguish natural code-switched sentences from manipulated ones as reliably as bilingual humans, and that the patterns learned on German–English transfer to other…

desk verdict A well-designed large-scale experiment showing that a syntax-only GNN matches humans on code-switching, but the paper's artifact-ceiling argument is flawed and needs a no-syntax control before the 'syntax alone' claim is fully established. read the letter →

arxiv 2506.01846 v2 pith:TXKEHTC6 submitted 2025-06-02 cs.CL

Igor Sterner , Simone Teufel This is my paper

classification cs.CL

keywords code-switchingsyntaxgraphneuralnetworksminimalpairsdependencieslanguageidentificationcross-lingualgeneralisationacceptabilityjudgement

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tests a long-standing but never large-scale claim in code-switching research: that bilingual speakers' decisions about where to switch languages are governed by syntactic structure alone. The authors train a small graph neural network, CSntax-GNN, whose only inputs are part-of-speech tags, dependency relations, and language IDs from the two monolingual paraphrases of a code-switched sentence. On German–English minimal pairs the model reaches 79.4% accuracy, statistically indistinguishable from bilingual human judges and above every large language model baseline. It still performs well on unseen written language pairs, and degrades on spoken data. The upshot is concrete evidence that general, cross-linguistically visible syntactic patterns in code-switching exist and can be learned without lexical semantics.

What carries the argument

The central object is CSntax-GNN, a graph isomorphism network with edge-featured message passing that operates on two dependency graphs per sentence, one for each monolingual translation, where nodes carry part-of-speech and language-ID features and edges carry dependency relation types. The graphs are embedded and the two embeddings are compared by a small MLP classifier, with only 2.7K trainable parameters updated end-to-end. The load-bearing role of the machinery is the ablation logic: randomizing language IDs yields 56.8%, used as an empirical ceiling on non-syntactic shortcuts from the pipeline, so the distance from that ceiling to the full model's 79.4% is the paper's operational measure of syntax's contribution.

What would settle it

A direct test is to train CSntax-GNN on the same 7,501 pairs but with the dependency edges randomly rewired while keeping POS and language features intact; if accuracy remains near 79.4%, the claim that relational dependency structure carries the result fails.

Watch

Extended reading notes

Core claim

The central claim is that syntactic structure alone is sufficient to determine which member of a minimal pair of code-switched sentences is the naturally produced one. CSntax-GNN receives no surface words, only the part-of-speech tags and dependency relations of the syntactic parse of each monolingual translation, together with the language of the aligned word in the original sentence. After training on 7,501 German–English pairs it achieves 79.4% accuracy on held-out pairs, statistically indistinguishable from human performance (p=0.293) and beating all baselines, including a 405-billion-parameter language model at 74.8%. Ablations that randomize the language IDs drop accuracy to 56.8%, which the authors treat as the ceiling of what the data-generation pipeline can leak; randomizing both POS tags and dependency relation types drops accuracy further, and randomizing all three features leaves 51.1%. The gap between the language-randomized ceiling and the full model is therefore attributed to genuine syntactic patterning.

Load-bearing premise

The load-bearing premise is that randomizing each token's language label destroys every shortcut the data-generation pipeline could leave, so all remaining accuracy is authentic syntax rather than an artifact of translation, alignment, or parsing.

Editorial extensions

If this is right

Syntactic acceptability of code-switching can be modelled without lexical semantics, so systems could be built for language pairs that lack code-switched training data, as long as dependency parsers and aligners are available.
The patterns learned from German–English transfer to other written language pairs with only a small drop, arguing for a shared syntactic core in written code-switching.
The larger drop on transcribed spoken pairs implies that written-trained syntax models should not be applied to spoken data without adaptation.
Because CSntax-GNN and the best language-model baseline disagree on many items, syntax-based and lexical-semantic cues appear complementary, motivating hybrid models.
The model's confidence carries no signal about human agreement, so its close overall accuracy should not be read as per-item human-like acceptability judgements.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An untested extension is to ablate one pipeline stage at a time—translation without alignment, alignment without dependency parsing—to pin down exactly which preprocessing step contributes to the 56.8% artifact ceiling.
The syntax operationalised here is dependency structure; a different syntactic theory, such as a tree-adjoining grammar or a converged-grammar model, might place switch constraints elsewhere, so the result supports dependency-structural syntax specifically.
A testable prediction is that re-injecting the language boundary in an alternative form, such as an edge feature rather than a node feature, should recover most of the gap between 73.5% and 79.4%, clarifying whether language identity contributes positionally or lexically.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here's my read of Sterner and Teufel. The new thing is a trained GNN that sees only POS tags, dependency relations, and token language IDs, plus a cross-language generalization experiment. The German-English result is real: 79.4% accuracy, statistically indistinguishable from bilingual humans, and better than a 405B Llama, on a test set that is temporally separated from training. The ablations and the XLM-R+FT baseline are the right controls, and the code is public. That deserves credit.

The main soft spot is the artifact-ceiling argument in Section 2.3. The authors treat the random-language-ID ablation (56.8%) as "the best possible performance attainable by exploiting the data generation procedure." That's not valid: randomizing language IDs removes the very feature that a non-syntactic shortcut would use, since the minimal-pair construction flips one word's language at the switch point. The ablation that randomizes POS and dependency labels but keeps language IDs and graph topology is more informative: it lands at 73.5%, which is 92.6% of baseline. That tells you the specific labels contribute only about six points, and the language IDs plus the dependency shape are doing most of the work. Without a control that has language IDs but no dependency structure, you cannot attribute the gap from 56.8% to 79.4% to syntax. That's a genuine hole, but fixable: run a sequence model on language IDs only, or a GNN with no edges. I don't think it overturns the main conclusion — the model still matches humans on the trained pair and generalizes to unseen pairs — but it does mean the paper's confident wording about "syntax alone" needs to be toned down.

Two smaller issues: human performance is reported without error bars or sample sizes, so the p=0.293 indistinguishability claim rests on thin evidence. And the generalization numbers for unseen pairs are not significance-tested; a per-pair test would help, since some pairs (tr-de, zh-en) are noticeably weaker.

Overall, this is a solid paper with a novel result and a careful methodology. I'd send it out to review and ask for the missing no-syntax baseline and better human metadata. It's worth a reading group slot.

Referee Report

2 major / 5 minor

Summary. The paper presents CSntax-GNN, a graph neural network trained to choose the original sentence from minimal pairs of code-switched (CS) sentences, using only the universal dependency parses of the two monolingual translations of each sentence, with POS tags and token-level language IDs as node features. On German–English written CS, the model reaches 79.4% accuracy on the ACS benchmark, statistically indistinguishable from human performance (p=0.293) and above all LLM baselines. Ablation studies and a temporal train/test split are used to argue that the model's performance is attributable to syntactic structure, and the model generalizes to unseen written language pairs (69.7–76.9%) and more weakly to spoken CS. The authors conclude that syntax alone is sufficient to model CS acceptability and that general syntactic patterns in CS exist.

Significance. If the attribution to syntax is established, this would be the largest-scale empirical confirmation of the theoretical consensus that syntax constrains code-switching, and the cross-lingual generalization would be a valuable new result. The paper has notable strengths: a temporal train/test split, permutation testing with reported p-values, several strong baselines (Llama-3 family, fine-tuned XLM-R), a simple and reproducible model (2.7K parameters, CPU training), and public code. The paper also includes a careful error analysis of the pipeline. However, the central interpretation of the ablations is currently under-supported, as the random-language-ID ablation does not bound the contribution of non-syntactic shortcuts in the way the paper claims. The significance of the paper therefore depends on whether the attribution can be made rigorous with additional controls.

major comments (2)

[Section 2.3, Table 2] The interpretation of the random-language-ID ablation (3) as 'an indication of the best possible performance attainable by exploiting the data generation procedure' is not valid. Because a minimal pair is created by changing the language of exactly one word at the switch point, the language-ID sequence of the two alternatives differs in a systematic way that is itself an artifact of the generation procedure. Randomizing language IDs removes that artifact, so the 56.8% figure is a lower bound for a model without language-ID access, not a ceiling for what a model with intact language IDs can exploit from shallow sequence statistics. The (1)+(2) ablation, which preserves language IDs and the dependency graph while randomizing POS tags and dependency relation labels, retains 73.5% accuracy, i.e., 92.6% of the baseline; this shows that the specifically syntactic labels contribute only a few points beyond language IDs and graph structure. Without a control that keeps language IDs but removes dependency structure (for example, a bag-of-nodes MLP on the language-ID/POS sequence, or a GNN with randomized edges), the 22.6-point gap from 56.8% to 79.4% cannot be uniquely attributed to syntax, and the central conclusion that 'general syntactic patterns in CS exist' is not yet established.
[Abstract and Section 2.1] The abstract's claim that the model has 'access only to syntactic information' is not consistent with the actual input. The node features defined in Section 2.1 are POS tags and token-level language IDs; language IDs are not part of the universal dependency representation and are not syntactic features in the usual sense. Ablation (3) shows that performance collapses toward chance (56.8%) when language IDs are randomized, so language IDs are a necessary and substantial component of the high accuracy. The authors should either justify language IDs as part of the syntactic representation for code-switching (e.g., because matrix/embedded language status is grammatically relevant), or restrict the claim to 'syntax plus language-ID information.' As it stands, the experimental design does not match the abstract's wording.

minor comments (5)

[Table 1] In the Human row, there is an extra value '3.1' after the de-en accuracy (79.4 3.1 83.1 ...). Please check whether this is a typo or an intended standard deviation, and if it is a standard deviation, report the corresponding uncertainty for all language pairs.
[Table 1, Section 2.2] Human accuracies are reported without sample sizes or confidence intervals, and the human-vs-model comparison uses an unpaired permutation test. The paper should state how many minimal pairs were judged by humans per language pair and report the uncertainty of the human estimates, so that the claim of statistical indistinguishability on de-en can be interpreted.
[Section 2.3] The sentence 'Randomising the token-level language features should yield random performance, but results show that it is 6.8% above random. This is an indication of the best possible performance attainable by exploiting the data generation procedure' conflates a floor with a ceiling; please rephrase to clarify that 56.8% is the accuracy obtained when language IDs are uninformative, not a bound on what intact language IDs can provide.
[Limitations] The Limitation section contains the duplicated phrase 'high resource resource language pair'; please correct this typo.
[Figure 2] The caption refers to yellow, green, and grey colors, but the figure appears to be rendered in grayscale; please ensure the color legend is visible in the published version.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark is self-cited but human-validated and held out, and the model's predictions do not reduce to its inputs.

full rationale

The paper's derivation chain is empirical: construct minimal pairs, encode them as dependency graphs with POS and language ID features, train a GNN, and evaluate on a temporal held-out test split. The self-cited ACS benchmark (Sterner and Teufel, 2025) is a published, human-validated dataset supplied to the model as input, not an output of this paper's fitted parameters. The central result, 79.4% accuracy statistically indistinguishable from human performance, is measured on test data never used for training, so it is a genuine prediction rather than a restatement of the training signal. The ablations, including randomizing language IDs, are interpretative controls and not definitions: the 56.8% random-language-ID condition is claimed as an 'indication' of a data-generation ceiling, but that is a construct-validity argument, not an equation-level reduction. Even if one doubts whether language IDs are properly excluded from 'syntax', such a concern targets the strength of the conclusion, not whether the result is circular. The paper does not fit a parameter and then rename it as a prediction, and no theorem or result is imported solely from the authors' prior work to force the conclusion. Therefore no circular step can be exhibited, and the derivation is self-contained with respect to circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the fidelity of the pipeline and the benchmark, not on ad hoc constants. The only fitted quantities are the model's learned weights (2.7K parameters), which are estimated on a held-out training set and evaluated on a temporal test split, so they are not post-hoc free parameters in the derivation sense. No new physical or theoretical entities are introduced.

assumptions (4)

domain assumption Universal Dependencies provide a language-neutral syntax representation sufficient for encoding code-switch constraints.
The model's input is UD parse graphs (Section 2.1). The conclusion that syntax alone suffices assumes UD captures the relevant syntactic distinctions across the 11 language pairs.
domain assumption The automatic pipeline (madlad-3B translation, awesome-align, stanza POS/dep parsing) introduces only bounded noise, not a systematic confound that explains the model's accuracy.
Section 2.2 describes the pipeline. The artifact-control baselines (XLM-R+FT, random-language-ID ablation) bound but do not eliminate this possibility.
domain assumption The gold label in each minimal pair (the observed CS sentence) is a valid target for acceptability, i.e., speakers' production is a reliable proxy for acceptability.
The ACS benchmark uses observed tweets as ground truth; human accuracy of 79.4% on de-en shows humans do not always prefer the observed sentence, so this premise is imperfect but used as ground truth.
standard math The permutation test assumptions are satisfied (exchangeability under the null for paired/unpaired comparisons).
Used in Section 2.2 Metrics for significance testing.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Code-Switching and Syntax: A Large-Scale Experiment." pith.science (2026). https://pith.science/paper/TXKEHTC6

@misc{pith2026250601846,
  author       = {Pith},
  title        = {Pith review of: Code-Switching and Syntax: A Large-Scale Experiment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TXKEHTC6}},
  note         = {Machine review of arXiv:2506.01846}
}

read the original abstract

The theoretical code-switching (CS) literature provides numerous pointwise investigations that aim to explain patterns in CS, i.e. why bilinguals switch language in certain positions in a sentence more often than in others. A resulting consensus is that CS can be explained by the syntax of the contributing languages. There is however no large-scale, multi-language, cross-phenomena experiment that tests this claim. When designing such an experiment, we need to make sure that the system that is predicting where bilinguals tend to switch has access only to syntactic information. We provide such an experiment here. Results show that syntax alone is sufficient for an automatic system to distinguish between sentences in minimal pairs of CS, to the same degree as bilingual humans. Furthermore, the learnt syntactic patterns generalise well to unseen language pairs.

Figures

Figures reproduced from arXiv: 2506.01846 by the authors.

**Figure 2.** Example input to the GNN. Original CS sentence and alignments provided only for reference. Yellow is [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** The training regime of CSntax-GNN. Nivre et al., 2017); UD was chosen because it provides a consistent syntactic representation across all languages. In UD dependency parse structures, words constitute nodes and edges between nodes represent directed dependencies. Nodes in our representation have two features: the POS of the word, and the language of the aligned word in the CS sentence. Each sentence in a minimal … view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Number of German–English training minimal pairs against accuracy of [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]

**Figure 5.** Figure 5: XLM-R+FT performance curves. Horizontal lines indicate training-free accuracies using pseudoprobabilities from the model directly [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Lost in Speech: Benchmarking, Evaluation, and Parsing of Spoken Bilingual Conversational Language Beyond Standard UD Assumptions
cs.CL 2026-02 reject novelty 5.0 of 10

A 126-sentence spoken code-switching benchmark, a flexible scoring metric, and an agentic LLM parser are introduced, but the paper's tables contradict its claimed accuracy gains.

Reference graph

Works this paper leans on

6 extracted references · 6 canonical work pages · cited by 1 Pith paper

[3]

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11908–11941, Miami, Florida, USA

Segment any Text: A universal approach for robust, efﬁcient and adaptable sentence segmentation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11908–11941, Miami, Florida, USA. Association for Computational Linguistics. Penelope Gardner-Chloros

work page 2024
[6]

In Proceedings of the Eleventh Workshop on NLP for Similar Languages, V arieties, and Dialects (V arDial 2024), pages 163–173, Mex- ico City, Mexico

Multilingual identiﬁcation of En- glish code-switching. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, V arieties, and Dialects (V arDial 2024), pages 163–173, Mex- ico City, Mexico. Association for Computational Lin- guistics. Igor Sterner and Simone Teufel

work page 2024
[1982]

In Coling 1982: Proceed- ings of the Ninth International Conference on Com- putational Linguistics

Processing of sentences with intra- sentential code-switching. In Coling 1982: Proceed- ings of the Ninth International Conference on Com- putational Linguistics. Thomas N. Kipf and Max Welling

work page 1982
[2021]

In Pro- ceedings of the Seventh Workshop on Noisy User- generated Text (W-NUT 2021), pages 158–167, On- line

Com- paring grammatical theories of code-mixing . In Pro- ceedings of the Seventh Workshop on Noisy User- generated Text (W-NUT 2021), pages 158–167, On- line. Association for Computational Linguistics. Peng Qi, Y uhao Zhang, Y uhui Zhang, Jason Bolton, and Christopher D. Manning

work page 2021
[2022]

In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing , pages 6191–6205, Abu Dhabi, United Arab Emirates

Ber- nice: A multilingual pre-trained encoder for Twitter . In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing , pages 6191–6205, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics. Zi-Yi Dou and Graham Neubig

work page 2022
[2024]

In Findings of the Association for Computational Lin- guistics: EACL 2024 , pages 1712–1721, St

Analyzing the role of part- of-speech in code-switching: A corpus-based study . In Findings of the Association for Computational Lin- guistics: EACL 2024 , pages 1712–1721, St. Julian’s, Malta. Association for Computational Linguistics. Michael Clyne

work page 2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [3]

In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11908–11941, Miami, Florida, USA

Segment any Text: A universal approach for robust, efﬁcient and adaptable sentence segmentation . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11908–11941, Miami, Florida, USA. Association for Computational Linguistics. Penelope Gardner-Chloros

work page 2024

[2] [6]

In Proceedings of the Eleventh Workshop on NLP for Similar Languages, V arieties, and Dialects (V arDial 2024), pages 163–173, Mex- ico City, Mexico

Multilingual identiﬁcation of En- glish code-switching. In Proceedings of the Eleventh Workshop on NLP for Similar Languages, V arieties, and Dialects (V arDial 2024), pages 163–173, Mex- ico City, Mexico. Association for Computational Lin- guistics. Igor Sterner and Simone Teufel

work page 2024

[3] [1982]

In Coling 1982: Proceed- ings of the Ninth International Conference on Com- putational Linguistics

Processing of sentences with intra- sentential code-switching. In Coling 1982: Proceed- ings of the Ninth International Conference on Com- putational Linguistics. Thomas N. Kipf and Max Welling

work page 1982

[4] [2021]

In Pro- ceedings of the Seventh Workshop on Noisy User- generated Text (W-NUT 2021), pages 158–167, On- line

Com- paring grammatical theories of code-mixing . In Pro- ceedings of the Seventh Workshop on Noisy User- generated Text (W-NUT 2021), pages 158–167, On- line. Association for Computational Linguistics. Peng Qi, Y uhao Zhang, Y uhui Zhang, Jason Bolton, and Christopher D. Manning

work page 2021

[5] [2022]

In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing , pages 6191–6205, Abu Dhabi, United Arab Emirates

Ber- nice: A multilingual pre-trained encoder for Twitter . In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing , pages 6191–6205, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics. Zi-Yi Dou and Graham Neubig

work page 2022

[6] [2024]

In Findings of the Association for Computational Lin- guistics: EACL 2024 , pages 1712–1721, St

Analyzing the role of part- of-speech in code-switching: A corpus-based study . In Findings of the Association for Computational Lin- guistics: EACL 2024 , pages 1712–1721, St. Julian’s, Malta. Association for Computational Linguistics. Michael Clyne

work page 2024