REVIEW 3 major objections 9 minor 15 references
Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review
T0 review · 3 major / 9 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read A systematic review of 127 peer-reviewed papers finds end-to-end code-switching ASR concentrated in a few language pairs—Mandarin-English alone is about 55% of studies—with no consistent benchmarking across the field.
desk verdict Useful E2E code-switching ASR field map; the 55% Mandarin-English share is directionally right but not a precise figure. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The machinery is the review protocol itself. The authors queried the Semantic Scholar API for papers matching (ASR | Speech recognition) together with (code-switch* | codeswitch* | code switch* | code-mix* | codemix*), kept only peer-reviewed work describing end-to-end code-switching systems (127 papers), and coded each paper along four annotated dimensions: problem setup and data (languages, datasets, accessibility), model design (monolingual vs multilingual modeling, language identification, text units, architecture, pretrained models, loss, decoding), training and evaluation (augmentation, translation, zero-shot, metrics), and best reported performance. Each paper was annotated by one of five annotators working from agreed dimensions in scheduled sessions. That grid is what turns heterogeneous papers into comparable statistics like the ~55% and ~77% figures, and every conclusion in the paper flows through it.
What would settle it
Re-run the collection through a second scholarly literature index with the query expanded by terms like 'mixed-language,' 'language mixing,' and 'code-alternation,' and have two annotators independently code a random sample of 30 papers. If the Mandarin-English share moves by more than a few percentage points, or if agreement on core fields (languages covered, dataset accessibility, model type) is poor, the review's headline statistics would need revision, even if the qualitative picture of concentration and fragmentation stands.
Extended reading notes
Core claim
On the paper's own terms, the discovery is a measured snapshot of a young field. From 378 candidate papers retrieved through the Semantic Scholar API, filtering to peer-reviewed venues and end-to-end systems—neural ASR trained jointly in a single computational graph—leaves 127 papers published between 2018 and 2024, over half of them since 2022. Manual annotation across problem setup, model design, training and evaluation, and performance yields three headline facts. First, research concentrates in a few pairs: Mandarin-English is covered by ~55% of papers; adding Hindi-English and Arabic-English reaches ~76%, and momentum tracks the availability of accessible datasets such as SEAME (~24% of papers) and the ASRU 2019 challenge (~11%). Second, ~77% of papers use accessible datasets while the rest rely on proprietary or unspecified data, and the Japanese-English cluster (~5%) has no public dedicated dataset at all, leaning on synthetic speech built from a machine-translation corpus. Third, no experimental standard exists: roughly 60% of studies add monolingual data, more than a third use multilingual modeling, ~47% use encoder-decoder architectures, ~33% incorporate language identification, ~45% an external language model, ~30% data augmentation, and best reported results on different datasets share no common recipe. The authors conclude that efforts are 'mostly sporadic, with no consistent benchmarking or clarity on directions for future research,' and note that no state-of-the-art result on the popular datasets comes from zero-shot evaluation, which they read as evidence that dedicated code-switched training data remains essential.
Load-bearing premise
The headline numbers rest on the assumption that the search query plus the peer-review filter captured essentially the whole universe of end-to-end code-switching ASR work, and that single-annotator coding is accurate enough to support figures like 55%, 76%, and 77%.
Editorial extensions
If this is right
- Dataset availability, not linguistic need, drives the research agenda: the Mandarin-English cluster grew around SEAME and the ASRU 2019 challenge, so creating one accessible corpus for an under-studied pair is the highest-leverage intervention the data support.
- Because there is no consistent benchmarking, positive findings from earlier studies have not been replicated on newer datasets; reported gains should be treated as provisional until a shared evaluation framework appears.
- Dedicated code-switched training data remains necessary: no state-of-the-art system in the surveyed tables relies on zero-shot evaluation, so multilingual pretraining alone does not substitute for CS data.
- Evaluation itself is unsettled: plain WER penalizes mixed-script output, which is why mixed metrics (MER, TER) and transliteration-aware metrics (toWER, poWER, PolyWER) are appearing; standardizing among them is part of the open problem.
- The field is still young and accelerating—over half the surveyed papers appeared since 2022—so the window for establishing standardized benchmarks is open now.
Reading between the lines
- The census itself is shaped by its search terms: work framed as 'mixed-language,' 'language mixing,' or dialectal alternation without code-switch/code-mix vocabulary would be invisible to the query, so the true literature is probably larger and more linguistically diverse than 127 papers; the concentration percentages could shift if that framing gap were closed.
- A quantitative extension the authors leave implicit: measuring citation continuity—whether later papers on the same dataset cite and improve on earlier baselines—would convert the 'sporadic efforts' claim from an impression into a number, and the paper's own reference table makes this computation straightforward.
- The paper's logic supports a testable, supply-side prediction: releasing a public Japanese-English or Indonesian-English corpus of roughly SEAME's scale would plausibly generate a visible publication cluster within two years; future bibliographies could confirm or refute this.
- Read with the fairness discussion, the findings imply a concrete allocation rule rather than a research direction: directing benchmark and dataset funding at the roughly thirty language pairs that each appear in a single paper would rebalance the field more than further architectural innovation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a systematic literature review of end-to-end (E2E) automatic speech recognition (ASR) for code-switching (CS). The authors query the Semantic Scholar API with a documented boolean query, retain 127 peer-reviewed papers, and manually annotate them across dimensions such as language pairs, datasets, model architecture, decoding strategy, and evaluation metrics. The central empirical claims are that research is heavily concentrated in a few language pairs (Mandarin–English ~55%, top-3 pairs ~76%), that ~77% of papers use accessible datasets, and that the field lacks standardized benchmarking and evaluation practices. The paper also discusses challenges and opportunities around data scarcity, coverage disparities, evaluation, and reproducibility.
Significance. If the quantitative findings are reliable, this review fills a real gap: it is the first systematic survey focused specifically on E2E ASR for code-switching, and it provides a DOI-level enumeration of all 127 surveyed papers (Table 7), a documented search query, and a transparent annotation scheme. The qualitative picture — that Mandarin–English dominates because of dataset availability and that evaluation is fragmented — is plausible and useful for guiding future research. The paper also makes honest statements about its own coding assumptions (e.g., Section 4.6, the greedy-decoding assumption), which is a strength. However, the headline percentages are load-bearing, and they rest on two methodological premises: search-query recall and single-annotator coding reliability. These premises are not adequately addressed in the manuscript, which limits the strength of the quantitative claims even though the overall qualitative conclusions may well survive.
major comments (3)
- [Section 2 (Data Collection) and Section 3] The search query is limited to variants of 'code-switch*' and 'code-mix*'. Work that studies the same phenomenon under alternative terminology — 'language mixing', 'mixed-language speech', 'language switching', or 'code-alternation' — would be omitted unless those terms co-occur with 'code-switch' in the indexed fields. Because the central quantitative claims in Section 3 (Mandarin–English ~55%, top-3 ~76%, accessible-dataset share ~77%) are computed from the 127-paper sample, the review needs to assess recall, for example by running a broader query and reporting how the distribution shifts, or by explicitly identifying known relevant papers that the query missed. Without such a sensitivity check, the headline statistics are not robust to plausible alternative terminologies.
- [Section 2 (Annotation)] Each paper was annotated by one annotator, and no inter-annotator agreement (IAA) is reported. The review's conclusions are expressed as percentages over categorical codes (language pair, dataset accessibility, architecture class, etc.), so the reliability of those codes is load-bearing. The authors should report IAA on a subsample, provide dual coding for the attributes that feed the central statistics, or otherwise justify the consistency of single-annotator coding across five annotators. Without this, the precision of numbers like 55% and 76% is unverified.
- [Section 2 and Table 7] The inclusion criterion 'published in peer-reviewed venues' is asserted but not operationalized, and Table 7's note says the year corresponds to the earliest available version, 'which can be a preprint'. Some entries in the reference list are clearly preprint-form (CoRR), and the manuscript does not explain how venue status was verified or whether preprint-only items were included. Since the 127-paper corpus is the basis for all of the review's statistics, the paper should clarify the verification procedure and state explicitly whether preprints were admitted or excluded.
minor comments (9)
- [Section 3.2] 'ITTG-HingCos' should be 'IITG-HingCos' to match the reference and Table 3.
- [Section 4.3] 'languauge independent vocabulary' contains a typo ('languauge' should be 'language').
- [Section 7] 'with the exception of of a small set of languages' has a duplicated 'of'.
- [Section 4.1] The phrase 'the combined system is retained using monolingual data' should read 'retrained'.
- [Section 4.6] The assumption that unmentioned decoding strategies are greedy is stated transparently, but only seven papers explicitly mention greedy decoding; the review would be stronger if it reported the sensitivity of any decoding-related conclusions to this assumption.
- [Section 2] A PRISMA-style flow diagram (378 retrieved, exclusions by reason) would improve reproducibility and make the filtering process easier to verify.
- [Section 5.2] The phrase 'or to used TTS synthesis' is grammatically incomplete; it should read 'or to use TTS synthesis'.
- [References] Several reference entries contain OCR-style spacing errors, e.g., 'V oice' in Grand View Research, 'V ancouver' in Zhang et al. (2018), and 'V enice' in Zhu et al. (2017).
- [Table 7] The entry 'Kilkarn' should be 'Killkan' to match the cited dataset name in the reference list.
Circularity Check
No circularity: the survey's statistics are computed from its annotated corpus, not derived from its conclusions or from self-cited prior work.
full rationale
This paper is a systematic literature review; it does not propose a model, fit parameters, or derive a result from assumptions that encode the conclusion. The central claims—that Mandarin-English accounts for roughly 55% of papers, that the top three language pairs account for about 76%, and that evaluation is non-standardized—are computed directly from the 127 annotated papers retrieved by the stated Semantic Scholar query. The annotation scheme, language-pair coding, and dataset tables are descriptive aggregations of the corpus, so no quantity is assumed into existence. The only self-citations (Mixat, PolyWER, and related prior work by the authors) appear as surveyed data points or as cited context, and the review's conclusions do not depend on the correctness of those prior results. The skeptic's concern about the search query omitting papers using alternative terminology such as 'language mixing' or 'mixed-language' is a legitimate validity risk that could shift the reported percentages, but it is a sampling/completeness issue, not circularity: the statistics would still be computed from whatever corpus the query defines. Similarly, the absence of inter-annotator agreement is a reliability limitation, not a circular derivation. No load-bearing step reduces to the paper's own inputs by definition, and no fitted parameter is renamed as a prediction. Under the review's criteria, the honest finding is no significant circularity.
Assumptions & free parameters
assumptions (4)
- domain assumption The Semantic Scholar API query with the stated code-switch/code-mix terms retrieves the relevant universe of peer-reviewed E2E ASR code-switching papers.
- ad hoc to paper Each paper can be reliably assigned to the annotation attributes by one annotator after consensus sessions.
- ad hoc to paper Most unmentioned decoding strategies are greedy.
- domain assumption Standard and mixed error metrics (WER, CER, MER, TER) are comparable enough across papers for 'best result' comparisons.
Cite this review
Pith. "Pith review of Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review." pith.science (2026). https://pith.science/paper/PY2VMWUX
@misc{pith2026250707741,
author = {Pith},
title = {Pith review of: Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review},
year = {2026},
howpublished = {\url{https://pith.science/paper/PY2VMWUX}},
note = {Machine review of arXiv:2507.07741}
}
read the original abstract
Motivated by a growing research interest into automatic speech recognition (ASR), and the growing body of work for languages in which code-switching (CS) often occurs, we present a systematic literature review of code-switching in end-to-end ASR models. We collect and manually annotate papers published in peer reviewed venues. We document the languages considered, datasets, metrics, model choices, and performance, and present a discussion of challenges in end-to-end ASR for code-switching. Our analysis thus provides insights on current research efforts and available resources as well as opportunities and gaps to guide future research.
Figures
Reference graph
Works this paper leans on
-
[6]
PolyWER: A holistic evaluation framework for code-switched speech recognition. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6144–6153, Miami, Florida, USA. Asso- ciation for Computational Linguistics. Abbas Khosravani, Philip N. Garner, and Alexandros Lararidis. 2021. An evaluation benchmark for auto- matic speech recogn...
work page 2024
-
[8]
Code-mixed text augmentation for Latvian ASR. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages 3469–3479, Torino, Italia. ELRA and ICCL. Mari Ganesh Kumar, Jom Kuriakose, Anand Thya- gachandran, Ashish Seth, Lodagala Durga Prasad, Saish Jaiswal, Anusha Prak...
arXiv 2024
-
[455]
IEEE. Charles A. Ferguson. 1959. Diglossia. WORD, 15(2):325–340. Grand View Research. 2024. V oice and speech recognition market size, share & trends analysis report by function (speech recognition, voice recognition), by technology, by vertical, by region, and segment forecasts, 2024 - 2030. https://www. grandviewresearch.com/industry-analysis/ voice-rec...
work page 1959
-
[816]
Gwantae Kim, Bokyeung Lee, Donghyeon Kim, and Hanseok Ko
IEEE. Gwantae Kim, Bokyeung Lee, Donghyeon Kim, and Hanseok Ko. 2024. Gated low-rank adaptation for personalized code-switching automatic speech recog- nition on the low-spec devices. InIEEE International Conference on Acoustics, Speech, and Signal Process- ing, ICASSP 2024 - Workshops, Seoul, Republic of Korea, April 14-19, 2024, pages 760–764. IEEE. Mar...
work page 2024
-
[2014]
Building Bilingual Corpora , pages 93–110
5. Building Bilingual Corpora , pages 93–110. Multilingual Matters, Bristol, Blue Ridge Summit. Kunal Dhawan, Ganji Sreeram, Kumar Priyadarshi, and Rohit Sinha. 2020. Investigating target set reduction for end-to-end speech recognition of Hindi-English code-switching data. In 2020 National Conference on Communications, NCC 2020, Kharagpur , India, Februar...
work page 2020
-
[2020]
It’s Kind of Like Code-Switching
ArzEn: A speech corpus for code-switched Egyptian Arabic-English. In Proceedings of The 12th Language Resources and Evaluation Confer- ence, LREC 2020, Marseille, France, May 11-16, 2020, pages 4237–4246. European Language Re- sources Association. Christina N. Harrington, Radhika Garg, Amanda T. Woodward, and Dimitri Williams. 2022. “It’s Kind of Like Cod...
work page 2020
-
[2022]
In Interspeech 2022, pages 3924–3928
Language-specific characteristic assistance for code-switching speech recognition. In Interspeech 2022, pages 3924–3928. Ganji Sreeram, Kunal Dhawan, and Rohit Sinha. 2019. IITG-HingCoS corpus: A Hinglish code-switching database for automatic speech recognition. Speech Commun., 110:76–89. Ganji Sreeram and Rohit Sinha. 2020. Exploration of end-to-end fram...
work page 2022
-
[2023]
Handling of Indonesian-English codeswitch- ing speech in end-to-end indonesian speech recogni- tion system using connectionist temporal classifica- tion model. In International Conference on Electri- cal Engineering and Informatics, ICEEI 2023, Ban- dung, Indonesia, October 10-11, 2023 , pages 1–6. IEEE. Hardik B. Sailor, Kiran Praveen, Vikas Agrawal, Abh...
work page 2023
Show all 15 references
-
[2024]
¿ te vienes? sure!
"¿ te vienes? sure!" joint fine-tuning of lan- guage detection and transcription improves automatic recognition of code-switching speech. In Proceed- ings of the 6th ACM Conference on Conversational User Interfaces, pages 1–7. Chris Hokamp and Qun Liu. 2017. Lexically con- str...
2017
-
[2251]
IEEE Computer Society. A Lists of Datasets, Languages, Metrics and Papers Language Pair ISO-3 Code # Papers Dataset Breakdown Mandarin–English zho-eng 70 ( , 32) ( , 15) ( , 6) ( , 5) ( , 2) ( , 1) ( , 1) ( , 1) ( , 1) ( , 11) Hindi–English hin-eng 15 ( , 5) ( , 3) ( , 1) ( , ...
2019
-
[2460]
Hiroshi Seki, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux, and John R
ISCA. Hiroshi Seki, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux, and John R. Hershey. 2019. End-to-end multilingual multi-speaker speech recognition. In 20th Annual Conference of the International Speech Communication Association, Interspeech 2019, Graz, Austria, September...
2019 arXiv
-
[5079]
Brian Yan, Matthew Wiesner, Ondrej Klejch, Preethi Jyothi, and Shinji Watanabe
European Language Resources Association. Brian Yan, Matthew Wiesner, Ondrej Klejch, Preethi Jyothi, and Shinji Watanabe. 2023. Towards zero- shot code-switched speech recognition. In IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing ICASSP 2023, Rhod...
2023 arXiv
-
[6243]
Cao Hong Nga, Duc-Quang Vu, Huong Hoang Lu- ong, Chien-Lin Huang, and Jia-Ching Wang
IEEE. Cao Hong Nga, Duc-Quang Vu, Huong Hoang Lu- ong, Chien-Lin Huang, and Jia-Ching Wang. 2023. Cyclic transfer learning for Mandarin-English code- switching speech recognition. IEEE Signal Process. Lett., 30:1387–1391. Tolulope Ogunremi, Christopher Manning, and Dan Ju- raf...
2023
-
[7268]
Yizhou Lu, Mingkun Huang, Hao Li, Jiaqi Guo, and Yanmin Qian
European Language Resources Association. Yizhou Lu, Mingkun Huang, Hao Li, Jiaqi Guo, and Yanmin Qian. 2020. Bi-encoder transformer network for Mandarin-English code-switching speech recog- nition using mixture of experts. In 21st Annual Con- ference of the International Speec...
2020
-
[7893]
Abigail Z
IEEE. Abigail Z. Jacobs and Hanna Wallach. 2021. Mea- surement and Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 375–385, Virtual Event Canada. ACM. Palash Jain and Anirban Bhowmick. 2024. Vitb-hebic: A bilingual corpu...
2021
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.