Pith. sign in

REVIEW 3 major objections 9 minor 15 references

Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review

T0 review · 3 major / 9 minor · reviewed 2026-08-06 · deepseek-v4-flash

Pith's one-line read A systematic review of 127 peer-reviewed papers finds end-to-end code-switching ASR concentrated in a few language pairs—Mandarin-English alone is about 55% of studies—with no consistent benchmarking across the field.

desk verdict Useful E2E code-switching ASR field map; the 55% Mandarin-English share is directionally right but not a precise figure. read the letter →

arxiv 2507.07741 v1 pith:PY2VMWUX submitted 2025-07-10 cs.CL cs.SDeess.AS

classification cs.CLcs.SDeess.AS
keywords code-switchingend-to-endautomaticspeechrecognitionsystematicliteraturereviewlanguage-paircoveragecode-switcheddatasetsASRevaluationmetricsmultilingualmodelsbenchmarking
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks, after a decade of growth, which languages, datasets, models, and metrics actually populate end-to-end speech recognition (ASR) for code-switched speech, and answers by collecting 127 peer-reviewed papers and manually annotating each one across four dimensions. Its central finding is that the field is lopsided and unstandardized: Mandarin-English alone covers about 55% of papers, the top three language pairs about 76%, and most of the 38 identified datasets each cover a single language pair, with 77% of studies relying on accessible data. It also finds no common training or evaluation recipe, so results across studies cannot be directly compared, and reports that none of the best-performing models on popular datasets uses zero-shot evaluation. A sympathetic reader should care because code-switching is a global everyday practice and speech recognition is a mass-market technology: if the review is right, the field's bottleneck is not new architectures but accessible datasets and shared benchmarks for under-studied language pairs.

What carries the argument

The machinery is the review protocol itself. The authors queried the Semantic Scholar API for papers matching (ASR | Speech recognition) together with (code-switch* | codeswitch* | code switch* | code-mix* | codemix*), kept only peer-reviewed work describing end-to-end code-switching systems (127 papers), and coded each paper along four annotated dimensions: problem setup and data (languages, datasets, accessibility), model design (monolingual vs multilingual modeling, language identification, text units, architecture, pretrained models, loss, decoding), training and evaluation (augmentation, translation, zero-shot, metrics), and best reported performance. Each paper was annotated by one of five annotators working from agreed dimensions in scheduled sessions. That grid is what turns heterogeneous papers into comparable statistics like the ~55% and ~77% figures, and every conclusion in the paper flows through it.

What would settle it

Re-run the collection through a second scholarly literature index with the query expanded by terms like 'mixed-language,' 'language mixing,' and 'code-alternation,' and have two annotators independently code a random sample of 30 papers. If the Mandarin-English share moves by more than a few percentage points, or if agreement on core fields (languages covered, dataset accessibility, model type) is poor, the review's headline statistics would need revision, even if the qualitative picture of concentration and fragmentation stands.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a measured snapshot of a young field. From 378 candidate papers retrieved through the Semantic Scholar API, filtering to peer-reviewed venues and end-to-end systems—neural ASR trained jointly in a single computational graph—leaves 127 papers published between 2018 and 2024, over half of them since 2022. Manual annotation across problem setup, model design, training and evaluation, and performance yields three headline facts. First, research concentrates in a few pairs: Mandarin-English is covered by ~55% of papers; adding Hindi-English and Arabic-English reaches ~76%, and momentum tracks the availability of accessible datasets such as SEAME (~24% of papers) and the ASRU 2019 challenge (~11%). Second, ~77% of papers use accessible datasets while the rest rely on proprietary or unspecified data, and the Japanese-English cluster (~5%) has no public dedicated dataset at all, leaning on synthetic speech built from a machine-translation corpus. Third, no experimental standard exists: roughly 60% of studies add monolingual data, more than a third use multilingual modeling, ~47% use encoder-decoder architectures, ~33% incorporate language identification, ~45% an external language model, ~30% data augmentation, and best reported results on different datasets share no common recipe. The authors conclude that efforts are 'mostly sporadic, with no consistent benchmarking or clarity on directions for future research,' and note that no state-of-the-art result on the popular datasets comes from zero-shot evaluation, which they read as evidence that dedicated code-switched training data remains essential.

Load-bearing premise

The headline numbers rest on the assumption that the search query plus the peer-review filter captured essentially the whole universe of end-to-end code-switching ASR work, and that single-annotator coding is accurate enough to support figures like 55%, 76%, and 77%.

Editorial extensions

If this is right

  • Dataset availability, not linguistic need, drives the research agenda: the Mandarin-English cluster grew around SEAME and the ASRU 2019 challenge, so creating one accessible corpus for an under-studied pair is the highest-leverage intervention the data support.
  • Because there is no consistent benchmarking, positive findings from earlier studies have not been replicated on newer datasets; reported gains should be treated as provisional until a shared evaluation framework appears.
  • Dedicated code-switched training data remains necessary: no state-of-the-art system in the surveyed tables relies on zero-shot evaluation, so multilingual pretraining alone does not substitute for CS data.
  • Evaluation itself is unsettled: plain WER penalizes mixed-script output, which is why mixed metrics (MER, TER) and transliteration-aware metrics (toWER, poWER, PolyWER) are appearing; standardizing among them is part of the open problem.
  • The field is still young and accelerating—over half the surveyed papers appeared since 2022—so the window for establishing standardized benchmarks is open now.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The census itself is shaped by its search terms: work framed as 'mixed-language,' 'language mixing,' or dialectal alternation without code-switch/code-mix vocabulary would be invisible to the query, so the true literature is probably larger and more linguistically diverse than 127 papers; the concentration percentages could shift if that framing gap were closed.
  • A quantitative extension the authors leave implicit: measuring citation continuity—whether later papers on the same dataset cite and improve on earlier baselines—would convert the 'sporadic efforts' claim from an impression into a number, and the paper's own reference table makes this computation straightforward.
  • The paper's logic supports a testable, supply-side prediction: releasing a public Japanese-English or Indonesian-English corpus of roughly SEAME's scale would plausibly generate a visible publication cluster within two years; future bibliographies could confirm or refute this.
  • Read with the fairness discussion, the findings imply a concrete allocation rule rather than a research direction: directing benchmark and dataset funding at the roughly thirty language pairs that each appear in a single paper would rebalance the field more than further architectural innovation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 9 minor

Summary. The paper presents a systematic literature review of end-to-end (E2E) automatic speech recognition (ASR) for code-switching (CS). The authors query the Semantic Scholar API with a documented boolean query, retain 127 peer-reviewed papers, and manually annotate them across dimensions such as language pairs, datasets, model architecture, decoding strategy, and evaluation metrics. The central empirical claims are that research is heavily concentrated in a few language pairs (Mandarin–English ~55%, top-3 pairs ~76%), that ~77% of papers use accessible datasets, and that the field lacks standardized benchmarking and evaluation practices. The paper also discusses challenges and opportunities around data scarcity, coverage disparities, evaluation, and reproducibility.

Significance. If the quantitative findings are reliable, this review fills a real gap: it is the first systematic survey focused specifically on E2E ASR for code-switching, and it provides a DOI-level enumeration of all 127 surveyed papers (Table 7), a documented search query, and a transparent annotation scheme. The qualitative picture — that Mandarin–English dominates because of dataset availability and that evaluation is fragmented — is plausible and useful for guiding future research. The paper also makes honest statements about its own coding assumptions (e.g., Section 4.6, the greedy-decoding assumption), which is a strength. However, the headline percentages are load-bearing, and they rest on two methodological premises: search-query recall and single-annotator coding reliability. These premises are not adequately addressed in the manuscript, which limits the strength of the quantitative claims even though the overall qualitative conclusions may well survive.

major comments (3)
  1. [Section 2 (Data Collection) and Section 3] The search query is limited to variants of 'code-switch*' and 'code-mix*'. Work that studies the same phenomenon under alternative terminology — 'language mixing', 'mixed-language speech', 'language switching', or 'code-alternation' — would be omitted unless those terms co-occur with 'code-switch' in the indexed fields. Because the central quantitative claims in Section 3 (Mandarin–English ~55%, top-3 ~76%, accessible-dataset share ~77%) are computed from the 127-paper sample, the review needs to assess recall, for example by running a broader query and reporting how the distribution shifts, or by explicitly identifying known relevant papers that the query missed. Without such a sensitivity check, the headline statistics are not robust to plausible alternative terminologies.
  2. [Section 2 (Annotation)] Each paper was annotated by one annotator, and no inter-annotator agreement (IAA) is reported. The review's conclusions are expressed as percentages over categorical codes (language pair, dataset accessibility, architecture class, etc.), so the reliability of those codes is load-bearing. The authors should report IAA on a subsample, provide dual coding for the attributes that feed the central statistics, or otherwise justify the consistency of single-annotator coding across five annotators. Without this, the precision of numbers like 55% and 76% is unverified.
  3. [Section 2 and Table 7] The inclusion criterion 'published in peer-reviewed venues' is asserted but not operationalized, and Table 7's note says the year corresponds to the earliest available version, 'which can be a preprint'. Some entries in the reference list are clearly preprint-form (CoRR), and the manuscript does not explain how venue status was verified or whether preprint-only items were included. Since the 127-paper corpus is the basis for all of the review's statistics, the paper should clarify the verification procedure and state explicitly whether preprints were admitted or excluded.
minor comments (9)
  1. [Section 3.2] 'ITTG-HingCos' should be 'IITG-HingCos' to match the reference and Table 3.
  2. [Section 4.3] 'languauge independent vocabulary' contains a typo ('languauge' should be 'language').
  3. [Section 7] 'with the exception of of a small set of languages' has a duplicated 'of'.
  4. [Section 4.1] The phrase 'the combined system is retained using monolingual data' should read 'retrained'.
  5. [Section 4.6] The assumption that unmentioned decoding strategies are greedy is stated transparently, but only seven papers explicitly mention greedy decoding; the review would be stronger if it reported the sensitivity of any decoding-related conclusions to this assumption.
  6. [Section 2] A PRISMA-style flow diagram (378 retrieved, exclusions by reason) would improve reproducibility and make the filtering process easier to verify.
  7. [Section 5.2] The phrase 'or to used TTS synthesis' is grammatically incomplete; it should read 'or to use TTS synthesis'.
  8. [References] Several reference entries contain OCR-style spacing errors, e.g., 'V oice' in Grand View Research, 'V ancouver' in Zhang et al. (2018), and 'V enice' in Zhu et al. (2017).
  9. [Table 7] The entry 'Kilkarn' should be 'Killkan' to match the cited dataset name in the reference list.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the survey's statistics are computed from its annotated corpus, not derived from its conclusions or from self-cited prior work.

full rationale

This paper is a systematic literature review; it does not propose a model, fit parameters, or derive a result from assumptions that encode the conclusion. The central claims—that Mandarin-English accounts for roughly 55% of papers, that the top three language pairs account for about 76%, and that evaluation is non-standardized—are computed directly from the 127 annotated papers retrieved by the stated Semantic Scholar query. The annotation scheme, language-pair coding, and dataset tables are descriptive aggregations of the corpus, so no quantity is assumed into existence. The only self-citations (Mixat, PolyWER, and related prior work by the authors) appear as surveyed data points or as cited context, and the review's conclusions do not depend on the correctness of those prior results. The skeptic's concern about the search query omitting papers using alternative terminology such as 'language mixing' or 'mixed-language' is a legitimate validity risk that could shift the reported percentages, but it is a sampling/completeness issue, not circularity: the statistics would still be computed from whatever corpus the query defines. Similarly, the absence of inter-annotator agreement is a reliability limitation, not a circular derivation. No load-bearing step reduces to the paper's own inputs by definition, and no fitted parameter is renamed as a prediction. Under the review's criteria, the honest finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The review's conclusions rest on the completeness of the literature sample and on the reliability of single-annotator coding. No free parameters or invented entities enter the analysis.

assumptions (4)
  • domain assumption The Semantic Scholar API query with the stated code-switch/code-mix terms retrieves the relevant universe of peer-reviewed E2E ASR code-switching papers.
    Section 2: the query returned 378 papers, filtered to 127 after requiring peer-reviewed venues and E2E systems. Unindexed or differently-worded work is excluded by construction.
  • ad hoc to paper Each paper can be reliably assigned to the annotation attributes by one annotator after consensus sessions.
    Section 2: papers were annotated by five annotators, but each paper by one annotator, and no inter-annotator agreement is reported.
  • ad hoc to paper Most unmentioned decoding strategies are greedy.
    Section 4.6: 'As greedy decoding is the simplest strategy and is commonly used in end-to-end models, we assume that most papers follow this strategy.' This assumption directly shapes the decoding-strategy statistics.
  • domain assumption Standard and mixed error metrics (WER, CER, MER, TER) are comparable enough across papers for 'best result' comparisons.
    Section 5.4 and Table 4 aggregate best reported results across heterogeneous datasets and metrics; the authors acknowledge methodological inconsistency in Section 7 but still use these numbers for their best-model summary.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review." pith.science (2026). https://pith.science/paper/PY2VMWUX

@misc{pith2026250707741,
  author       = {Pith},
  title        = {Pith review of: Code-Switching in End-to-End Automatic Speech Recognition: A Systematic Literature Review},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PY2VMWUX}},
  note         = {Machine review of arXiv:2507.07741}
}
read the original abstract

Motivated by a growing research interest into automatic speech recognition (ASR), and the growing body of work for languages in which code-switching (CS) often occurs, we present a systematic literature review of code-switching in end-to-end ASR models. We collect and manually annotate papers published in peer reviewed venues. We document the languages considered, datasets, metrics, model choices, and performance, and present a discussion of challenges in end-to-end ASR for code-switching. Our analysis thus provides insights on current research efforts and available resources as well as opportunities and gaps to guide future research.

Figures

Figures reproduced from arXiv: 2507.07741 by the authors.

Figure 1
Figure 1. Total number of papers per year, and the [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 14 canonical work pages

  1. [6]

    In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6144–6153, Miami, Florida, USA

    PolyWER: A holistic evaluation framework for code-switched speech recognition. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6144–6153, Miami, Florida, USA. Asso- ciation for Computational Linguistics. Abbas Khosravani, Philip N. Garner, and Alexandros Lararidis. 2021. An evaluation benchmark for auto- matic speech recogn...

  2. [8]

    In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages 3469–3479, Torino, Italia

    Code-mixed text augmentation for Latvian ASR. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , pages 3469–3479, Torino, Italia. ELRA and ICCL. Mari Ganesh Kumar, Jom Kuriakose, Anand Thya- gachandran, Ashish Seth, Lodagala Durga Prasad, Saish Jaiswal, Anusha Prak...

  3. [455]

    Charles A

    IEEE. Charles A. Ferguson. 1959. Diglossia. WORD, 15(2):325–340. Grand View Research. 2024. V oice and speech recognition market size, share & trends analysis report by function (speech recognition, voice recognition), by technology, by vertical, by region, and segment forecasts, 2024 - 2030. https://www. grandviewresearch.com/industry-analysis/ voice-rec...

  4. [816]

    Gwantae Kim, Bokyeung Lee, Donghyeon Kim, and Hanseok Ko

    IEEE. Gwantae Kim, Bokyeung Lee, Donghyeon Kim, and Hanseok Ko. 2024. Gated low-rank adaptation for personalized code-switching automatic speech recog- nition on the low-spec devices. InIEEE International Conference on Acoustics, Speech, and Signal Process- ing, ICASSP 2024 - Workshops, Seoul, Republic of Korea, April 14-19, 2024, pages 760–764. IEEE. Mar...

  5. [2014]

    Building Bilingual Corpora , pages 93–110

    5. Building Bilingual Corpora , pages 93–110. Multilingual Matters, Bristol, Blue Ridge Summit. Kunal Dhawan, Ganji Sreeram, Kumar Priyadarshi, and Rohit Sinha. 2020. Investigating target set reduction for end-to-end speech recognition of Hindi-English code-switching data. In 2020 National Conference on Communications, NCC 2020, Kharagpur , India, Februar...

  6. [2020]

    It’s Kind of Like Code-Switching

    ArzEn: A speech corpus for code-switched Egyptian Arabic-English. In Proceedings of The 12th Language Resources and Evaluation Confer- ence, LREC 2020, Marseille, France, May 11-16, 2020, pages 4237–4246. European Language Re- sources Association. Christina N. Harrington, Radhika Garg, Amanda T. Woodward, and Dimitri Williams. 2022. “It’s Kind of Like Cod...

  7. [2022]

    In Interspeech 2022, pages 3924–3928

    Language-specific characteristic assistance for code-switching speech recognition. In Interspeech 2022, pages 3924–3928. Ganji Sreeram, Kunal Dhawan, and Rohit Sinha. 2019. IITG-HingCoS corpus: A Hinglish code-switching database for automatic speech recognition. Speech Commun., 110:76–89. Ganji Sreeram and Rohit Sinha. 2020. Exploration of end-to-end fram...

  8. [2023]

    In International Conference on Electri- cal Engineering and Informatics, ICEEI 2023, Ban- dung, Indonesia, October 10-11, 2023 , pages 1–6

    Handling of Indonesian-English codeswitch- ing speech in end-to-end indonesian speech recogni- tion system using connectionist temporal classifica- tion model. In International Conference on Electri- cal Engineering and Informatics, ICEEI 2023, Ban- dung, Indonesia, October 10-11, 2023 , pages 1–6. IEEE. Hardik B. Sailor, Kiran Praveen, Vikas Agrawal, Abh...

Show all 15 references
  1. [2024]

    ¿ te vienes? sure!

    "¿ te vienes? sure!" joint fine-tuning of lan- guage detection and transcription improves automatic recognition of code-switching speech. In Proceed- ings of the 6th ACM Conference on Conversational User Interfaces, pages 1–7. Chris Hokamp and Qun Liu. 2017. Lexically con- str...

  2. [2251]

    IEEE Computer Society. A Lists of Datasets, Languages, Metrics and Papers Language Pair ISO-3 Code # Papers Dataset Breakdown Mandarin–English zho-eng 70 ( , 32) ( , 15) ( , 6) ( , 5) ( , 2) ( , 1) ( , 1) ( , 1) ( , 1) ( , 11) Hindi–English hin-eng 15 ( , 5) ( , 3) ( , 1) ( , ...

  3. [2460]

    Hiroshi Seki, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux, and John R

    ISCA. Hiroshi Seki, Takaaki Hori, Shinji Watanabe, Jonathan Le Roux, and John R. Hershey. 2019. End-to-end multilingual multi-speaker speech recognition. In 20th Annual Conference of the International Speech Communication Association, Interspeech 2019, Graz, Austria, September...

  4. [5079]

    Brian Yan, Matthew Wiesner, Ondrej Klejch, Preethi Jyothi, and Shinji Watanabe

    European Language Resources Association. Brian Yan, Matthew Wiesner, Ondrej Klejch, Preethi Jyothi, and Shinji Watanabe. 2023. Towards zero- shot code-switched speech recognition. In IEEE In- ternational Conference on Acoustics, Speech and Sig- nal Processing ICASSP 2023, Rhod...

  5. [6243]

    Cao Hong Nga, Duc-Quang Vu, Huong Hoang Lu- ong, Chien-Lin Huang, and Jia-Ching Wang

    IEEE. Cao Hong Nga, Duc-Quang Vu, Huong Hoang Lu- ong, Chien-Lin Huang, and Jia-Ching Wang. 2023. Cyclic transfer learning for Mandarin-English code- switching speech recognition. IEEE Signal Process. Lett., 30:1387–1391. Tolulope Ogunremi, Christopher Manning, and Dan Ju- raf...

  6. [7268]

    Yizhou Lu, Mingkun Huang, Hao Li, Jiaqi Guo, and Yanmin Qian

    European Language Resources Association. Yizhou Lu, Mingkun Huang, Hao Li, Jiaqi Guo, and Yanmin Qian. 2020. Bi-encoder transformer network for Mandarin-English code-switching speech recog- nition using mixture of experts. In 21st Annual Con- ference of the International Speec...

  7. [7893]

    Abigail Z

    IEEE. Abigail Z. Jacobs and Hanna Wallach. 2021. Mea- surement and Fairness. In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 375–385, Virtual Event Canada. ACM. Palash Jain and Anirban Bhowmick. 2024. Vitb-hebic: A bilingual corpu...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.