REVIEW 1 major objections 4 minor 31 references
CAFE A Novel Code switching Dataset for Algerian Dialect French and English
T0 review · 1 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read The paper introduces CAFE, presented as the first publicly available speech dataset for spontaneous Algerian dialect–French–English code-switching, and argues that a preprocessing pipeline plus advanced Whisper decoding lowers the Mixed…
desk verdict A genuinely novel Algerian code-switching corpus, but the manual references and the missing release make the headline benchmark numbers fragile. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing resource is CAFE-small: 170 chunks, 2 hours 36 minutes, 35 speakers, annotated to the transcription guidelines of a prior Arabic-English spoken corpus, covering dialect level (L0-L4), code-switching points, overlap, and events such as noise and laughter, with a mean Code Mixing Index of 0.254 and code-switching rates measured by CESAR, a metric that scores how much a corpus departs from a reference language. The analysis tool is Whisper large-v3, a widely used multilingual ASR model, used with its temperature-controlled greedy decoding and fallback strategy at temperature 0.2. The preprocessing layer uses a speaker-diarization model to cut non-speech segments longer than 0.4 seconds while leaving overlapped speech in place, and the same pipeline produced the 34.58-hour CAFE-large pseudo-labeled set.
What would settle it
Take a random sample of CAFE-small chunks, have at least two independent transcribers fluent in Algerian Darja, French, and English retranscribe them from scratch, and measure agreement against the released annotations; low agreement would show the ground truth is too unstable to support the reported MER, CER, and WER. Alternatively, run the same Whisper-plus-preprocessing pipeline on recordings from other Algerian sources with independent references; a large jump in error rates would indicate the single-podcast source, not the pipeline, drives the reported results.
Extended reading notes
Core claim
The paper's central claim is that code-switched Algerian dialect, French, and English speech is now available as a public resource, and that the main gains in recognizing it come from decoding and preprocessing rather than from language prompting. On CAFE-small, the authors report that a prompting-based Whisper variant with the large-v2 model gives MER/CER of 0.73, that switching to large-v3 with bilingual and multilingual prompts gives 0.64/0.66, and that Whisper's own implementation with temperature 0.2 plus preprocessing reaches MER 0.310, CER 0.329, and WER 0.538. They also report that language token prompts have no measurable effect, that keeping overlapping speech is better than deleting it, and that trimming non-speech segments at chunk starts repairs Whisper's tendency to hallucinate or emit too few words.
Load-bearing premise
The benchmark error rates rest on the assumption that the authors' own manual transcriptions of CAFE-small are reliable ground truth, but no inter-annotator agreement is reported and the audio comes from one podcast channel with mostly male voices, so noisy or unrepresentative references would make the reported error rates and broad dialectal claims untrustworthy.
Editorial extensions
If this is right
- The released CAFE-small annotations give the community a public benchmark for Algerian-French-English ASR with code-switch, overlap, and event labels.
- The 34.58 hours of pseudo labels can expand training data for Algerian code-switched ASR, provided the manual review the paper describes is completed.
- For this kind of spontaneous multilingual speech, decoding strategy matters more than language prompting.
- Keeping overlapping speech, and only trimming long non-speech segments, is the preprocessing choice that lowers error rates in the paper's experiments.
Reading between the lines
- The paper leaves open whether the dataset's dialectal breadth is real, since the source is a single podcast and the speaker pool is mostly male; collecting from other Algerian channels would test that claim.
- The paper does not report inter-annotator agreement, so the published MER, CER, and WER values should be read as provisional until independent transcription agreement is measured.
- The temperature-0.2 result suggests slightly stochastic decoding helps on noisy code-switched speech, a rule that could be tested on Tunisian or Moroccan French-code-switched ASR.
- If CAFE-large pseudo labels are used for training, Whisper's own systematic errors could be baked into the 'ground truth,' making the planned manual review essential.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents CAFE, a new code-switching speech corpus for Algerian dialect (Darja), French, and English, collected from the Gusra Podcast YouTube channel. It contains about 37 hours of speech, with a 2-hour-36-minute manually annotated subset (CAFE-small) and roughly 34.58 hours of pseudo-labeled data (CAFE-large). The authors describe a processing pipeline using pydub for segmentation, pyannote for diarization and overlap removal, and Whisper large-v3 for pseudo-labeling. They benchmark Whisper-based systems with and without prompting and preprocessing, and report that a preprocessing pipeline plus temperature 0.2 improves performance to a Mixed Error Rate of 0.310, Character Error Rate of 0.329, and Word Error Rate of 0.538 on CAFE-small.
Significance. The resource addresses a genuine gap: spontaneous, code-switched Algerian dialect speech with French and English is scarce, and the manuscript gives concrete pipeline details and a useful failure analysis. The corpus could be valuable for ASR and dialectal Arabic research if the annotations and data are validated and released. The paper does not ship verifiable data or code in the preprint, and the benchmark numbers rest on author-produced references with no inter-annotator reliability measure and on a temperature selected on the same evaluation set. These issues are fixable but currently limit how much weight the empirical claims can carry.
major comments (1)
- [Section 4.4, Tables 2 and 12] The manual reference transcriptions are the foundation for every reported error rate, but Section 4.4 describes them as post-editing of Whisper output by the authors and reports no inter-annotator agreement, no cross-version consistency check between the two annotation protocols in Section 4.4.1 and 4.4.2, and no independent verification. Moreover, Table 2 shows that CAFE-small contains only 35 voices (4 female) and Section 4.1 identifies a single YouTube channel, Gusra Podcast, as the source, so the abstract's claim that CAFE captures dialectal variation from various parts of Algeria is not evidenced by speaker-region metadata. Please add IAA statistics, a comparison of the two annotation versions, and speaker demographics or regional metadata, or soften the generalization claims.
minor comments (4)
- [Section 1] The duration of CAFE-small is given as "2.52 hours" in Section 1 and as "2 hours and 36 minutes" in the abstract and Table 2; these values differ by about 5 minutes, so please use one figure consistently.
- [Section 3.1] In the bullet list for QASR-Dataset, the citation "[17, 23]" appears to be a typo for "[23]", since [17] is the ZAEBUC-Spoken corpus already listed separately.
- [Section 4.6] The superscript reference in "In our study and analysis6" is not expanded in the manuscript; either provide the footnote or remove the marker.
- [Tables 2 and 12] The speaker counts in CAFE-small (35 voices; 4 female, 31 male) and CAFE-large (90 voices; 18 female, 70 male) do not obviously support the abstract's claim of broad sociolinguistic coverage, so please add an explicit discussion of the distributional limits.
Circularity Check
The headline benchmark numbers are partly self-selected: temperature 0.2 is chosen on CAFE-small by minimizing MER, then the same selected value is reported as the pipeline's best MER/CER/WER.
-
fitted input called prediction
[Section 5.2 (Table 7/8), Section 6.2, Section 6.3]
"These calculations were performed for temperature values of 0 (default) and 0.2 (the optimal temperature for CAFE-small based on the MER metric). ... Then we started the evaluation of WhisperOriginal with different temperature values for each chunk and selected the best value based on the lowest MER value."
The reported best MER 0.310 and CER 0.329 for Preprocessing-V2 are obtained after sweeping temperature on the same CAFE-small evaluation set and picking the value that minimizes MER. Since the temperature is selected using the evaluation metric on the evaluation set, the resulting numbers are the minima of the temperature grid by construction, not an independent test of the pipeline's advantage. The paper then presents these selected values as evidence that the preprocessing pipeline and decoding techniques improve ASR performance, and Section 7 applies the same self-tuned temperature (0.2) to produce CAFE-large pseudo-labels.
full rationale
The CAFE dataset resource itself is not circular: the authors collected spontaneous YouTube audio, segmented it, post-edited Whisper transcriptions into manual references, and released statistics. That is legitimate dataset construction, though the lack of inter-annotator agreement is a reliability risk, not a circularity step. The main circularity is confined to the ASR benchmark story. The paper tunes temperature on CAFE-small by minimizing MER and then reports the numbers at that temperature as the achieved performance (Table 7/8 and the abstract's MER 0.310 / CER 0.329 / WER 0.538). This is hyperparameter selection on the test set presented as a benchmarking result, which makes the headline improvement partially forced by construction. No load-bearing self-citation or uniqueness-import issue exists: citations to prior work by one of the authors (e.g., QASR) are contextual only, and ZAEBUC guidelines are external. The pseudo-labeling of CAFE-large is an application, not a circular evaluation. Overall, the dataset contribution stands, but the reported benchmark gains should be read as a selected optimum rather than an independent prediction; hence score 6.
Assumptions & free parameters
free parameters (3)
- Chunking thresholds for pydub split-on-silence =
min_silence_len=1000 ms; silence_thresh=-45 dB; target chunk 25-60 s; fallback 15-120 s; max 10 chunks
- Preprocessing-V2 non-speech removal threshold =
0.4 seconds
- ASR decoding temperature =
0.2
assumptions (4)
- domain assumption Audio from the Gusra Podcast is representative of Algerian dialectal variation and sociolinguistic contexts.
- domain assumption ZAEBUC-Spoken annotation guidelines transfer to Algerian-French-English code-switching.
- domain assumption Whisper large-v3 and pyannote models produce acceptable pseudo-labels for CAFE-large.
- ad hoc to paper Temperature 0.2, optimal on CAFE-small, transfers to CAFE-large.
Cite this review
Pith. "Pith review of CAFE A Novel Code switching Dataset for Algerian Dialect French and English." pith.science (2026). https://pith.science/paper/7E5VY2CD
@misc{pith2026241113424,
author = {Pith},
title = {Pith review of: CAFE A Novel Code switching Dataset for Algerian Dialect French and English},
year = {2026},
howpublished = {\url{https://pith.science/paper/7E5VY2CD}},
note = {Machine review of arXiv:2411.13424}
}
read the original abstract
The paper introduces and publicly releases (Data download link available after acceptance) CAFE -- the first Code-switching dataset between Algerian dialect, French, and english languages. The CAFE speech data is unique for (a) its spontaneous speaking style in vivo human-human conversation capturing phenomena like code-switching and overlapping speech, (b) addresses distinct linguistic challenges in North African Arabic dialect; (c) the CAFE captures dialectal variations from various parts of Algeria within different sociolinguistic contexts. CAFE data contains approximately 37 hours of speech, with a subset, CAFE-small, of 2 hours and 36 minutes released with manual human annotation including speech segmentation, transcription, explicit annotation of code-switching points, overlapping speech, and other events such as noises, and laughter among others. The rest approximately 34.58 hours contain pseudo label transcriptions. In addition to the data release, the paper also highlighted the challenges of using state-of-the-art Automatic Speech Recognition (ASR) models such as Whisper large-v2,3 and PromptingWhisper to handle such content. Following, we benchmark CAFE data with the aforementioned Whisper models and show how well-designed data processing pipelines and advanced decoding techniques can improve the ASR performance in terms of Mixed Error Rate (MER) of 0.310, Character Error Rate (CER) of 0.329 and Word Error Rate (WER) of 0.538.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
Kheireddine Abainia. 2019. DZDC12: a new multipurpose parallel Algerian Arabizi–French code-switched corpus. Language Resources and Evaluation 54, 2 (April 2019), 419–455.https://doi.org/10.1007/s10579-019-09454-8
-
[3]
Karima Abidi and Kamel Smaïli. 2022. CESAR: A new metric to measure the level of code-switching in corpora-Application to Maghrebian dialects. In Intelligent Systems and Applications: Proceedings of the 2021 Intelligent Systems Conference (IntelliSys) Volume 2 . Springer, 793–803
work page 2022
-
[4]
Bobbi Aditya, Mahdin Rohmatillah, Liang-Hsuan Tai, and Jen-Tzung Chien. 2024. Attention-Guided Adaptation for Code-Switching Speech Recognition. In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE. https://doi.org/10. 1109/icassp48485.2024.10446258
arXiv 2024
-
[5]
Ahmed Ali, Stephan Vogel, and Steve Renals. 2017. Speech recognition challenge in the wild: Arabic MGB-3. In2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE. https://doi.org/10.1109/asru.2017.8268952
arXiv 2017
-
[6]
Djegdjiga Amazouz, Martine Adda-Decker, and Lori Lamel. 2017. Addressing Code-Switching in French/Algerian Arabic Speech. InInterspeech
work page 2017
-
[7]
Team Audacity. 2017. Audacity.The name audacity (R) is a registered trademark of dominic mazzoni retrieved from http://audacity. sourceforge. net (2017)
work page 2017
-
[8]
Abdelhakim Benali, Mohamed Hédi Maaloul, and Lamia Hadrich Belguith. 2023. Automatic Processing of Algerian Dialect: Corpus Construction and Segmentation. SN Computer Science 4, 5 (Aug. 2023).https://doi.org/10.1007/s42979-023-02097-1
-
[9]
Abdel Hakim Benali, Mohamed Hedi Maaloul, and Lamia Hadrich Belguith. 2018. Identification of the Algiers Dialect Using the Linguistic Rules. In 2018 JCCO Joint International Conference on ICT in Education and Training, International Conference on Computing in Arabic, and International Conference on Geocomputing (JCCO: TICET-ICCA-GECO). IEEE. https://doi....
Show all 31 references
-
[10]
Soumia Bougrine, Aicha Chorana, Abdallah Lakhdari, and Hadda Cherroun. 2017. Toward a Web-based Speech Corpus for Algerian Dialectal Arabic Varieties. In Proceedings of the Third Arabic Natural Language Processing Workshop . Association for Computational Linguistics. https: //...
2017 doi
-
[11]
Hervé Bredin. 2023. pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe. InINTERSPEECH 2023. ISCA. https: //doi.org/10.21437/interspeech.2023-105
2023 doi
-
[12]
Hervé Bredin and Antoine Laurent. 2021. End-To-End Speaker Segmentation for Overlap-Aware Resegmentation. InInterspeech 2021. ISCA. https: //doi.org/10.21437/interspeech.2021-560
2021 doi
-
[13]
Pyannote.Audio:NeuralBuildingBlocksforSpeakerDiarization.In ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
Herve Bredin, Ruiqing Yin, Juan Manuel Coria, Gregory Gelly, Pavel Korshunov, Marvin Lavechin, Diego Fustes, Hadrien Titeux, Wassim Bouaziz, andMarie-PhilippeGill.2020. Pyannote.Audio:NeuralBuildingBlocksforSpeakerDiarization.In ICASSP 2020 - 2020 IEEE International Conference...
2020
-
[14]
Shammur Absar Chowdhury, Amir Hussein, Ahmed Abdelali, and Ahmed Ali. 2021. Towards one model to rule all: Multilingual strategy for dialectal code-switching Arabic ASR.arXiv preprint arXiv:2105.14779 (2021)
2021 arXiv
-
[15]
Shammur A Chowdhury, Younes Samih, Mohamed Eldesouki, and Ahmed Ali. 2020. Effects of Dialectal Code-Switching on Speech Modules: A Study Using Egyptian Arabic Broadcast Speech.. InInterspeech. 2382–2386
2020
-
[16]
Manraj Singh Grover, Pakhi Bamdev, Ratin Kumar Brala, Yaman Kumar, Mika Hama, and Rajiv Ratn Shah. 2020. audino: A modern annotation tool for audio and speech.arXiv preprint arXiv:2006.05236 (2020)
2020 arXiv
-
[17]
Injy Hamed, Fadhl Eryani, David Palfreyman, and Nizar Habash. 2024. ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus. arXiv preprint arXiv:2403.18182 (2024)
2024 arXiv
-
[18]
Injy Hamed, Nizar Habash, Slim Abdennadher, and Ngoc Thang Vu. 2022. ArzEn-ST: A Three-way Speech Translation Corpus for Code-Switched Egyptian Arabic-English. InProceedings of the The Seventh Arabic Natural Language Processing Workshop (W ANLP). Association for Computational ...
2022 doi
-
[19]
Harrat, K
S. Harrat, K. Meftouh, M. Abbas, and K. Smaili. 2014. Building resources for Algerian Arabic dialects. InInterspeech 2014. ISCA. https://doi.org/10. 21437/interspeech.2014-481
2014
-
[20]
Salima Harrat, Karima Meftouhy, Mourad Abbasz, Khaled-Walid Hidoucix, and Kamel Smaili. 2016. An Algerian dialect: Study and Resources. International Journal of Advanced Computer Science and Applications 7, 3 (2016). https://doi.org/10.14569/ijacsa.2016.070353
2016
-
[21]
Manal A. Ismail. 2015. The Sociolinguistic Dimensions of Code-Switching between Arabic and English by Saudis.International Journal of English Linguistics 5, 5 (Oct. 2015).https://doi.org/10.5539/ijel.v5n5p99
2015 doi
-
[22]
Shivam Mehta. 2024. The need for sampling temperature and differences between whisper, GPT-3, and probabilistic model’s temperature.https: //shivammehta25.github.io/posts/temperature-in-language-models-open-ai-whisper-probabilistic-machine-learning/ . Accessed: May 16, 2024
2024
-
[23]
Hamdy Mubarak, Amir Hussein, Shammur Absar Chowdhury, and Ahmed Ali. 2021. QASR: QCRI Aljazeera Speech Resource A Large Scale AnnotatedArabicSpeechCorpus.In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint...
2021 doi
-
[24]
ANovelDatasetforArabicSpeechRecognitionRecordedbyTamazightSpeakers
NourredineOUKAS,TiziriChabi,andTilelliSari.2024. ANovelDatasetforArabicSpeechRecognitionRecordedbyTamazightSpeakers. Authorea Preprints (2024)
2024
-
[25]
Nourredine OUKAS, Samia HABOUSSI, Chafik MAIZA, and Nassim BENSLIMANE. 2024. ArabAlg: A new Dataset for Arabic Speech Command Recognition for Machine Learning Applications.International Journal of Computing and Digital Systems 15, 1 (Feb. 2024), 989–1005.https://doi. org/10.12...
2024 doi
-
[26]
Puyuan Peng, Brian Yan, Shinji Watanabe, and David Harwath. 2023. Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization. InINTERSPEECH 2023. ISCA. https://doi.org/10.21437/interspeech.2023-2032
2023 doi
-
[27]
Alexis Plaquet and Hervé Bredin. 2023. Powerset multi-class cross entropy loss for neural speaker diarization. InINTERSPEECH 2023. ISCA. https://doi.org/10.21437/interspeech.2023-205
2023 doi
-
[28]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. InProceedings of the 40th International Conference on Machine Learning (Honolulu, Hawaii, USA)(ICML’23). JMLR.org, Arti...
2023
-
[29]
Sid Ahmed Selouani and Malika Boudraa. 2010. Algerian Arabic speech database (ALGASD): corpus design and automatic speech recognition application. Arabian Journal for Science and Engineering 35, 2 (2010), 157–166
2010
-
[30]
Yuhang Yang, Yizhou Peng, Xionghu Zhong, Hao Huang, and Eng Siong Chng. 2023. Adapting OpenAI’s Whisper for Speech Recognition on Code-Switch Mandarin-English SEAME and ASRU2019 Datasets.arXiv preprint arXiv:2311.17382 (2023). Manuscript submitted to ACM 24 Lachemat et al
2023 arXiv
-
[31]
Zheng Zhao, Emilio Monti, Jens Lehmann, and Haytham Assem. 2024. Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human...
2024 doi
-
[2017]
https://doi.org/10.21437/interspeech.2017-1373 Manuscript submitted to ACM CAFE: A Novel Code-switching Dataset for Algerian Dialect, French, and English 23
ISCA. https://doi.org/10.21437/interspeech.2017-1373 Manuscript submitted to ACM CAFE: A Novel Code-switching Dataset for Algerian Dialect, French, and English 23
2017 doi
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.