{"id":"f2ca0a32-8999-4b8b-b545-d54e3314ae14","arxiv_id":"2411.13424","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CAFE is a new spontaneous speech corpus for Algerian dialect, French, and English code-switching, with 2.6 hours manually annotated and a Whisper benchmark reaching MER 0.310.","lead":"This paper introduces CAFE, a roughly 37-hour speech dataset of Algerians switching between the local dialect, French, and English in natural podcast conversations. It also benchmarks Whisper-based speech recognizers on the data, reporting that preprocessing and decoding changes improve error rates from about 0.73 to 0.31, but the dataset is not yet downloadable from the preprint.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth reliability is the load-bearing assumption: CAFE-small's manual transcriptions are author post-edits with no reported inter-annotator agreement, and those same references drive every benchmark number, so annotation noise could dominate the claimed gains.","rationale":"This is a dataset-and-benchmark paper, so the correctness of the CAFE-small transcriptions is the load-bearing condition for both the corpus contribution and the ASR numbers. The paper already contains two manual annotation versions but never compares them, and no external reliability measure is reported. That is not an accusation of bad faith; it is a missing evidence problem. Annotation of spontaneous code-switched dialectal speech is genuinely difficult, and the paper's own examples show ambiguous transcription choices, so the absence of agreement data is not a formality. The proposed check is cheap and decisive: if independent re-annotation agrees closely with the released references, the concern is resolved and the conditional verdict can move toward acceptance. If agreement is poor, the headline improvements and the dataset's usability are both called into question. I do not see an internal contradiction that would justify rejection; the issue is missing verification rather than demonstrated error, so the appropriate verdict remains conditional on the annotation reliability being established.","tokens_in":20264,"tokens_out":7763,"duration_ms":88247,"concrete_test":"Take a random 20-minute subset of CAFE-small and have two independent annotators (not the paper's authors) transcribe it from audio using the ZAEBUC guidelines; compute token-level MER/CER between the two independent transcriptions and between each independent transcription and the released reference. Also compute the same agreement between the paper's two existing manual versions (Section 4.4.1 vs Section 4.4.2). If inter-annotator MER exceeds roughly 0.10, the reference noise is comparable to or larger than the headline 0.023 MER improvement, and the benchmark should be re-reported on a consensus reference.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claims—the CAFE corpus itself and the headline MER 0.310 / CER 0.329 / WER 0.538—are computed against manual references whose construction is described in Section 4.4 as post-editing of Whisper output. Section 4.4.1 and Section 4.4.2 describe two manual versions (raw and ZAEBUC-guideline), yet no cross-version or inter-annotator consistency measure is reported. Table 2 also shows the 2h36m subset has only 35 voices (4 female), and Section 4.1 names the Gusra Podcast as the source, so the abstract's claim of dialectal variation across Algeria is not supported by speaker-region metadata. If the references contain systematic post-editing errors, or if two plausible Algerian transcriptions of the same audio differ by more than a few percent MER, the reported improvement from WhisperOriginal (0.333) to Preprocessing-V2 (0.310) is within annotation noise, and the benchmark comparisons lose their meaning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents CAFE, a new code-switching speech corpus for Algerian dialect (Darja), French, and English, collected from the Gusra Podcast YouTube channel. It contains about 37 hours of speech, with a 2-hour-36-minute manually annotated subset (CAFE-small) and roughly 34.58 hours of pseudo-labeled data (CAFE-large). The authors describe a processing pipeline using pydub for segmentation, pyannote for diarization and overlap removal, and Whisper large-v3 for pseudo-labeling. They benchmark Whisper-based systems with and without prompting and preprocessing, and report that a preprocessing pipeline plus temperature 0.2 improves performance to a Mixed Error Rate of 0.310, Character Error Rate of 0.329, and Word Error Rate of 0.538 on CAFE-small.","tokens_in":20436,"tokens_out":5311,"duration_ms":55407,"significance":"The resource addresses a genuine gap: spontaneous, code-switched Algerian dialect speech with French and English is scarce, and the manuscript gives concrete pipeline details and a useful failure analysis. The corpus could be valuable for ASR and dialectal Arabic research if the annotations and data are validated and released. The paper does not ship verifiable data or code in the preprint, and the benchmark numbers rest on author-produced references with no inter-annotator reliability measure and on a temperature selected on the same evaluation set. These issues are fixable but currently limit how much weight the empirical claims can carry.","major_comments":[{"comment":"The manual reference transcriptions are the foundation for every reported error rate, but Section 4.4 describes them as post-editing of Whisper output by the authors and reports no inter-annotator agreement, no cross-version consistency check between the two annotation protocols in Section 4.4.1 and 4.4.2, and no independent verification. Moreover, Table 2 shows that CAFE-small contains only 35 voices (4 female) and Section 4.1 identifies a single YouTube channel, Gusra Podcast, as the source, so the abstract's claim that CAFE captures dialectal variation from various parts of Algeria is not evidenced by speaker-region metadata. Please add IAA statistics, a comparison of the two annotation versions, and speaker demographics or regional metadata, or soften the generalization claims.","section":"Section 4.4, Tables 2 and 12"}],"minor_comments":[{"comment":"The duration of CAFE-small is given as \"2.52 hours\" in Section 1 and as \"2 hours and 36 minutes\" in the abstract and Table 2; these values differ by about 5 minutes, so please use one figure consistently.","section":"Section 1"},{"comment":"In the bullet list for QASR-Dataset, the citation \"[17, 23]\" appears to be a typo for \"[23]\", since [17] is the ZAEBUC-Spoken corpus already listed separately.","section":"Section 3.1"},{"comment":"The superscript reference in \"In our study and analysis6\" is not expanded in the manuscript; either provide the footnote or remove the marker.","section":"Section 4.6"},{"comment":"The speaker counts in CAFE-small (35 voices; 4 female, 31 male) and CAFE-large (90 voices; 18 female, 70 male) do not obviously support the abstract's claim of broad sociolinguistic coverage, so please add an explicit discussion of the distributional limits.","section":"Tables 2 and 12"}],"recommendation":"major_revision","confidential_remarks":"The manuscript describes a potentially useful resource, but the core artifacts (data and code) are not currently public, the manual references lack reliability measures, and the benchmark improvement is selected on the evaluation set. These are fixable within a revision, so I recommend major revision rather than rejection, provided the authors add the missing validation and make the resource available."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly, this is a legitimate resource paper with a real gap to fill. The CAFE corpus appears to be the first public spontaneous speech corpus combining Algerian Darja, French, and English, with manual annotation of code-switching, overlap, and non-lexical events. I found the dataset construction narrative honest and fairly detailed: pydub segmentation parameters, pyannote diarization, Whisper versions, post-editing protocols, and a useful failure analysis (music at chunk start causing hallucination is a good catch). The benchmarking against several Whisper variants is systematic enough, and the paper is candid about PromptingWhisper not helping.\n\nThe soft spots are real but fixable. First, the actual dataset, code, and annotation website are promised only after acceptance, so the central artifact cannot be inspected. Second, the manual references for CAFE-small are author post-edits of Whisper output with no inter-annotator agreement or cross-version consistency reported. Since every headline number is computed against those references, annotation noise could easily explain the difference between WhisperOriginal's 0.333 MER and Preprocessing-V2's 0.310 MER. That's the load-bearing issue, and the stress-test note is right to put it first. Third, the claim of dialectal variation across Algeria rests on a single podcast channel with 35 voices (4 female) in CAFE-small; that is not enough to support 'various parts of Algeria' without speaker-region metadata. Finally, temperature 0.2 is selected on the same CAFE-small set and then used to report the best numbers; that is in-sample optimization and should be labeled as such.\n\nNone of this sinks the paper. The resource itself is novel, the pipeline write-up is useful for people building similar corpora, and the limitations are acknowledged in passing. But the paper cannot be fully evaluated until the data is public and the annotation reliability is documented. For readers working on Maghrebi ASR or code-switching, this is worth a look once released; I would not cite it yet.\n\nI would send it to peer review rather than desk reject, with the clear expectation that the authors release the artifacts and add annotation reliability measures, or the benchmark claims should be presented only as an in-sample illustration.","headline":"A genuinely novel Algerian code-switching corpus, but the manual references and the missing release make the headline benchmark numbers fragile.","tokens_in":21046,"tokens_out":2652,"would_cite":false,"duration_ms":28619,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces CAFE, presented as the first publicly available speech dataset for spontaneous Algerian dialect–French–English code-switching, and argues that a preprocessing pipeline plus advanced Whisper decoding lowers the Mixed…","keywords":["code-switching","Algerian dialect","Darja","automatic speech recognition","Whisper","CAFE dataset","pseudo-labeling","spontaneous speech"],"falsifier":"Take a random sample of CAFE-small chunks, have at least two independent transcribers fluent in Algerian Darja, French, and English retranscribe them from scratch, and measure agreement against the released annotations; low agreement would show the ground truth is too unstable to support the reported MER, CER, and WER. Alternatively, run the same Whisper-plus-preprocessing pipeline on recordings from other Algerian sources with independent references; a large jump in error rates would indicate the single-podcast source, not the pipeline, drives the reported results.","tokens_in":20028,"feed_emoji":"🎙️","tokens_out":13631,"duration_ms":121605,"temperature":0.7,"pith_summary":"CAFE is introduced as the first speech dataset built around spontaneous code-switching among Algerian dialect (Darja), French, and English, drawn from real podcast conversations. It contains about 37 hours of audio: a hand-annotated subset, CAFE-small, of 2 hours 36 minutes with transcriptions, code-switching marks, overlapping speech, and event labels, plus 34.58 hours of pseudo-labeled speech (automatic transcriptions used as training labels). The paper argues that the way to get usable automatic transcriptions of such data is not language prompting but a preprocessing layer that trims long non-speech segments combined with Whisper large-v3's temperature-controlled fallback decoding, which brings MER down from 0.73 to 0.310, CER to 0.329, and WER to 0.538. This matters because Algerian Darja is an under-resourced dialect, and existing Algerian or Arabic code-switching datasets do not combine this language mix with spontaneous, publicly available speech.","feed_headline":"First open dataset targets Algerian-French-English code-switching","feed_subtitle":"A 2.6-hour hand-annotated benchmark plus 34.6 hours of pseudo labels cover Algerian, French, and English mixing.","key_machinery":"The load-bearing resource is CAFE-small: 170 chunks, 2 hours 36 minutes, 35 speakers, annotated to the transcription guidelines of a prior Arabic-English spoken corpus, covering dialect level (L0-L4), code-switching points, overlap, and events such as noise and laughter, with a mean Code Mixing Index of 0.254 and code-switching rates measured by CESAR, a metric that scores how much a corpus departs from a reference language. The analysis tool is Whisper large-v3, a widely used multilingual ASR model, used with its temperature-controlled greedy decoding and fallback strategy at temperature 0.2. The preprocessing layer uses a speaker-diarization model to cut non-speech segments longer than 0.4 seconds while leaving overlapped speech in place, and the same pipeline produced the 34.58-hour CAFE-large pseudo-labeled set.","core_discovery":"The paper's central claim is that code-switched Algerian dialect, French, and English speech is now available as a public resource, and that the main gains in recognizing it come from decoding and preprocessing rather than from language prompting. On CAFE-small, the authors report that a prompting-based Whisper variant with the large-v2 model gives MER/CER of 0.73, that switching to large-v3 with bilingual and multilingual prompts gives 0.64/0.66, and that Whisper's own implementation with temperature 0.2 plus preprocessing reaches MER 0.310, CER 0.329, and WER 0.538. They also report that language token prompts have no measurable effect, that keeping overlapping speech is better than deleting it, and that trimming non-speech segments at chunk starts repairs Whisper's tendency to hallucinate or emit too few words.","pith_inferences":["The paper leaves open whether the dataset's dialectal breadth is real, since the source is a single podcast and the speaker pool is mostly male; collecting from other Algerian channels would test that claim.","The paper does not report inter-annotator agreement, so the published MER, CER, and WER values should be read as provisional until independent transcription agreement is measured.","The temperature-0.2 result suggests slightly stochastic decoding helps on noisy code-switched speech, a rule that could be tested on Tunisian or Moroccan French-code-switched ASR.","If CAFE-large pseudo labels are used for training, Whisper's own systematic errors could be baked into the 'ground truth,' making the planned manual review essential."],"forward_implications":["The released CAFE-small annotations give the community a public benchmark for Algerian-French-English ASR with code-switch, overlap, and event labels.","The 34.58 hours of pseudo labels can expand training data for Algerian code-switched ASR, provided the manual review the paper describes is completed.","For this kind of spontaneous multilingual speech, decoding strategy matters more than language prompting.","Keeping overlapping speech, and only trimming long non-speech segments, is the preprocessing choice that lowers error rates in the paper's experiments."],"supporting_citations":[{"why":"Supplies the Whisper large-v2 and large-v3 models and the decoding strategies whose performance the paper benchmarks and credits for the main improvement.","marker":"[28]"},{"why":"Provides the prompting-based Whisper variant whose language-token prompting and decoding choices are compared against, and found lacking.","marker":"[26]"},{"why":"Defines the transcription and annotation guidelines used for CAFE-small, enabling direct comparability with a prior multilingual Arabic-English spoken corpus.","marker":"[17]"},{"why":"Provide the speaker-diarization model used in the preprocessing pipeline and the CAFE-large pseudo-labeling framework.","marker":"[11, 27]"},{"why":"The closest existing Algerian Arabic speech corpus, used to show that available resources are read speech without code-switching.","marker":"[29]"},{"why":"The main existing multi-dialect Algerian corpus, used as a contrast because it lacks code-switching annotations and clear annotation statistics.","marker":"[10]"},{"why":"The prior French-Algerian code-switching corpus, used to define CAFE's novelty because it is not public and does not cover English.","marker":"[6]"},{"why":"Earlier work adapting Whisper to code-switched speech, cited for the observation that switching language prompt tokens has little effect.","marker":"[30]"}],"fun_headline_variants":["First open dataset for Algerian-French-English code-switching","Preprocessing beats prompting for Algerian code-switching","CAFE dataset: 37 hours of Algerian-French-English speech","New code-switching dataset for Algerian, French, English"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark error rates rest on the assumption that the authors' own manual transcriptions of CAFE-small are reliable ground truth, but no inter-annotator agreement is reported and the audio comes from one podcast channel with mostly male voices, so noisy or unrepresentative references would make the reported error rates and broad dialectal claims untrustworthy.","fun_headline_variants_meta":{"raw":{"variants":["First open dataset for Algerian-French-English code-switching","Preprocessing beats prompting for Algerian code-switching","CAFE dataset: 37 hours of Algerian-French-English speech","New code-switching dataset for Algerian, French, English"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3143,"prompt_tokens":988,"completion_tokens":2155,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2100}},"tokens_in":604,"tokens_out":2155,"duration_ms":15861,"temperature":1.0,"reasoning_tokens":2100,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T16:25:14.812375+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of CAFE-small chunks, have at least two independent transcribers fluent in Algerian Darja, French, and English retranscribe them from scratch, and measure agreement against the released annotations; low agreement would show the ground truth is too unstable to support the reported MER, CER, and WER. Alternatively, run the same Whisper-plus-preprocessing pipeline on recordings from other Algerian sources with independent references; a large jump in error rates would indicate the single-podcast source, not the pipeline, drives the reported results.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Whisper large-v2 and large-v3 models and the decoding strategies whose performance the paper benchmarks and credits for the main improvement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the prompting-based Whisper variant whose language-token prompting and decoding choices are compared against, and found lacking."},{"cited_title":"ZAEBUC-Spoken: A Multilingual Multidialectal Arabic-English Speech Corpus","cited_arxiv_id":"2403.18182","evidence_quote":"Defines the transcription and annotation guidelines used for CAFE-small, enabling direct comparability with a prior multilingual Arabic-English spoken corpus."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The closest existing Algerian Arabic speech corpus, used to show that available resources are read speech without code-switching."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The main existing multi-dialect Algerian corpus, used as a contrast because it lacks code-switching annotations and clear annotation statistics."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The prior French-Algerian code-switching corpus, used to define CAFE's novelty because it is not public and does not cover English."}],"review_version":1}