{"id":"707f790d-638f-4ec1-b733-203df07b08de","arxiv_id":"2607.18912","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Full-parameter fine-tuning of a streaming multilingual backbone yields internally evaluated 42.97% WER (Kikuyu) and 33.98% WER (Dholuo), with Kalenjin at 68.74% on a filtered diagnostic that the paper itself flags as non-independent.","lead":"This engineering study adapts NVIDIA's streaming speech-recognition model to three Kenyan languages, reporting 42.97% word error for Kikuyu and 33.98% for Dholuo on internal, repeatedly consulted test sets. It documents in unusual detail how data cleaning, split integrity, and streaming evaluation shaped those results, while explicitly disclaiming any state-of-the-art benchmark claim.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Kikuyu lacks the recorder-disjointness audit performed for Dholuo in §5.3; if Kikuyu train/validation and test share speakers, the 42.97% internal WER is not a valid speaker-independent estimate.","rationale":"The reader's conditional verdict already captures the adaptive-test-set problem. I looked for the most load-bearing issue not fully disclosed by the paper. The Dholuo result is supported by a zero-recorder-overlap audit, but the Kikuyu result — one of the two central completed systems — has no such audit. If Kikuyu test speakers overlap with training, the 42.97% WER can be optimistically biased in a way that the paper's extensive and admirable disclosure about repeated consultation does not address: row-level exclusion from gradients does not prevent speaker-level leakage. This is a concrete, falsifiable gap rather than a stylistic objection. The proposed check uses metadata already present in the project manifests, so it is feasible. If the audit shows zero overlap or the disjoint subset WER is stable, the concern is resolved and the paper's claim stands; if not, the Kikuyu headline needs re-scoping to matched-speaker/internal-protocol performance. I therefore leave the reader's CONDITIONAL verdict unchanged; the concern reinforces it. I mark agreement as partial because the reader's weakest assumption concerns label accuracy and adaptive optimism, whereas my concern is speaker-level split integrity — adjacent but not identical.","tokens_in":36762,"tokens_out":7864,"duration_ms":82146,"concrete_test":"Perform the same recorder-overlap audit for Kikuyu that §5.3 performed for Dholuo: compute the intersection of recorder IDs in the train (114,457), validation (13,667), and test (6,795) manifests. Then re-run the v13 true-streaming evaluation on the subset of test rows whose recorder IDs do not appear in train or validation. Report the size of that subset and the resulting WER/NS-CER; if the disjoint-speaker subset is nonempty and its WER differs materially from 42.97% (e.g., >2 absolute points), the headline should be qualified as speaker-overlapping, not speaker-independent.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim's Kikuyu result (42.97% WER, §10.1) depends on the internal test set being a valid held-out sample. Section 5.3 reports a recorder-overlap audit for Dholuo and finds zero overlap, but the corresponding audit is not reported for Kikuyu. Instead, Kikuyu 'uses the official ANV train, validation, and test splits as the source of truth' with no explicit speaker-disjointness check. Because the Kikuyu validation set (13,667 rows, 19.64 h) was used for checkpoint selection and the test set (6,795 rows, 10.62 h) was repeatedly consulted, any recorder overlap between these splits and the training set would let selection and reporting track speaker-specific acoustic properties rather than language-level adaptation. The paper's own future-work item — freeze a 'speaker-disjoint' one-shot benchmark (§17) — indicates the current Kikuyu sets have not been demonstrated to be speaker-disjoint. Without this control, the headline Kikuyu number is not yet an estimate of performance for unseen speakers, and the 'language-specific system' claim is weaker than it appears.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper documents an engineering study in which NVIDIA Nemotron 3.5 ASR Streaming 0.6B, initialized from a public Kenyan Swahili-adapted checkpoint, is fully fine-tuned into streaming RNN-T systems for Kikuyu, Dholuo, and Kalenjin. The main contributions are a detailed data-curation and normalization pipeline with per-split accounting, staged low-rate continuation, true-streaming evaluation under a cache-aware [56,13] context, checkpoint selection and promotion, and a prototype isolated deployment. Results are reported on internal, adaptively consulted held-out sets: Kikuyu v13 reaches 42.9677% WER and 7.7947% no-space CER; Dholuo v7 reaches 33.9791% WER and 8.1340% no-space CER under a frozen historical label policy. Kalenjin v1-v reaches 68.74% WER on a clean-v3 diagnostic subset, explicitly not an independent generalization estimate. The paper makes no state-of-the-art claim and repeatedly discloses adaptive test-set use, mixed-source validation in one Kalenjin run, and missing ablations.","tokens_in":36934,"tokens_out":9291,"duration_ms":105570,"significance":"The paper's value is its systematic, honest documentation of the full adaptation lifecycle. It provides reproducible corpus ledgers, normalization decisions, negative results (e.g., marker cleanup effect, checkpoint averaging failure), true-streaming evaluation details, artifact checksums, and operational failure modes. It explicitly distinguishes parameter improvement from label correction and avoids overclaiming benchmark status. The main weaknesses are the unverified speaker-disjointness of the Kikuyu test set and the untested value of the Swahili bridge; both are disclosed or partially acknowledged, but the Kikuyu issue affects the headline number. If the missing audit is supplied, or the Kikuyu result is explicitly demoted to a non-speaker-independent diagnostic, the central engineering claims remain defensible.","major_comments":[{"comment":"The Kikuyu headline result (42.9677% WER in Tables 7 and 10) requires the internal test set to be a valid held-out sample. Section 5.3 reports a recorder-overlap audit for Dholuo (zero overlap) but not for Kikuyu; the official ANV splits are simply taken as the source of truth. Because the Kikuyu validation and test sets were also consulted adaptively (§9.4) and the model was fully fine-tuned, recorder/speaker overlap between train and test would let the reported WER track speaker-specific traits rather than language-level adaptation. The paper's own §17 (freeze a speaker-disjoint benchmark) indicates this has not been demonstrated. This is load-bearing for the abstract's and §10.1's Kikuyu number. Please run the same recorder-overlap audit on Kikuyu metadata and report overlap statistics; if metadata is unavailable, revise the claim to state that the Kikuyu result is not a demonstrated","section":"§5.3, §10.1 (Table 7), §17"}],"minor_comments":[{"comment":"The paper repeatedly acknowledges that the advantage of the Kenyan Swahili bridge over direct adaptation from the raw NVIDIA base was not isolated. This is acceptable given the paper's framing, but the title and contribution list emphasize 'from a multilingual streaming ASR backbone' while all runs actually start from the Swahili checkpoint. Clarify the wording or add a raw-base control run in future work as a stated priority.","section":"§4, §16.1"},{"comment":"The Kikuyu v13 description says the best validation WER occurred 'near epoch 65' while the run is described in optimizer steps (70,000 steps). Define epoch size or restate the milestone in steps to avoid ambiguity.","section":"§10.1"},{"comment":"For Kikuyu, state explicitly whether recorder/speaker identifiers exist in the source metadata and whether any audit was attempted. The current text reports the Dholuo audit but leaves the reader unsure whether Kikuyu metadata even supports such an audit.","section":"§5.3"},{"comment":"The header 'Train h' is cryptic; use 'Training hours' or 'Train hours' for clarity. Also consider adding a footnote that Kikuyu and Dholuo rows are adaptively consulted and not externally benchmarked, even though this is stated in the text.","section":"Table 10"},{"comment":"The reproducibility-level discussion is useful; however, the paper's own 'scientific reproducibility' level is not achieved because checkpoints and manifests are private. Since the paper values auditability, consider releasing at minimum the corpus ledgers, normalization unit tests, and evaluation manifests as a concrete step toward that level.","section":"§13.2"}],"recommendation":"major_revision","confidential_remarks":"This is an unusually transparent engineering report. The only blocking issue is the missing Kikuyu recorder-overlap audit; once that is supplied or the Kikuyu claim is explicitly downgraded to a non-speaker-independent diagnostic, I would support acceptance. The bridge ablation, while desirable, is not essential because the paper does not claim causal benefit. The manuscript is long, but the appendices are substantive and valuable for applied ASR venues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — this one is worth knowing about, though not for the headline numbers. It is a genuinely careful engineering report on adapting a streaming RNN-T backbone to Kikuyu, Dholuo, and Kalenjin. The author does the unglamorous work well: corpus-level accounting, split provenance, deterministic normalization, true-streaming evaluation, artifact preservation, and a real attempt to separate model improvement from label correction. The negative findings are the useful part — marker cleanup barely moved Kalenjin WER, checkpoint averaging did not help, and the mixed-source validation bug is disclosed rather than buried.\n\nThe central claims are modest and mostly supported. Kikuyu and Dholuo reach 42.97% and 33.98% internal WER, and the paper repeatedly says these are adaptively consulted internal numbers, not public-benchmark results. That is the right frame. The learning trajectories and the WER/NS-CER gap are coherent, and the metrics are defined cleanly.\n\nThe soft spots are real but mostly disclosed. The Kikuyu test set gets the same official splits as Dholuo, but the zero-recorder-overlap audit is only reported for Dholuo. Without that check, Kikuyu's headline number is not yet an estimate for unseen speakers. The paper's own future-work item on a speaker-disjoint benchmark confirms the gap. Second, the Swahili bridge is the foundation of every run but is never compared to direct adaptation from the NVIDIA base, so the central methodological choice is unvalidated. The author says exactly this in Section 4. Neither flaw sinks the paper because it disclaims external generalization, but both cap what can be claimed.\n\nAlso minor: single trajectories, no seeds or confidence intervals, private artifacts, and a historical Dholuo normalizer that keeps 'cs' as a token — all acknowledged.\n\nWho should read this: practitioners planning low-resource adaptation, and anyone who wants a model of candid reporting. It deserves a serious referee, not a desk reject. For a journal or workshop I'd want the Kikuyu recorder audit added (or the claim softened), the bridge ablation run, and at least one public artifact or manifest release.\n\nI'd give it a conditional accept with those revisions.","headline":"An unusually honest, data-centric adaptation case study for three Kenyan languages; within its explicitly limited scope the results hold up, but the Kikuyu headline number lacks a recorder-disjointness check and the Swahili bridge is never ablated.","tokens_in":37534,"tokens_out":2409,"would_cite":true,"duration_ms":33908,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A multilingual streaming ASR backbone can be adapted into language-specific streaming recognizers for Kikuyu and Dholuo, reaching internal word-error rates of about 43% and 34% through full-parameter fine-tuning and disciplined corpus audit","keywords":["automatic speech recognition","low-resource languages","Kikuyu","Dholuo","Kalenjin","streaming ASR","RNN-Transducer","data curation"],"falsifier":"Run a controlled comparison where the same curated data, seeds, and compute budget are used to fine-tune from the untouched multilingual base versus the Swahili-adapted bridge, and score both on a freshly frozen, speaker-disjoint test set whose references were never consulted during development; if the bridge branch does not beat the base branch, or if both fail to improve over the cleaned labels themselves, the central adaptation claim collapses.","tokens_in":36491,"feed_emoji":"🎙️","tokens_out":4640,"duration_ms":39335,"temperature":0.7,"pith_summary":"The paper's central claim is that a large multilingual streaming speech-recognition model can be adapted into working streaming recognizers for the low-resource Kenyan languages Kikuyu and Dholuo by full-parameter fine-tuning plus rigorous corpus auditing, reaching internal word-error rates of about 43% and 34% while keeping the streaming architecture intact from training to deployment. A third language, Kalenjin, is documented as work in progress (about 69% WER on a filtered diagnostic set) precisely because its failures reveal how label artifacts and evaluation design distort results. The author argues that for such languages, data quality decisions are model decisions: normalizing Unicode, removing annotation markers, and deleting uncertain rows often matter as much as the neural architecture. This matters because it offers a reproducible, auditable route from a multilingual backbone to language-specific speech systems without discarding the low-latency streaming behavior needed for real use. The paper explicitly disclaims state-of-the-art status, since the internal test sets were consulted repeatedly and differ from public benchmarks.","feed_headline":"Kikuyu and Dholuo streaming ASR reach 43% and 34% WER","feed_subtitle":"Cleaning the corpus mattered as much as the model: data audits and true-streaming evaluation drove the gains.","key_machinery":"The central load-bearing component is the cache-aware FastConformer RNN-T transducer with a 128-dimensional one-hot language prompt (sw-KE), which lets the model be fine-tuned for a target language while retaining bounded-future streaming inference. Around this, the method uses full-parameter fine-tuning at a low 5e-7 learning rate in staged continuations, and a data pipeline that performs Unicode normalization, deletes ambiguous rows instead of guessing, and splits corpora by provenance. The companion evaluation tool is a true-streaming evaluator that feeds chunks with carried caches rather than transcribing whole files offline; this catches bugs that offline evaluation would miss. A second","core_discovery":"The discovery is that the adaptation works, and that the engineering discipline around data and evaluation is what makes it work. Starting from a Kenyan Swahili-adapted checkpoint of the Nemotron 3.5 streaming ASR backbone, the author performs full-parameter fine-tuning on curated Kikuyu and Dholuo corpora, preserving the cache-aware FastConformer RNN-T, prompt conditioning, and streaming decoder. On internal held-out sets (excluded from gradient updates but consulted during development), the best Kikuyu checkpoint reaches 42.97% WER and 7.79% no-space character error; the Dholuo final export reaches 33.98% WER and 8.13% no-space character error under a frozen historical label policy. Kalenj","pith_inferences":["The strong WER reduction may partly reflect label correction rather than acoustic model improvement; a frozen external benchmark with independent references is needed to separate the two.","The Swahili bridge is plausible but untested against direct adaptation from the raw base; a matched ablation could quantify whether regional acoustic overlap or mere initialization matters.","The method of deleting uncertain rows is conservative and reversible but risks pruning dialectal or hard-valid speech; a confidence-based curation with native-speaker adjudication is a natural next step.","The no-space CER numbers suggest that much residual error is about orthographic boundary conventions, so a lexical normalization policy shared with downstream applications could make the systems more usable than WER alone implies."],"forward_implications":["If the claim holds, one can adapt a streaming multilingual backbone into language-specific streaming recognizers for low-resource languages without switching to an offline architecture.","Corpus auditing and row deletion can yield measurable gains; a cleaner, smaller label set can outperform a larger noisier one.","True-streaming evaluation must accompany training validation; offline evaluation can conceal state-reset bugs and future-context leakage.","Staged continuation with a fixed small learning rate produces diminishing returns; the paper's trajectory shows where to stop and switch to data or decoding work.","Dholuo and Kikuyu champions are deployable as isolated streaming services with preserved artifacts and private checkpoints."],"fun_headline_variants":["Kikuyu and Dholuo streaming ASR: 43% and 34% WER","Data-centric adaptation yields 43% WER for Kikuyu, 34% for Dholuo","Multilingual backbone tuned for Kenyan languages: WERs 43% and 34%","Streaming ASR for Kikuyu and Dholuo hits 43% and 34% WER"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire result rests on the assumption that the source corpora's transcriptions, after the author's normalization and deletions, are accurate enough that falling word-error rates reflect real model improvement rather than the model merely learning corrected labels.","fun_headline_variants_meta":{"raw":{"variants":["Kikuyu and Dholuo streaming ASR: 43% and 34% WER","Data-centric adaptation yields 43% WER for Kikuyu, 34% for Dholuo","Multilingual backbone tuned for Kenyan languages: WERs 43% and 34%","Streaming ASR for Kikuyu and Dholuo hits 43% and 34% WER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000305,"raw_usage":{"total_tokens":1683,"prompt_tokens":940,"completion_tokens":743,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":684,"completion_tokens_details":{"reasoning_tokens":636}},"tokens_in":684,"tokens_out":743,"duration_ms":6596,"temperature":1.0,"reasoning_tokens":636,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T13:56:23.244893+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled comparison where the same curated data, seeds, and compute budget are used to fine-tune from the untouched multilingual base versus the Swahili-adapted bridge, and score both on a freshly frozen, speaker-disjoint test set whose references were never consulted during development; if the bridge branch does not beat the base branch, or if both fail to improve over the cleaned labels themselves, the central adaptation claim collapses.","supporting_citations":[],"review_version":1}