{"id":"b20374dc-441a-4515-bd22-4b39b26b1492","arxiv_id":"2506.00267","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"CASPER presents a 102-hour spontaneous English conversation dataset with per-speaker channels, speaker metadata, and baseline ASR and diarization results.","lead":"The paper releases CASPER, a new dataset of about 102 hours of recorded, unscripted conversations between pairs of people who know each other. It also shares the web-based recording pipeline and initial speech recognition and speaker-diarization baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Diarization baselines in §IV.C.2 are scored against WhisperX-derived reference segments, so Table IV may measure agreement with WhisperX rather than with true speaker activity; a manual annotation check should settle whether the reported DER/JER are reliable.","rationale":"The paper is a credible dataset contribution with reproducible collection infrastructure, genuine spontaneous conversational audio, and honest plans for future expansion. The central claim has two parts: (1) CASPER is a large-scale, high-quality spontaneous speech resource, and (2) current ASR and diarization systems perform poorly on it. The ASR evidence is based on 1.95 hours of manually aligned transcripts and is a small but legitimate sample. The diarization evidence, however, rests on a reference built entirely from WhisperX word-level timestamps on per-participant channels. No manual verification of those timestamps is reported, and the paper itself does not state a limitation about this. Since the benchmark section explicitly uses these labels as 'ground truth,' the validity of the DER/JER numbers depends on WhisperX timestamps being accurate enough on spontaneous, overlapping, crosstalk-prone speech. This is the weakest load-bearing assumption in the paper. The reader's weakest-assumption analysis identifies exactly this issue, so I agree. A targeted manual annotation check on a small subset would either confirm the machine reference or show that the reported errors are substantially inflated. Because the concern is localized and fixable, it does not change the reader's conditional verdict: the paper should be accepted only after the diarization reference is validated or its uncertainty acknowledged and quantified.","tokens_in":10192,"tokens_out":3134,"duration_ms":34654,"concrete_test":"Manually annotate a random 5-hour subset of the diarization test set (about 20 recordings) with speaker turn boundaries and overlap regions on the combined single-channel audio, following the same scoring protocol (no collar, overlap included). Compute pyannote.audio-v3.1 and NeMo DER/JER against both the human labels and the paper's WhisperX-derived labels. If the two references yield aggregate DER differences larger than about 2 absolute points, the Table IV numbers are not trustworthy as a measure of diarization difficulty; if they agree within that margin, the machine-labeled reference is adequate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The diarization benchmark's reference labels in §IV.C.2 are not human annotations. For each conversation, speech segments are obtained by running WhisperX on each participant's isolated channel and then mapping word-level timestamps onto the combined single-channel recording. This makes the reported DER/JER in Table IV a measure of agreement with WhisperX's word-level timestamps, not with ground-truth speaker activity. WhisperX forced alignment can produce systematic boundary drift, missed words, and spurious insertions, especially on spontaneous speech with disfluencies, overlapping talk, and crosstalk between the two participant channels. Any such reference error directly inflates DER/JER relative to human-labeled truth. Because the 102-hour diarization results are the paper's main evidence that CASPER is challenging for state-of-the-art diarization systems, the central claim is partly contingent on an unvalidated machine-generated reference. The issue is concrete and fixable: a small manual subset can validate whether WhisperX-derived segments are accurate enough to support the reported error rates.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CASPER, a spontaneously spoken English conversation dataset collected from pairs of acquaintances via a web-based platform with per-participant audio channels. The authors report 200 hours of recorded audio, 158 hours of estimated speech, and a public release of 102 hours with train/test splits, speaker metadata, and partial transcriptions. They describe their collection pipeline, de-identification method, metadata statistics, and baseline experiments: ASR (Whisper-large-v3 and SeamlessM4T-large) on a 1.95-hour manually transcribed subset, and speaker diarization (pyannote.audio v3.1 and NeMo) on the full 102-hour release using a WhisperX-generated reference. Baseline WERs are 0.31 and 0.53; DER ranges from 25.93% (pyannote) to 30.77% (NeMo), with corresponding JER values. The paper argues that these error rates show CASPER is challenging for current systems and valuable for spontaneous-speech research.","tokens_in":10384,"tokens_out":4594,"duration_ms":46520,"significance":"If the dataset and its baselines are reliable, CASPER addresses a real scarcity of large-scale spontaneous conversational speech with a reproducible collection framework, per-speaker channels, and rich self-reported metadata. Releasing the pipeline and the first 102 hours is a concrete contribution to the community. The strongest assets are the scale relative to existing spontaneous-speech resources, the recording of acquainted pairs (which plausibly increases naturalness), and the open availability. However, the paper's evidence that CASPER is a difficult benchmark rests on two evaluation pillars: a small ASR sample and a machine-generated diarization reference. The diarization reference in particular has not been validated against human annotation, so the magnitude of the reported difficulty is uncertain. These issues are fixable with additional analyses and should be addressed before the dataset's benchmark claims are accepted.","major_comments":[{"comment":"The diarization reference labels are generated automatically by WhisperX forced alignment on each participant's isolated channel and then mapped onto the combined recording. This is not a human-annotated ground truth. DER/JER computed against such a reference measure agreement with WhisperX's word-level timestamps, not with true speaker activity. On spontaneous speech with disfluencies, crosstalk, and overlaps, WhisperX alignment is known to exhibit boundary drift, missed words, and spurious insertions, all of which would inflate the reported error rates. Because Table IV is the paper's primary evidence that CASPER is challenging for diarization systems, the central claim is contingent on the accuracy of an unvalidated reference. I request a manual validation subset (e.g., 30–60 minutes stratified across conversations) in which segment boundaries and speaker labels are annotated by humans, allowing computation of reference-quality metrics and corrected DER/JER. Without this, the reported 25.93–33.87% error rates cannot be interpreted as measuring the difficulty of CASPER.","section":"IV.C.2 (Diarization)"},{"comment":"Table III reports WER on a 1.95-hour sample of the dataset. This is less than 2% of the released audio, and the paper gives no information about how many conversations or speakers are included, how the sample was stratified (or not) across accent, device, or recording conditions, or any confidence intervals. For conversational speech with high per-conversation variability, a sample of this size makes the claim that CASPER 'highlights the challenges of spontaneous speech' statistically fragile. The authors should report the sample composition (number of conversations, speakers, gender/accent distribution), compute confidence intervals by bootstrap across conversations, and ideally enlarge the evaluation set or show that the WER is stable across random subsamples. This does not require new data collection, only more care in reporting the existing manual transcript effort.","section":"IV.C.1 (ASR)"},{"comment":"The de-identification procedure replaces occurrences of participant-provided names with silence based on WhisperX word-level timestamps. This introduces artificial silent gaps into the released audio. The paper does not quantify what fraction of the audio is affected, nor does it discuss how these gaps interact with the downstream evaluations. For ASR, if the reference transcripts omit the silenced regions, the WER is computed on a modified signal that no longer reflects fully natural speech. For diarization, the WhisperX-derived reference is generated after the same silencing, so consistency is preserved operationally, but the resulting reference—and the benchmark—depend on the de-identification implementation. At minimum, the paper should state the proportion of audio replaced by silence, describe how the reference transcripts treat these regions, and discuss whether the silence replacement could bias the naturalness or the error rates.","section":"III.D (De-Identification)"}],"minor_comments":[{"comment":"The paper states that 'a significant portion' of conversations occurred between acquaintances but never reports the actual fraction. Since this is a claimed differentiator from crowdsourced datasets, a count or percentage in Table I or the text would be more informative.","section":"III.A/D and IV"},{"comment":"Please clarify whether the 1.95-hour ASR sample is drawn from the held-out 20% test split or from the full dataset, and specify the number of conversations and speakers included.","section":"IV.C.1"},{"comment":"The accent category 'CMN English' is used without definition; please spell out that it refers to English influenced by Mandarin Chinese (Mandarin in ISO 639-3 is 'cmn').","section":"Table I"},{"comment":"There are typographical and spacing issues throughout, e.g., 'CAsualSpeech inPeerEngagement' in the introduction, 'V AD' for voice activity detection in Section III.C, and inconsistent spacing around citation markers. A thorough copyedit is needed.","section":"III.C"},{"comment":"The code link is given through an anonymous.4open.science URL, which is appropriate for anonymized review but should be replaced with a permanent public repository before publication. The HuggingFace dataset link should also be listed as a resource in the conclusion.","section":"References/Footnotes"},{"comment":"The 'naturalness' claim is supported only by a single transcript excerpt. Adding simple quantitative measures—such as filler-word rate, restart frequency, or overlap ratio—would strengthen the characterization of spontaneous speech without requiring additional annotation.","section":"IV.B.2"},{"comment":"The sentence 'We evaluated the 102 hours of publicly available conversations' is ambiguous about whether this is 102 hours of total audio or 102 hours of speech after VAD. Please clarify.","section":"IV.C.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a dataset paper whose main contribution is the resource itself, and the release appears well-intentioned and potentially useful. However, the current evaluation section—especially the diarization benchmark—does not yet establish the paper's headline claim that CASPER is challenging for state-of-the-art systems. The WhisperX-based reference is the single largest risk; if a manual check shows that the reference is close to human labels, the paper could be accepted after standard revisions. If not, the diarization results would need to be substantially reinterpreted. I did not see evidence of methodological misconduct or circularity; the evaluations are standard, and the concerns are about validation and statistical grounding. Fit to a speech/language journal is appropriate, but the evaluation depth is on the lighter side for a full-length paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a solid dataset contribution with one methodological soft spot that needs fixing before the diarization numbers should be trusted. The CASPER release—102 hours of spontaneous conversations between acquaintances, per-channel audio, metadata, and an open-source collection pipeline—is genuinely new and fills a real gap. The recruitment through existing social ties rather than MTurk, the web-based scheduling, and the topic rooms are sensible design choices, and the authors are transparent about their pipeline. The related work is adequate, and the plan to expand to 1,000 hours is plausible.\n\nWhat the paper does well: the collection framework is reproducible, per-channel recording is a real advantage for diarization and overlap analysis, and the metadata (accent, age, device) will be useful. The ASR baselines on a 1.95-hour manually aligned subset are honest—WER 0.31 and 0.53 on spontaneous speech are plausible and consistent with the literature. The example transcript in Table II illustrates natural disfluencies and long-context dependencies well.\n\nWhere I'd push back. The diarization benchmark in Section IV.C.2 builds the reference from WhisperX word-level timestamps on each participant's isolated channel, then maps them to the combined recording. That means Table IV's DER/JER measure agreement with WhisperX's notion of speaker activity, not human-verified ground truth. On spontaneous speech with overlaps and crosstalk, forced alignment can drift, miss words, or insert spurious segments, so the absolute error rates may be inflated. The paper does not flag this as a limitation; it reads as if these are ground-truth labels. A small manually annotated subset would settle whether the numbers hold. This doesn't invalidate the dataset, but it does mean the claim that CASPER is challenging for state-of-the-art diarization systems is currently contingent.\n\nTwo smaller points. The ASR evaluation is only 1.95 hours with no confidence intervals; fine for a baseline, but don't over-read it. The de-identification only replaces participant-provided names and surnames; other personal references in conversation would remain, and that limitation should be stated explicitly.\n\nWho this is for: anyone building or evaluating ASR, diarization, or speech-language models on conversational data. The dataset itself is the contribution; the baselines are a sketch. It deserves a serious referee. I'd send it to review with a request for the diarization validation and a limitations section. I'd probably cite it if I worked in this area, but I'd wait for the final version before relying on the diarization numbers.","headline":"Solid new spontaneous-speech dataset with a reproducible collection pipeline; the diarization numbers rest on unvalidated WhisperX-derived references and need a manual check before being taken at face value.","tokens_in":10970,"tokens_out":3022,"would_cite":true,"duration_ms":25988,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CASPER, a new dataset of 102 hours of unscripted conversations between acquaintances, shows that state-of-the-art ASR and diarization systems make many more errors on spontaneous speech than on scripted corpora.","keywords":["spontaneous speech dataset","conversational speech","automatic speech recognition benchmark","speaker diarization","data collection pipeline","speech language models","disfluency","accent diversity"],"falsifier":"Take a random 10-hour sample of the released conversations, have human annotators label speaker turns and overlaps on the combined two-channel audio, and recompute DER and JER against this manual reference. If the human-referenced DER for pyannote and NeMo falls well below the reported 25.93% and 30.77%, the central claim that CASPER is especially hard for diarization would be disproven; if it stays in the same range, the claim would survive.","tokens_in":9990,"feed_emoji":"🗣️","tokens_out":5634,"duration_ms":48245,"temperature":0.7,"pith_summary":"CASPER is a new speech dataset built from 200 hours of recorded conversations between people who already know each other, with 102 hours released publicly. The paper's claim is that this kind of unscripted, peer-to-peer dialogue, full of hesitations, restarts, overlaps, and long-range callbacks, is exactly what current speech models lack, because most large corpora contain scripted or read speech. To back that claim, the authors benchmark two ASR systems and two diarization systems, finding high error rates (WER 0.31–0.53; DER 25.93–30.77%). The dataset, the recording pipeline, and the speaker metadata are released to support future work on spontaneous speech.","feed_headline":"102 hours of casual speech expose weak spots in ASR and diarization","feed_subtitle":"A new release records pairs of friends in real settings, capturing the ums, restarts, and overlaps scripted corpora lack.","key_machinery":"The load-bearing mechanism is the web-based recording pipeline built on Daily.co and Firebase: it pairs acquainted peers, offers topic prompts as conversation starters, records each participant's audio on a separate channel, and stores metadata in a SQL database. This per-channel design is what makes per-speaker ASR, diarization, and overlap analysis possible, while the acquaintance pairing is what keeps conversations natural. Post-processing, including Silero VAD for speech detection, WhisperX for word-level timestamps, and de-identification by replacing identifying names with silence, turns the raw recordings into a usable dataset.","core_discovery":"The central discovery is a reproducible pipeline that elicits natural conversations at scale: pairs of participants who already know each other register on a web platform, pick a broad topic prompt, and record a two-channel conversation from their own devices, with each channel captured separately. The released subset contains 102 hours of speech from 208 participants, with demographic metadata, train/test splits, and partial human transcriptions. On a 1.95-hour manually transcribed sample, Whisper-large-v3 reaches 0.31 WER and SeamlessM4T-large 0.53; on automatic diarization of the full release, pyannote.audio-v3.1 scores 25.93% DER and NeMo 30.77%. The paper interprets these results as evidence that spontaneous conversation is underrepresented in current training data and that CASPER can serve as a benchmark and resource for improving ASR, diarization, and speech-language models.","pith_inferences":["Because the diarization reference labels are generated by WhisperX timestamps per channel rather than by human annotation of the mixed conversation, the reported DER/JER may not be an accurate measure of true difficulty, especially in overlaps; a human-labeled diarization subset would settle this.","The per-channel recording format makes CASPER a natural testbed for target-speaker extraction and source-separation methods, which the paper does not evaluate.","The 'callback' long-context dependencies could be turned into a benchmark for discourse coherence and dialogue-state tracking, an extension beyond the paper's ASR and diarization focus.","The acquaintance-recruitment method could be compared against crowdsourced stranger recruitment on measures of naturalness and engagement, offering a direct test of the paper's claim that familiarity drives conversational quality."],"forward_implications":["If CASPER delivers what it claims, training and evaluating ASR on it should lower word error rates on unscripted conversational speech, since the dataset contains exactly the disfluencies and turn-taking patterns that current models miss.","The diarization baselines (DER roughly 26–31%) show clear headroom, making CASPER a public benchmark with fixed train/test splits for improving diarization under overlap and variable device acoustics.","The per-channel recordings and speaker metadata enable research on speaker-conditional recognition, accent-robust ASR, and analysis of long-context dialogue coherence.","The reproducible pipeline can be reused to grow the corpus toward the announced 1,000-hour goal and to collect spontaneous speech in other languages or settings."],"supporting_citations":[{"why":"Supplies the classic telephone-corpus baseline that CASPER compares against on conversation length and audio quality.","marker":"[11]"},{"why":"Provides the 10,000-hour multi-domain ASR corpus whose media-derived, scripted content CASPER is designed to complement.","marker":"[10]"},{"why":"Whisper-large-v3 is the ASR system used to establish the 0.31 WER baseline on a manually transcribed sample.","marker":"[23]"},{"why":"WhisperX supplies word-level timestamps used for de-identification and for constructing the diarization reference labels.","marker":"[24]"},{"why":"pyannote.audio is the diarization pipeline whose DER 25.93% and JER 29.63% establish the difficulty benchmark.","marker":"[27]"},{"why":"NeMo provides the second diarization baseline, with DER 30.77% and JER 33.87%.","marker":"[29]"},{"why":"Silero-v4 VAD is used to estimate the speech-to-nonspeech ratio, yielding the 158-hour actual-speech figure.","marker":"[22]"},{"why":"SeamlessM4T-large is the second ASR baseline, reporting a 0.53 WER.","marker":"[25]"}],"fun_headline_variants":["100+ hours of casual talk reveal ASR and diarization gaps","New CASPER dataset captures real conversations, exposing speech model limits","Reproducible pipeline yields 102 hours of spontaneous speech for AI training","Natural dialogue dataset CASPER benchmarks ASR and diarization weak spots","CASPER: 102-hour spontaneous speech dataset stresses speech models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diarization error rates are only as trustworthy as the automatic WhisperX word-level timestamps used as ground truth; if those timestamps mislabel or miss overlapping speech, the reported difficulty of the dataset is not a reliable measure.","fun_headline_variants_meta":{"raw":{"variants":["100+ hours of casual talk reveal ASR and diarization gaps","New CASPER dataset captures real conversations, exposing speech model limits","Reproducible pipeline yields 102 hours of spontaneous speech for AI training","Natural dialogue dataset CASPER benchmarks ASR and diarization weak spots","CASPER: 102-hour spontaneous speech dataset stresses speech models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000203,"raw_usage":{"total_tokens":1338,"prompt_tokens":851,"completion_tokens":487,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":394}},"tokens_in":467,"tokens_out":487,"duration_ms":4480,"temperature":1.0,"reasoning_tokens":394,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:08:48.221875+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random 10-hour sample of the released conversations, have human annotators label speaker turns and overlaps on the combined two-channel audio, and recompute DER and JER against this manual reference. If the human-referenced DER for pyannote and NeMo falls well below the reported 25.93% and 30.77%, the central claim that CASPER is especially hard for diarization would be disproven; if it stays in the same range, the claim would survive.","supporting_citations":[{"cited_title":"Switchboard: Telephone speech corpus for research and development,","cited_arxiv_id":null,"evidence_quote":"Supplies the classic telephone-corpus baseline that CASPER compares against on conversation length and audio quality."},{"cited_title":"Whisperx: Time-accurate speech transcription of long-form audio,","cited_arxiv_id":null,"evidence_quote":"WhisperX supplies word-level timestamps used for de-identification and for constructing the diarization reference labels."},{"cited_title":"pyannote.audio 2.1 speaker diarization pipeline: principle, benchmark, and recipe,","cited_arxiv_id":null,"evidence_quote":"pyannote.audio is the diarization pipeline whose DER 25.93% and JER 29.63% establish the difficulty benchmark."}],"review_version":1}