{"id":"e5665074-bd92-4e40-9179-809c83a67fce","arxiv_id":"2607.23813","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Earnings25 releases ~500 hours of 2025 earnings-call audio with aligned transcripts, speaker/industry metadata, and reproducible Whisper and Parakeet-TDT baselines.","lead":"Earnings25 is a ~500-hour English earnings-call speech benchmark with full calls and industry-balanced segments, plus speaker and industry metadata. It lets researchers measure finance ASR beyond aggregate word error rate under realistic call conditions.","discovery_kind":"extension","skeptic_critique":{"model":"moonshotai/kimi-k3","headline":"The benchmark's central use case — industry-aware WER comparison — rests on reference transcripts whose accuracy and provenance are never specified, and any industry-correlated reference error would directly confound the paper's own headline analysis (biotech/pharma \"harder\" than average).","rationale":"The reader correctly identified transcript accuracy/provenance as the weakest assumption; I agree and sharpen why it is load-bearing rather than merely hygienic: the paper's distinguishing analytical contribution (industry-stratified evaluation revealing that aggregate WER hides domain variation, §6.2/Table 5) is exactly where unverified references would bias results, because reference errors and ASR errors are both plausibly concentrated in terminology-dense industries. Without a documented transcript pipeline and a human-audited error rate, the industry-level WER deltas cannot be attributed to model difficulty with confidence. The one-segment-per-industry design adds variance on top. That said, this is a dataset/resource paper and the resource itself (498h of aligned audio with metadata, seeded sampling, Zenodo release, standardized scoring) has clear value even if some numbers shift; the fix is documentation plus a bounded audit, not a redesign. Hence CONDITIONAL: accept the resource, but condition the benchmark-validity claims on (a) stating transcript provenance and QA procedure, and (b) publishing a human-audited subset with measured reference error rates, so downstream users can calibrate how much of any WER difference is model versus reference. If the audit shows reference error <1–2% and industry-uncorrelated, the reader's ACCEPT stands as-is.","tokens_in":7452,"tokens_out":1628,"duration_ms":36394,"concrete_test":"Audit the references: have human transcribers independently re-transcribe a stratified ~15–20h subset (e.g., 2 segments each from the 10 highest-WER and 10 lowest-WER industries in testset-segmented, plus ~5h of testset-full), then compute reference-vs-audit WER as the transcript error rate. If the reference error rate exceeds ~2% overall, or correlates with industry such that the biotech/pharma gap in Table 5 shrinks materially after correcting references, the industry-comparison claim needs qualification. As a cheap first pass, run Whisper-large-v3 and Parakeet on the same audio and flag segments where both models agree with each other but disagree with the reference — high-confidence reference-error candidates.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"For a benchmark paper, the ground truth is the load-bearing component. Earnings25 never states where the reference transcripts come from (vendor? Bloomberg internal? machine-generated plus correction?) and describes no human QA, no inter-annotator agreement, and no reference error-rate estimate (§4 describes only alignment and segmentation, not transcript production). This matters beyond generic hygiene because of the paper's central analytical claim in §6.2: that aggregate WER \"masks substantial domain-dependent variation,\" illustrated by Table 5 showing biotech (15.4%) and pharma (15.3%) WER well above the 10.8% corpus average. If reference transcripts are machine-generated or vendor-produced without domain-adapted review, transcription errors will concentrate precisely in jargon-dense industries — entity names, drug names, technical terms — i.e., the same industries reported as hardest for ASR. In that case part of the measured WER gap is reference error, not model error, and the direction of the confound exactly manufactures the paper's illustrative finding. A second, related soft spot: testset-segmented has exactly one ~9.5-minute segment per each of 290 industries (Table 1), so per-industry WER estimates on the flagship \"industry-balanced\" track rest on a single segment — high variance, and sensitive to any alignment/segmentation artifacts from the 0.5s max-word-duration cap and greedy block construction (§4.2–4.3). Neither issue is disqualifying for the resource itself, but both bear directly on the \"beyond aggregate WER\" claims that distinguish this benchmark from Earnings-21/22.","agreement_with_reader":"agree"},"referee_report":{"model":"moonshotai/kimi-k3","summary":"The paper introduces Earnings25, an English finance-domain ASR benchmark with two test sets: testset-full (498 h of complete S&P 500 earnings calls from 2025 Q4; ~514 calls, 12 countries, 284 industry categories) and testset-segmented (46 h; 290 industry-balanced 5–10 min segments, one per industry, U.S.-only, sampled from >2,000 U.S. calls in 2025). Construction is documented: two-stage U.S. filter plus disproportionate stratified sampling, CTC forced alignment in NeMo with a 0.5 s max-word-duration cap, greedy block aggregation, boilerplate filtering, one segment per call, all seeded (2025). Aligned transcripts and metadata (speaker roles, industry labels, call structure) are released via Zenodo. Baselines for Whisper (base/medium/large-v2) and Parakeet-TDT-0.6B-v2 are reported under four consistent WER normalizations; the segmented set is slightly harder than the full set, and an illustrative industry cut (Table 5) shows biotech/pharma WER ≈15.3–15.4% vs. a 10.8% aggregate.","tokens_in":7780,"tokens_out":8919,"duration_ms":340448,"significance":"If reference quality is documented, Earnings25 is a useful complement to Earnings-21/22 and SPGISpeech: it adds industry-balanced evaluation, metadata for role- and industry-aware error analysis, and a genuinely reproducible pipeline — fixed seed 2025, seeded stratified sampling, logged decoding configurations, and four WER variants with identical normalization applied to reference and hypothesis. Public release of audio, transcripts, and alignments lowers the barrier to comparable financial-ASR evaluation. The baseline numbers are plausible and internally consistent (Table 4 model ordering matches expectations; Tables 1–3 cross-check arithmetically). The industry-stratified design is a real methodological contribution over frequency-weighted corpora, and the resource is likely to see use.","major_comments":[{"comment":"Every reported number is measured against reference transcripts, yet §4 describes only alignment and segmentation: the source of the references (vendor service, internal human transcription, ASR-plus-correction), the transcription conventions (verbatim? disfluencies? number formatting?), and any QA or error-rate estimate are never stated. For a benchmark, the ground truth is the product, so this is load-bearing. It also interacts with the paper's own illustrative claim: Table 5 shows biotech (15.4%) and pharma (15.3%) well above the 10.8% aggregate, and if reference errors concentrate in jargon-dense industries (drug names, entity names), part of that gap is reference error rather than model error — the confound would manufacture the finding. This is fixable within scope: (i) state transcript provenance and style conventions; (ii) audit a stratified subsample (e.g., double-transcribe ~5–","section":"§4 (Corpus Generation), §3, §5.3, Tables 4–5"},{"comment":"Statistical support for the industry-level analysis is thin and under-specified. Table 5 rests on 2–3 calls per subsector with no stated selection procedure and no uncertainty quantification; Table 3 shows 9 biotech and 6 pharma calls exist in testset-full, so the basis for the 2–3-call subset must be given (seeded subsample? convenience? worst cases?). Relatedly, testset-segmented contains exactly one ~9.5-minute segment per industry (Table 1), so per-industry WER on the flagship balanced track is a single-sample estimate and cannot support industry rankings. Please (i) state the Table 5 selection procedure, (ii) add bootstrap confidence intervals over calls, and (iii) clarify that the segmented track is intended to yield one balanced aggregate number rather than per-industry inference, or redesign accordingly.","section":"Table 5, §6.2, Table 1"},{"comment":"The attribution that higher WER on testset-segmented 'reflects the effect of industry stratification' is confounded. Relative to testset-full, the segmented set also (a) removes operator boilerplate, i.e., some of the easiest speech, which should raise WER; (b) spans Q1–Q4 2025 rather than Q4 only; (c) consists of mid-call excerpts lacking long-form decoding context; and (d) is U.S.-only, which should lower WER. The comparison conflates at least four factors, so the sentence as written overclaims. Either soften it to a hypothesis or add a small control — e.g., a frequency-weighted U.S.-only sample processed with identical filtering and excerpting — to isolate the stratification effect.","section":"§6.2"}],"minor_comments":[{"comment":"Name the specific NeMo CTC checkpoint used for forced alignment, and justify the 0.5 s maximum word-duration cap (could it truncate genuinely long tokens such as slowly pronounced company names?). A small alignment-quality spot check would substantiate the 'reliable timing' claim in §4.3.1.","section":"§4.2.1"},{"comment":"State the industry taxonomy and version underlying the 284/290 categories (e.g., BICS level, GICS). The granularity includes categories such as 'Adult Nightclubs,' so the source matters for reproducing the stratified sampling and for cross-study comparability.","section":"§3, §5.2"},{"comment":"The text says testset-full covers 'all S&P 500 companies,' but Tables 2–3 sum to 514 calls. Please reconcile (intra-quarter index changes? dual share classes? multiple calls per company?).","section":"§5.1, Tables 2–3"},{"comment":"Clarify the overlap between the two test sets: can a testset-segmented segment be excerpted from a call that also appears in testset-full? This matters for disjointness claims when both tracks are used together.","section":"§3"},{"comment":"Specify Whisper decoding precisely (temperature, greedy vs. beam and beam size, condition-on-previous-text) or point to the exact configuration file in the released code; 'loaded from a fixed model configuration' is not sufficient for exact replication.","section":"§6.1"},{"comment":"Consider adding bootstrap confidence intervals over calls/segments. The full-vs-segmented gaps (e.g., large-v2: 0.14039 → 0.15333) are modest relative to plausible call-level variance.","section":"Table 4"},{"comment":"Clarify what is actually distributed via the Zenodo DOI — is the audio itself included, or only transcripts/metadata/alignments? The phrase 'terms of the original content providers' leaves unclear what obligations users assume and whether the benchmark is usable out of the box; this affects the practical reproducibility claim.","section":"§8"},{"comment":"The metadata-gap claim should be tempered with respect to SPGISpeech 2.0 [7], which provides speaker-tagged transcription; differentiate Earnings25's speaker-role metadata from SPGISpeech 2.0's speaker tags explicitly.","section":"§2.2"},{"comment":"Typographical: missing spaces in 'forevaluating' (§1.2) and 'testset-fullconsists' (§3); letter-spaced 'W A V' (§4.2.1, Table 1). Align industry labels across tables — Table 5 uses 'Pharma'/'Biotech' while Table 3 uses 'Large Pharma'/'Biotech'.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The resource engineering is solid and the internal numbers cross-check; my concerns are documentation and validation of the ground truth rather than the construction pipeline, and none require new data collection beyond a modest human audit. Two items the editor may wish to verify directly: (1) that the Zenodo artifact at the stated DOI actually contains the audio, given the redistribution language in §8 regarding 'original content providers' — earnings-call audio redistribution rights are worth confirming; (2) the industry taxonomy appears to be a proprietary fine-grained classification, which the authors should name even if the full taxonomy cannot be released."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"Punchline: this is a competent systems/dataset paper that ships something people can actually run—recent earnings audio, two complementary tracks, speaker/industry metadata, and clean Whisper/Parakeet baselines. It is incremental on Earnings-21/22 and SPGISpeech, not a conceptual leap, but the increments matter for evaluation practice inside financial ASR.\n\nWhat is new is the packaging, not the genre. testset-full (~498h S&P 500 Q4 2025) keeps long-form context; testset-segmented (46h, one segment per 290 industries) forces long-tail coverage. Construction is specified well enough: U.S. filter, disproportionate industry stratification, CTC Viterbi alignment in NeMo, greedy 5–10 min blocks, boilerplate filter, fixed seed 2025. Table 4 is reproducible in spirit (four WER variants, no LM, fixed decoding). Limitations (English, U.S.-heavy) are stated plainly. Zenodo release under CC BY for transcripts/metadata is real credit.\n\nSoft spots, in proportion. The stress-test lands on something real but not fatal: the paper never says where the reference transcripts come from or what human QA was done. For a benchmark, ground truth is load-bearing. That weakens the §6.2 claim that biotech/pharma are “harder” (Table 5) more than it weakens the resource itself—if reference error concentrates on jargon, industry gaps partly measure the references. Separately, one ~9.5 min segment per industry means per-industry numbers on the stratified track are high-variance and sensitive to the 0.5s max-word-duration and greedy cuts. Neither issue invents the contribution; both mean “beyond aggregate WER” should be read as enabled analysis, not settled ranking. Citation pattern is appropriate; no circularity in the WERs.\n\nWho it is for: people building or buying ASR on earnings calls, and anyone who needs a modern finance speech test set with role/industry tags. Not required reading outside that lane. Math is scoring protocol, not theory—fine for the genre. I would send it to referees; ask them to demand transcript provenance and a short reference-quality note before camera-ready. Engage if you evaluate financial speech; skim the tables if you only need a citation pointer.","headline":"Solid, usable finance ASR eval resource with real metadata value; the industry-gap story is thinner than the packaging because reference provenance is unspecified.","tokens_in":8887,"tokens_out":578,"would_cite":true,"duration_ms":22266,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Earnings25 is a 500-hour finance ASR benchmark that pairs full S&P 500 earnings calls with industry-balanced segments and rich metadata so systems can be judged beyond overall word error rate.","keywords":["automatic speech recognition","ASR benchmark","financial speech","earnings calls","industry-aware evaluation","long-form speech","word error rate","speaker metadata"],"falsifier":"Re-transcribe a stratified sample of segments with independent human annotators, re-align with an alternate aligner, and check whether model rankings and the large WER gaps between terminology-dense industries (e.g., biotech/pharma) and the aggregate score reverse or shrink materially.","tokens_in":8660,"feed_emoji":"🎙️","tokens_out":872,"duration_ms":20698,"temperature":0.7,"pith_summary":"The paper introduces Earnings25, a public evaluation resource for automatic speech recognition on English earnings calls under realistic conditions. It offers two complementary test sets: nearly 500 hours of complete S&P 500 calls from late 2025, and a 46-hour set of 290 industry-stratified segments drawn from U.S. calls across the year. Aligned transcripts come with speaker roles, industry labels, and call structure, so researchers can measure errors by speaker type and sector rather than a single aggregate score. Reproducible baselines for Whisper and Parakeet-TDT show that industry balance and terminology-dense sectors raise error rates that a frequency-weighted overall metric can hide. A sympathetic reader cares because finance speech mixes jargon, numbers, accents, and rapid Q&A in ways general ASR benchmarks do not capture, and progress needs a shared, metadata-rich yardstick.","feed_headline":"500-hour earnings-call benchmark scores ASR by industry","feed_subtitle":"Full S&P 500 calls plus 290 balanced segments expose jargon errors aggregate WER hides.","key_machinery":"Earnings25’s dual test design—testset-full (long-form natural distribution) plus testset-segmented (one segment per industry via disproportionate stratified sampling)—with CTC forced alignment, quality filtering, and speaker/industry/call-structure metadata that make stratified scoring possible.","core_discovery":"The authors establish Earnings25 as a purpose-built finance-domain ASR benchmark: a 498-hour full-call S&P 500 set plus a 46-hour industry-balanced 290-segment set, both with aligned transcripts and structured metadata for speaker- and industry-aware evaluation beyond aggregate WER, together with standardized Whisper and Parakeet-TDT baselines.","pith_inferences":["If industry-balanced evaluation becomes standard, model selection for finance products may shift toward systems that win on long-tail sectors rather than on high-frequency utilities and banks.","The same stratified-segment recipe could be reused for other jargon-heavy meeting domains (legal, medical, earnings in other languages) where aggregate WER also masks failure modes.","Open release of alignments and speaker tags invites secondary benchmarks for punctuation, numeral formatting, and role-conditioned language models on the same audio."],"forward_implications":["ASR papers on financial speech can report industry- and role-stratified WER on a shared public set instead of only corpus-level averages.","Domain adaptation and jargon-handling methods can be stress-tested on long-tail sectors that frequency-weighted corpora under-represent.","Speaker-aware and diarization-aware pipelines gain a metadata-rich earnings-call evaluation path with executive vs analyst roles.","Baseline numbers for Whisper and Parakeet-TDT under fixed scoring become a reproducible reference point for later systems."],"fun_headline_variants":["Earnings25: 500-hour finance ASR benchmark with industry breakdowns","498-hour S&P 500 earnings set exposes ASR jargon gaps","Finance ASR benchmark pairs full calls with balanced segments","Earnings25 enables speaker- and industry-aware ASR scoring","500-hour earnings-call set benchmarks ASR beyond aggregate WER"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The reference transcripts and the CTC forced-alignment cuts are accurate enough that measured word error rates and industry comparisons reflect true system behavior rather than alignment or transcription artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Earnings25: 500-hour finance ASR benchmark with industry breakdowns","498-hour S&P 500 earnings set exposes ASR jargon gaps","Finance ASR benchmark pairs full calls with balanced segments","Earnings25 enables speaker- and industry-aware ASR scoring","500-hour earnings-call set benchmarks ASR beyond aggregate WER"]},"model":"grok-4.5","effort":"low","cost_usd":0.005285,"raw_usage":{"total_tokens":1408,"prompt_tokens":682,"num_sources_used":0,"completion_tokens":70,"cost_in_usd_ticks":52848000,"prompt_tokens_details":{"text_tokens":682,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":656,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":682,"tokens_out":70,"duration_ms":10298,"temperature":1.0,"reasoning_tokens":656,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-30T11:39:05.047614+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-transcribe a stratified sample of segments with independent human annotators, re-align with an alternate aligner, and check whether model rankings and the large WER gaps between terminology-dense industries (e.g., biotech/pharma) and the aggregate score reverse or shrink materially.","supporting_citations":[],"review_version":1}