{"id":"a16c0280-ceb4-4e1e-8e77-7eff424a06d0","arxiv_id":"2505.19103","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"WHISTRESS extends Whisper with a token-level stress classifier trained on a new synthetic dataset, and shows zero-shot transfer to natural speech benchmarks.","lead":"This paper adds a stress-detection head to Whisper, so transcription systems can mark which words a speaker emphasizes without needing word timestamps or forced alignment. It also introduces a synthetic dataset with LLM-generated stress labels and TTS speech, reporting strong zero-shot results on real expressive speech benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Expresso zero-shot comparison may be invalid: Table 2 caption says WHISTRESS 0-shot is evaluated on all four Expresso speakers, while Section 5.1 and the EmphaClass comparison use only ex01/ex02; the 'outperforms existing methods' claim hinges on this mismatch.","rationale":"The reader's weakest_assumption focused on the fidelity of synthetic TTS prosody, which is a legitimate concern about construct validity. I agree that the synthetic labels are unvalidated and that zero-shot transfer from Google-TTS SSML modifications to natural expressive speech is not guaranteed. However, the more immediately load-bearing issue is the evaluation inconsistency: the paper's own text contradicts the table caption regarding the Expresso test speakers. If the 0-shot result was computed over all four speakers while EmphaClass is evaluated only on ex01/ex02, the central comparison is invalid regardless of synthetic-data fidelity. This is a concrete, checkable flaw that could overturn the headline claim, and it deserves priority because it affects the interpretation of the main experimental evidence. I do not dispute the general plausibility of the approach, but the reported superiority on Expresso cannot be trusted until the speaker split is clarified and the comparison is rerun on identical test data. The reader's rationale did mention the 'Expresso test speakers' inconsistency, so we are partially aligned, but the reader did not elevate it to the primary weakest assumption. I recommend keeping the original CONDITIONAL verdict pending the test; if the test shows a large drop, the verdict should move toward REJECT.","tokens_in":8085,"tokens_out":4058,"duration_ms":34515,"concrete_test":"Re-run the WHISTRESS 0-shot evaluation on the Expresso dataset restricted to speakers ex01 and ex02 only (as described in Section 5.1), using the same filtering criteria as EmphaClass, and compute word-level precision, recall, and F1. Compare directly with EmphaClass's reported F1 of 0.654 on the same speakers. If WHISTRESS's F1 falls below 0.654, or even if it remains above but the gap narrows substantially, the 'outperforms existing methods' claim on natural speech is not supported. The released repository should be checked to confirm which speakers were used in the reported Expresso results.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result that WHISTRESS surpasses EmphaClass on natural speech (Expresso) is undermined by an apparent test-set inconsistency. Section 5.1 states: 'For fair comparison, we use the same test set configuration as [29], selecting the speakers with IDs ex01 and ex02.' However, the caption of Table 2 states: 'For Expresso and EmphAssess, we compare two variants of WHISTRESS: (i) 0-shot and evaluation over all four speakers, and (ii) Training on speakers ex03, ex04, and evaluation over speakers ex01, ex02.' The reported Expresso row for WHISTRESS [0-shot] (Prec 0.573, Rec 0.863, F1 0.689) is thus ambiguous: if it was computed over all four speakers, it is not comparable to EmphaClass's 0.654 F1, which was obtained on ex01/ex02 only. Since EmphaClass was trained on closed-source data 'with a distribution closely matching Expresso' (Section 5.3), a fair comparison on the same test speakers is essential. If WHISTRESS's performance on ex01/ex02 alone is lower, the claim that WHISTRESS 'outperforms existing methods' on natural speech loses its primary support. The paper provides no per-speaker breakdown, and the released code was not audited here, so the actual evaluation split remains unclear. This is a concrete, load-bearing flaw because the abstract's central promise of 'strong zero-shot generalization' rests on the Expresso comparison; the EmphAssess zero-shot result (F1 0.797 vs. EmphaClass 0.938) actually shows WHISTRESS underperforming.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces WHISTRESS, an alignment-free extension of Whisper for token-level sentence stress detection. A stress head consisting of one Whisper decoder block and a two-layer FCNN is trained on a new synthetic dataset TINYSTRESS-15K (~15 hours), created by using GPT-4o-mini to label stressed words in TinyStories sentences and Google TTS with SSML prosodic modifications. The backbone Whisper is frozen. The paper reports that WHISTRESS outperforms a BLSTM acoustic baseline, CRF [11], hierarchical BLSTM [10], and EmphaClass [29] on Aix-MARSEC and Expresso, with zero-shot transfer from synthetic to natural speech. It also includes an analysis of Whisper layers' prosodic information and a layer-selection study.","tokens_in":8472,"tokens_out":5839,"duration_ms":47443,"significance":"WHISTRESS addresses a real gap: existing sentence stress detectors require forced alignment or manual annotations, while this model needs no word timestamps at inference. The synthetic data pipeline is a practical contribution, and the authors release code, weights, and dataset. The layer analysis (Section 4) is informative and connects to known acoustic/semantic trade-offs in self-supervised speech models. The main empirical claim—strong zero-shot generalization to natural benchmarks—would be significant if confirmed, since it would show that synthetic prosodic training transfers across speakers and recording conditions.","major_comments":[{"comment":"The evaluation protocol for Expresso is inconsistent. Section 5.1 states 'we use the same test set configuration as [29], selecting the speakers with IDs ex01 and ex02,' but the caption of Table 2 says the 0-shot variant is evaluated 'over all four speakers,' with only the in-domain variant (marked *) evaluated on ex01/ex02. If the WHISTRESS[0-shot] Expresso row (F1 0.689) includes ex03/ex04, it is not directly comparable to EmphaClass's 0.654, which was computed on ex01/ex02. The claim that WHISTRESS 'surpasses EmphaClass on Expresso' therefore rests on an ambiguous split. The same ambiguity applies to EmphAssess: the 0-shot row may include all four voices, while EmphaClass's F1 is presumably on ex01/ex02. Please report the exact speaker split for every row, provide per-speaker results, and re-compute the comparison on the same subset.","section":"§5.1 vs Table 2"},{"comment":"No confidence intervals, significance tests, or multiple-seed results are reported. The margins that support the abstract's comparative claim are small (e.g., WHISTRESS 0.689 vs EmphaClass 0.654 on Expresso; 0.943 vs 0.938 on EmphAssess after in-domain training). Given that training involves random initialization and stochastic optimization, the reported F1 differences may not be stable. Please provide standard errors or confidence intervals over several seeds for the key comparisons, or at least state the number of runs.","section":"§5 / Table 2"},{"comment":"The abstract states that WHISTRESS 'outperforms existing methods,' but the results in Table 2 show that WHISTRESS[0-shot] on EmphAssess (F1 0.797) is substantially below EmphaClass (F1 0.938). The paper's discussion acknowledges that EmphaClass was trained on a closed-source, domain-matched dataset, but the unqualified abstract claim is too strong. Please qualify the claim to the specific benchmarks and settings where the improvement holds, or add a direct comparison controlling for training data.","section":"Abstract / §5.3"},{"comment":"The reported validation on TINYSTRESS-15K is partially circular: the test labels are produced by the same automated pipeline (GPT-4o-mini selection plus SSML-modified TTS) that generated the training data. This makes the test set useful for model selection but not for measuring absolute accuracy on natural speech. The external zero-shot evaluations on Aix-MARSEC, Expresso, and EmphAssess are independent and should remain the primary evidence; please explicitly state this limitation in Section 5.3 or Section 2. The choice of layer 9 (Table 1) is also made on this same test set, so the circularity may influence architecture decisions.","section":"§2 / §5.3 (TINYSTRESS-15K validation)"}],"minor_comments":[{"comment":"The paper uses 'W HISTRESS' and 'T INY STRESS -15K' with unusual spacing in several places (e.g., Abstract, Sections 2 and 3); the dataset name is written 'TINYSTRESS-15K' in the abstract and title but 'T INY STRESS -15K' in the body. Please normalize the notation.","section":"Throughout"},{"comment":"The EmphAssess WHISTRESS row has concatenated values '0.945*0.942*0.943*' that are hard to read; use proper column separators or formatting for the starred entries.","section":"Table 2"},{"comment":"References [9] and [29] are the same work (de Seyssel et al., 'Emphassess: a prosodic benchmark on assessing emphasis transfer in speech-to-speech models'); please consolidate them into a single reference.","section":"References"},{"comment":"The size and composition of the CREMA-D subset used for the prosodic information analysis are not specified; adding this detail would improve reproducibility.","section":"§4.1"},{"comment":"The baseline feature extraction is described as 'duration, mean energy and max pitch' inspired by [10], but the exact windowing, aggregation, and alignment procedure are not specified, making the baseline difficult to reproduce.","section":"§5.2"}],"recommendation":"major_revision","confidential_remarks":"The paper makes a useful contribution and the release of code, weights, and data is commendable. The main risk is the Expresso evaluation ambiguity and the abstract overclaim; I would ask the authors to provide the exact evaluation scripts and per-speaker breakdown. If the 0-shot Expresso result does not hold on ex01/ex02 alone, the 'outperforms existing methods' claim should be substantially revised. The circularity of the TINYSTRESS-15K validation is secondary but should be acknowledged. The lack of statistical confidence intervals is also worth addressing in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Morning,\n\nThe short version: this paper has a genuinely useful idea — a frozen Whisper plus a small stress head, trained on synthetic data that combines LLM stress labels with TTS prosody — and the authors did the field a favor by releasing the code, weights, and dataset. The layer analysis (deeper layers encode less prosodic info, but intermediate layers work best for stress) is a nice, concrete finding. I believe the central architecture is sound and the Aix-MARSEC numbers look credible.\n\nThat said, the paper's headline claim that WHISTRESS 'outperforms existing methods' on natural speech rests on an Expresso comparison that, as written, is inconsistent. Section 5.1 says the test set uses ex01/ex02 for fairness with EmphaClass, but the Table 2 caption says the 0-shot WHISTRESS variant is evaluated over all four speakers. If that latter reading is correct, the 0-shot F1 of 0.689 is not comparable to EmphaClass's 0.654 on two speakers. The paper gives no per-speaker breakdown, and I couldn't resolve the split from the text. That's a load-bearing ambiguity because the other zero-shot benchmark — EmphAssess — actually shows WHISTRESS below EmphaClass (0.797 vs 0.938). The claim of zero-shot superiority over existing methods may still hold after clarification, but right now it's unverified.\n\nOther soft spots: no confidence intervals or multiple seeds for the main comparisons, which is standard in this subfield and not fatal on its own. The TINYSTRESS validation is partially circular (same LLM+SSML pipeline generates train and test labels), though the external benchmarks mitigate it. The synthetic stress labels also aren't validated against human judgments. The training filter that drops samples where Whisper's transcription length differs from ground truth is a reasonable mitigation, but it could bias the training set.\n\nOn the positive side, the EmphAssess in-domain training result (ex03/ex04 to ex01/ex02) is a fair comparison and looks good. And the Aix-MARSEC results are solid. The method is simple, reproducible, and alignment-free, which is a real practical advantage.\n\nWho this is for: anyone working on prosody, expressive TTS, or enriching ASR outputs. It deserves a serious referee — I'd send it out rather than desk reject — but it needs a significant revision: resolve the Expresso split, add per-speaker numbers, report significance, and ideally validate the synthetic labels on a small human-judged sample.\n\nRegards.","headline":"Useful alignment-free stress head and a clever synthetic data pipeline, but the Expresso comparison that drives the 'outperforms' claim is internally inconsistent and needs fixing.","tokens_in":8997,"tokens_out":3185,"would_cite":true,"duration_ms":23598,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A frozen transcription model can read sentence stress from its own internal embeddings, trained entirely on synthetic speech.","keywords":["sentence stress detection","prosody","speech emphasis","automatic speech recognition","zero-shot generalization","synthetic training data","alignment-free","Whisper"],"falsifier":"Look for a result that would break the transfer claim: take a natural, multi-speaker corpus with expert linguistic annotations of sentence stress and run the synthetic-trained WHISTRESS zero-shot alongside an aligner-based baseline; if its F1 falls to near chance while the baseline stays strong, the text-to-speech-to-human transfer assumption fails. A complementary check is to test whether WHISTRESS only learns the exact SSML recipe—longer, louder, higher—by inspecting false positives on naturally stressed words that do not follow that recipe.","tokens_in":7889,"feed_emoji":"🎙️","tokens_out":8593,"duration_ms":53047,"temperature":0.7,"pith_summary":"WHISTRESS attempts to show that sentence stress—which words in a spoken sentence carry emphasis—can be detected and added to a transcript without needing word timestamps, forced alignment, or any extra input at inference time. The proposed method attaches a small learned head to a frozen Whisper speech-recognition model, and the training signal comes entirely from TINYSTRESS-15K, a roughly 15-hour synthetic dataset built by having a language model mark stressed words in everyday sentences and a text-to-speech service render them with exaggerated prosody. The central claim is that this synthetic-only training transfers to natural human speech: WHISTRESS outperforms aligner-dependent baselines on its validation sets and shows zero-shot performance that beats an existing emphasis classifier on one benchmark and is competitive on another. If the claim holds, stress annotation becomes a cheap add-on for automatic speech recognition rather than a data-hungry specialist task.","feed_headline":"Synthetic speech alone teaches a model to flag stressed words","feed_subtitle":"WHISTRESS adds emphasis marks to transcripts from frozen Whisper embeddings, with no timestamps or aligners.","key_machinery":"The load-bearing object is the stress-detection head: one additional Whisper decoder block that cross-attends over the encoder and decoder embeddings of the 9th backbone layer, followed by a two-layer fully connected binary classifier that scores each generated token. The 9th layer is chosen empirically because it balances prosodic detail and linguistic content better than earlier or later layers. The synthetic data pipeline is the other half: sentences from TinyStories, stress labels produced by a language model, and text-to-speech synthesis that modifies emphasized words with reduced speaking rate (30–85%), raised volume (+3–6 dB), and higher pitch (+1.5 semitones), with Gaussian noise added for variability.","core_discovery":"The paper's core discovery is that a frozen speech-recognition model already stores enough prosodic information in its intermediate layers that a small decoder-style head can read out sentence stress per token. The head is trained with binary cross-entropy on token-level stress labels, after a relaxed word-length filter prevents Whisper transcription errors from shifting labels; no word timestamps are used in training, and none are needed at inference. The key reported findings are that WHISTRESS reaches 0.909 F1 on TINYSTRESS-15K and 0.961 F1 on Aix-MARSEC, beating baselines that receive true or force-aligned word boundaries, and that the version trained only on synthetic data reaches 0.689 F1 zero-shot on Expresso, ahead of EmphaClass's 0.654. On EmphAssess, zero-shot WHISTRESS reaches 0.797 F1 with high recall, while a version trained on about 1,500 in-domain samples reaches 0.943 F1, slightly above EmphaClass.","pith_inferences":["Editorial inference: the same frozen-backbone-plus-head recipe could plausibly be applied to other word-level paralinguistic labels, such as emotion or uncertainty, where synthetic data can be generated with controlled prosody; the paper only anticipates this possibility.","Editorial inference: the layer-9 sweet spot suggests a general design heuristic—intermediate layers of speech encoders may be better than final layers for prosody-related tasks—that could be tested on other languages and model sizes.","Editorial inference: the label-alignment trick could be strengthened by edit-distance matching over whole words instead of dropping samples with mismatched word counts, perhaps extending the method to noisier automatic speech recognition conditions.","Editorial inference: because the paper's Aix-MARSEC analysis shows most unstressed words are a small recurring set, part of the reported gain may come from lexical predictability rather than acoustic sensitivity; a synthetic dataset with more unpredictable unstressed words would test whether true prosodic transfer remains."],"forward_implications":["Transcription systems can enrich outputs with emphasis markup without changing the recognized words, since the backbone is frozen and the head is purely additive.","A fully automated pipeline can produce stress-annotated training data at scale, avoiding closed-source corpora and human annotators.","The alignment-free design means inference does not require forced alignment or timestamp estimation, removing an error source and a scalability bottleneck.","The layer analysis implies that prosody-aware heads should be attached at intermediate layers; the 9th layer outperforms the final layer for stress detection.","If the zero-shot numbers transfer, stress detection becomes feasible for domains where no in-domain stress annotations exist."],"supporting_citations":[{"why":"Supplies the Whisper backbone and its frozen encoder and decoder embeddings that the stress head reads.","marker":"[14]"},{"why":"Provides the acoustic correlates of duration, amplitude, and pitch on which the synthetic prosody modifications are based.","marker":"[4]"},{"why":"Supplies TinyStories paragraphs used as source text for TINYSTRESS-15K sentences.","marker":"[18]"},{"why":"Produces the semantically meaningful stressed-word labels used as training targets.","marker":"[20]"},{"why":"Supplies the Aix-MARSEC corpus used for validation with sentence-stress labels in NRU notation.","marker":"[27]"},{"why":"Defines the EmphAssess benchmark and the EmphaClass baseline compared in zero-shot evaluation.","marker":"[29]"},{"why":"A hierarchical BLSTM sentence-stress baseline from prior work, compared on Aix-MARSEC.","marker":"[10]"},{"why":"A CRF-based sentence-stress baseline from prior work, compared on Aix-MARSEC.","marker":"[11]"}],"fun_headline_variants":["Frozen Whisper layers reveal sentence stress without alignment","Synthetic stress training beats aligned baselines on real speech","No aligners needed: prosody read from frozen embeddings","Zero-shot stress detection from synthetic-only training data","WHISTRESS reads sentence stress straight from Whisper features"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim stands on the assumption that the synthetic text-to-speech renderings, with stressed words made longer, louder, and higher-pitched, capture enough of natural sentence stress that a model trained on them will transfer to real human speech.","fun_headline_variants_meta":{"raw":{"variants":["Frozen Whisper layers reveal sentence stress without alignment","Synthetic stress training beats aligned baselines on real speech","No aligners needed: prosody read from frozen embeddings","Zero-shot stress detection from synthetic-only training data","WHISTRESS reads sentence stress straight from Whisper features"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000167,"raw_usage":{"total_tokens":1252,"prompt_tokens":932,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":241}},"tokens_in":548,"tokens_out":320,"duration_ms":4589,"temperature":1.0,"reasoning_tokens":241,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:19:48.994229+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look for a result that would break the transfer claim: take a natural, multi-speaker corpus with expert linguistic annotations of sentence stress and run the synthetic-trained WHISTRESS zero-shot alongside an aligner-based baseline; if its F1 falls to near chance while the baseline stays strong, the text-to-speech-to-human transfer assumption fails. A complementary check is to test whether WHISTRESS only learns the exact SSML recipe—longer, louder, higher—by inspecting false positives on naturally stressed words that do not follow that recipe.","supporting_citations":[{"cited_title":"Furthermore, we investigate how WHISTRESS identifies sentence stress by evaluating different layers as input for the stress detection head","cited_arxiv_id":null,"evidence_quote":"Provides the acoustic correlates of duration, amplitude, and pitch on which the synthetic prosody modifications are based."},{"cited_title":"Joint detection of sentence stress and phrase boundary for prosody,","cited_arxiv_id":null,"evidence_quote":"Supplies TinyStories paragraphs used as source text for TINYSTRESS-15K sentences."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Aix-MARSEC corpus used for validation with sentence-stress labels in NRU notation."},{"cited_title":"Aix-MARSEC is a speech corpus containing over 5 hours of 1980s BBC radio recordings, featuring 53 speakers across dif- ferent speech styles [27]","cited_arxiv_id":null,"evidence_quote":"Defines the EmphAssess benchmark and the EmphaClass baseline compared in zero-shot evaluation."},{"cited_title":"Accent is predictable (if you’re a mind-reader),","cited_arxiv_id":null,"evidence_quote":"A hierarchical BLSTM sentence-stress baseline from prior work, compared on Aix-MARSEC."},{"cited_title":"Van Heuven,Acoustic Correlates and Perceptual Cues of Word and Sentence Stress: Theories, Methods and Data , 12 2018, pp","cited_arxiv_id":null,"evidence_quote":"A CRF-based sentence-stress baseline from prior work, compared on Aix-MARSEC."}],"review_version":1}