{"id":"dd7025bc-f1c0-4f53-a771-bb2bb4bd8e87","arxiv_id":"2507.20091","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A speech language model trained only on audiobooks with explicit word-level prosody tokens displays emerging abilities in prosody-controlled generation, emphasis and emotion understanding, and prosodic consistency across dialogue.","lead":"This paper introduces ProsodyLM, a speech language model that represents speech as text plus word-level prosody tokens, and shows that pre-training on audiobooks alone produces a range of prosody abilities: following style descriptions, detecting emphasis and emotion, and carrying prosody across dialogue turns.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Measured prosody contrasts may be partly decoder artifacts; a decoder-fidelity control is needed before crediting the LM with emerging prosody capabilities.","rationale":"The reader's verdict is CONDITIONAL, driven by uncertainty about encoder/decoder fidelity and missing statistics. I agree with that diagnosis. My stress-test pass identified no internal inconsistency that would force rejection, and the empirical case is credible in structure: group-A baselines share the pre-training data and setup, the focus and emphasis designs are reasonable, and the self-reported limitations (Section 5) are honest. The most load-bearing concern is not that the method is fake, but that the headline claim—emerging prosody capabilities from pre-training alone—requires the evaluation chain to be transparent: the decoder must not be injecting the very contrasts being measured, and the small effects (Happy, Angry, Neutral emotion; energy pairs in Tables 1 and 4) need significance testing. The paper itself notes in Appendix B that only the duration and prosody predictors are retrained, with all other StyleTTS2 modules frozen; this makes the decoder-fidelity control particularly important. The concrete test I propose would settle whether the LM's choices, rather than the decoder's priors or averaging behavior, are responsible for the measured contrasts. If the control passes (re-synthesized oracle audio shows near-zero emotion-conditional log-probability increases, and synthesized contrasts correlate with the LM's token choices), the central claim is solidly supported and the paper could be accepted. If the control fails, the claim weakens to 'the ProsodyLM+decoder system produces expressive speech', which is still useful but not the same as 'the LM performs prosody-to-content inference'. Hence I keep the verdict CONDITIONAL, matching the reader's recommendation, with the concrete control experiment as the condition.","tokens_in":19867,"tokens_out":2158,"duration_ms":20337,"concrete_test":"Run a decoder-fidelity control: take the held-out reference utterances used in the emotion and emphasis evaluations, encode them into prosody tokens with the ProsodyLM encoder, and then re-synthesize them with the ProsodyLM decoder. Measure the F0, duration, and energy differences between original and re-synthesized utterances, and also compute the emotion log-probability metric of Section 4.3.2 on the re-synthesized audio as if it were a ProsodyLM-generated sample. If the re-synthesized oracle audio shows emotion log-probability increases similar to Table 3 (e.g., Sad/Excited around 0.2), then the measured capability is partly a decoder artifact rather than an LM-internal prosody→content inference.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that ProsodyLM learns diverse prosody processing capabilities through pre-training alone. The weakest premise, identified by the reader and confirmed here, is the fidelity of the tokenization/decoder chain: the LM's behavioral scores are measured after the StyleTTS2-derived encoder and decoder have quantized and re-synthesized the prosody. If the decoder's word-level conditioning is coarse or the predictor heads partially retain their original StyleTTS2 priors, the measured F0/duration/energy differences could be at least partly a property of the decoder's realization of any input tokens, not of the LM's token choices. Two observations make this concern concrete. First, in Table 1, ProsodyLM's direct F0 pair effect (18.51) is ~5-10x larger than StyleTTS2 (3.25) and MIMI-tok (1.86); if the decoder were a neutral renderer of arbitrary word-level prosody tokens, this gap would be entirely attributable to the LM, but the paper does not report the decoder's own contrast when fed oracle (ground-truth) prosody tokens versus fed the LM's tokens. Second, the emotion recognition results in Table 3 (Happy 0.014, Sad 0.203, Excited 0.203, Angry 0.021, Neutral 0.063) show small log-probability increases relative to the standard deviations shown (e.g., Happy has sd 0.007 vs mean 0.014; a two-sd interval includes zero), and the paper provides no significance test or multiple-comparison correction for these five emotion words. The 'surprising emergence' claim depends on the measured contrasts being robust and attributable to the LM rather than to decoder priors or encoder artifacts. The paper's own Section 5 limitation notes that the decoder parameters are mostly fixed to StyleTTS2 weights, so the decoder could be smoothing or exaggerating contrasts in a way that the evaluation metrics measure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ProsodyLM, a speech language model pre-trained on text interleaved with word-level prosody tokens (duration, log-F0 range/median/slope, log-energy). The authors claim that this tokenization scheme enables a standard text LLM (Llama-3.1-8B-Instruct, LoRA-tuned on 29.9k hours of Librilight audiobooks) to acquire diverse prosody processing capabilities through pre-training alone, without task-specific fine-tuning. The evaluation covers three dependency types: content-to-prosody (direct and indirect style following, contrastive focus in clarification dialogues), prosody-to-content (emphasis detection, emotion recognition), and prosody-to-prosody (dialogue style continuation), using acoustic metrics, listening tests, and comparisons with group-A (same training pipeline, different tokenizers) and group-B (commercial or stronger systems) baselines.","tokens_in":20206,"tokens_out":5620,"duration_ms":55792,"significance":"If the claims hold, the paper is significant for the speech LM community: it demonstrates that a simple, interpretable prosody tokenization can elicit emerging prosody behaviors from a text LM, and it contributes a reusable evaluation suite covering three prosody-dependency categories. The paper includes objective acoustic measurements, MOS listening tests, ablations (global token, content-to-content capability retention), and audio demos, and the evaluations are zero-shot with respect to the target tasks, so circularity is low. However, the interpretation of the headline results depends on the fidelity of the StyleTTS2-derived encoder/decoder chain and on the statistical support for some of the smaller effects; both need strengthening before the claims can be fully credited.","major_comments":[{"comment":"The central claim of emergent prosody processing is evaluated on synthetic audio produced by a modified StyleTTS2 decoder whose duration and prosody predictors are retrained to condition on the LM's word-level tokens (Eq. 2 in Appendix B). The manuscript does not report the decoder's fidelity: it never quantifies how faithfully the retrained predictors realize a given word-level prosody token sequence, nor does it provide an oracle-token control (feeding ground-truth word-level prosody tokens into the decoder and measuring the resulting F0, duration, and energy contrasts). Without such a control, the large gaps in Table 1 (e.g., direct F0 pair: ProsodyLM 18.51 vs. MIMI-tok 1.86 and StyleTTS2 3.25) cannot be unambiguously attributed to the LM's token choices, because the decoder itself may amplify or attenuate prosodic differences. Please add a decoder-fidelity experiment and, ideally, an oracle-token condition, and also specify the decoder architectures used for the group-A baselines, since the StyleTTS2 decoder cannot directly consume MIMI or GLM4V tokens.","section":"§3.2, Appendix B, Table 1"},{"comment":"The claim that ProsodyLM recognizes all five emotions is not statistically supported for Happy and Angry. The average log-probability increases are 0.014 (sd 0.007) for Happy and 0.021 (sd 0.010) for Angry, so the two-standard-deviation intervals include zero; no significance tests, bootstrap confidence intervals, or multiple-comparison correction across the five emotion words are reported. The paper should provide per-emotion significance testing and discuss these small effect sizes explicitly, or restrict the emotion-recognition claim to the emotions for which effects are robust.","section":"§4.3.2, Table 3"},{"comment":"The contrastive-focus claim (on-focus stress and post-focus compression) is based on average F0 curves plotted without error bars or significance tests, yet the text states that the F0 in the post-focus case is 'always significantly suppressed.' Please add per-condition error bars and pairwise statistical tests (e.g., on-focus vs. pre-focus and post-focus vs. pre-focus for each sentence component) in both the main figure and the full results in Appendix E.2.","section":"§4.2.3, Figure 2, Appendix E.2"}],"minor_comments":[{"comment":"The special token for sentence-level prosody is written inconsistently as '[global]' and '[Global]' in the same paragraph; please unify the notation.","section":"§3.1"},{"comment":"The manuscript contains a placeholder 'Figure xx' in the decoder description; this should be replaced with a real figure reference.","section":"Appendix B"},{"comment":"For the group-A baselines, the paper says they are 'almost the same' as ProsodyLM except for the tokenization, but it never explains how the StyleTTS2 decoder is adapted to MIMI or GLM4V tokens; please clarify the decoding pipeline for each baseline.","section":"§4.1"},{"comment":"In the text, 'classification' should be 'clarification' (e.g., 'during the classification setting' appears in the paragraph about symbol-rate adjustment).","section":"Appendix E.2"},{"comment":"The caption 'Average increase in log output probability emotion recognition' is missing a preposition; it should read '...in log output probability for emotion recognition.'","section":"Table 3 caption"},{"comment":"The sentence 'The percentiles of capping the five dimension are listed below' should be 'the five dimensions'; also, the closing parenthesis after '100' in the cap table appears misplaced.","section":"Appendix A"},{"comment":"Only a few examples of the instruction paraphrases are listed; for reproducibility, please provide the full set of 65 paraphrases or state where they will be made available.","section":"Appendix C"}],"recommendation":"major_revision","confidential_remarks":"This is an interesting and timely paper with a well-structured evaluation covering three prosody-dependency types. The main gap is the missing decoder-fidelity control: because all objective metrics are measured after StyleTTS2-based synthesis, the contribution of the LM versus the decoder is not fully disentangled. This is fixable in revision with an oracle-token experiment and clear baseline-decoder specifications. The small emotion effects for Happy and Angry and the absence of significance testing for Figure 2 also need to be addressed. The manuscript also contains some presentation artifacts (e.g., 'Figure xx') that should be cleaned. I believe the work merits a major revision rather than rejection, provided the authors supply the requested controls and statistics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nHere is my read on ProsodyLM. The central claim is that a text LLM can, through pre-training alone, acquire a broad set of prosody abilities if its input interleaves text with word-level prosody tokens, and that codec-style tokens (MIMI, GLM4V) do not elicit the same under matched conditions. That claim is largely supported by the experiments and is worth taking seriously. The tokenization is a simple, sensible combination of Shechtman's five word-level prosody dimensions with a text LLM vocabulary, and the evaluation spans all three prosody dependencies: content to prosody, prosody to content, and prosody to prosody. The contrastive-focus and dialogue carry-over results are especially striking relative to the codec baselines, and the human listening tests are properly run.\n\nTwo soft spots keep me from being fully convinced.\n\nFirst, attribution. The measured contrasts in generated speech pass through a StyleTTS2-derived decoder whose weights are mostly frozen. The paper never shows a control where oracle ground-truth prosody tokens are fed into that same decoder and the output is measured. Without that control, we do not know how much of the difference between ProsodyLM and, say, StyleTTS2 comes from the LM's token choices versus the decoder's prior on how tokens map to acoustics. That is the single most important missing experiment.\n\nSecond, statistics. The emotion results for happy and angry show mean log-probability increases of 0.014 and 0.021 with standard deviations around 0.007-0.010; a two-SD interval includes zero, and there is no multiple-comparison correction across the five emotion words. The contrastive-focus figures have no error bars or significance tests. The large effects are solid, but the paper leans on the small ones when it says \"surprisingly diverse.\"\n\nI would not call these fatal issues. The core idea is novel, the zero-shot evaluation design is clean, and the baseline comparison, though not perfectly controlled (the codec models use their own decoders), is a reasonable first pass. The paper deserves a serious referee, and the authors should be asked to add the decoder-fidelity control and a statistical pass. I would cite it and bring it to a reading group.","headline":"The word-level prosody token idea is new and the evaluation is broad, but the decoder-fidelity control and the statistics on small emotion effects need strengthening.","tokens_in":20811,"tokens_out":4086,"would_cite":true,"duration_ms":40095,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ProsodyLM shows that word-level prosody tokens let a text LLM learn focus, emotion, and style from pretraining alone.","keywords":["prosody","speech language models","emergent abilities","tokenization","expressive speech synthesis","contrastive focus","emotion recognition","prosody consistency"],"falsifier":"Shuffle the word-level prosody tokens across the words of the reference utterances in the emphasis-detection setup: if the model's emphasis probability lift stays as large as with correctly aligned tokens, the measured understanding is not actually driven by the prosody tokens, and the central claim would collapse.","tokens_in":1647,"feed_emoji":"🗣️","tokens_out":2119,"duration_ms":94527,"temperature":0.7,"pith_summary":"ProsodyLM claims that the reason speech large language models fail to pick up prosody is the tokenization, not the scale or the pretraining data. The paper proposes representing each utterance as its text transcription followed by word-level prosody tokens—five numbers per word for duration, pitch level, pitch range, pitch slope, and energy—and pretraining an 8-billion-parameter text LLM on roughly 30k hours of audiobooks in this hybrid format. After continual pretraining alone, with no task-specific fine-tuning, the model shows measurable prosody understanding and generation: it places contrastive focus on the right word in clarifications, recognizes emphasized words and five acted emotions from input speech, and carries style contrasts across speakers in long contexts. An ordinary LLM can therefore acquire surprisingly diverse prosody processing as an emergent capability if the speech is represented in a form the text model can read.","feed_headline":"Word-level prosody tokens unlock emergent focus, emotion, and style","feed_subtitle":"Pretraining on 30k hours of audiobooks alone teaches focus, emotion, and style continuation.","key_machinery":"The load-bearing object is the prosody token stream: each utterance is a [Text] section followed by a [Prosody] section in which every word is followed by a five-dimensional vector, covering average phone duration, log-F0 range, log-F0 median, log-F0 slope, and log-energy. Each dimension is clipped, normalized, and quantized into 512 shared bins, and an optional global token encodes sentence-level prosody extremity. Because each dimension has a clear high/low meaning and the content is already separated as text, the LM does not have to re-align or re-infer what the tokens mean; the decoder then maps the chosen word-level values back to phone-level durations and frame-level F0 and energy through retrained predictors built on a pretrained TTS decoder.","core_discovery":"The central claim is that interleaving text with explicit word-level prosody tokens is sufficient for a standard text LLM to develop prosody processing capabilities through next-token pretraining alone. The conventional alternative, feeding discrete acoustic or semantic codec tokens into the LLM, does not produce obvious emerging prosody skills at the same data budget. With the proposed scheme, the model learns three dependencies: content determines prosody (style following and contrastive focus), prosody determines content (emphasis and emotion detection shown through relative log-probability increases), and prosody shapes prosody (style copying across dialogue turns). The authors report that the model also slows down in clarification repeats, and that the hybrid tokenization barely increases text perplexity compared with a text-only model trained on the same audiobooks.","pith_inferences":["The mechanism suggests that token legibility, not model scale alone, may be what unlocks prosody emergence; a much smaller LM with the same tokens would be a direct test of that hypothesis.","The paper's five prosody dimensions do not cover voice quality, which the authors state as a limitation; extending the same separable-token idea to voice quality or articulation rate is a natural next step, but it is not yet demonstrated.","The probability-lift metrics used for emphasis and emotion may understate the model's skill when content and prosody conflict; listener forced-choice tests on the synthesized audio would reveal whether the perceived prosody matches the measured token-level behavior.","Because pretraining is audiobook-only, conversational prosody such as question contours, backchannels, and repaired turns remains untested; showing the same emergence on spontaneous dialogue would be the strongest confirmation."],"forward_implications":["Prosody understanding and prosody control can be trained with the same next-token prediction objective used for text, so new prosody benchmarks do not need task-specific fine-tuning.","A speech LM trained this way can read emphasis and emotion from a user's voice and use that to adjust its response, because the Prosody-to-Content dependency is learned during pretraining.","Expressive synthesis can be steered with ordinary style words such as \"quickly\" or \"in a high voice,\" and the model realizes them as measurable pitch, duration, and energy differences.","The same architecture can handle long-form narration and dialogue, maintaining prosody consistency across sentences and speakers without an explicit style-conditioning module.","Content modeling is largely preserved: the hybrid token format raises perplexity on held-out audiobooks by only about two points relative to a text-only model fine-tuned on the same data."],"supporting_citations":[{"why":"Provides the encoder and decoder modules, including the aligner, pitch and energy extractors, and the duration and prosody predictors that convert speech to prosody tokens and back.","marker":"Li et al., 2023"},{"why":"Supplies the speech recognition used to obtain each utterance's text transcription for the [Text] section.","marker":"Radford et al., 2023"},{"why":"Provides the 29.9k hours of audiobooks used for continuous pretraining of ProsodyLM and the group-A baselines.","marker":"Kahn et al., 2020"},{"why":"Defines the five-dimensional word-level prosody vector (duration, F0 range, F0 median, F0 slope, energy) that forms the token vocabulary.","marker":"Shechtman (2023)"},{"why":"Provides the low-rank adaptation method used to continually pretrain the 8B text LLM.","marker":"Hu et al., 2022"},{"why":"Supplies the contrastive-focus phenomenon that the clarification experiments are designed to test.","marker":"Chafe, 1976"},{"why":"Supplies the parallel-emphasis reference utterances used in the emphasis-detection probe.","marker":"de Seyssel et al., 2023"},{"why":"Supplies the acted five-emotion corpus used for the emotion-recognition probe.","marker":"James et al., 2018"},{"why":"Defines the MIMI codec-token baseline that the paper contrasts with its prosody-token scheme.","marker":"D´efossez et al., 2024"}],"fun_headline_variants":["Word-level prosody tokens give LLMs emergent focus and stress","Prosody tokens, not speech tokens, unlock emergent LLM skills","Pretraining on audiobooks alone teaches prosody skills to LLMs","Interleaving text with prosody tokens sparks emergent speech skills","Prosody tokens teach LLMs focus, stress, and emotion"],"cache_read_input_tokens":22784,"weakest_assumption_plain":"The result depends on the audio-to-token and token-to-audio pipeline faithfully preserving real prosody; if that pipeline distorts or compresses the prosody contrasts, the measured skills could be overstated or misattributed.","fun_headline_variants_meta":{"raw":{"variants":["Word-level prosody tokens give LLMs emergent focus and stress","Prosody tokens, not speech tokens, unlock emergent LLM skills","Pretraining on audiobooks alone teaches prosody skills to LLMs","Interleaving text with prosody tokens sparks emergent speech skills","Prosody tokens teach LLMs focus, stress, and emotion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000962,"raw_usage":{"total_tokens":4072,"prompt_tokens":894,"completion_tokens":3178,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":510,"completion_tokens_details":{"reasoning_tokens":3099}},"tokens_in":510,"tokens_out":3178,"duration_ms":23398,"temperature":1.0,"reasoning_tokens":3099,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:49:41.823873+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Shuffle the word-level prosody tokens across the words of the reference utterances in the emphasis-detection setup: if the model's emphasis probability lift stays as large as with correctly aligned tokens, the measured understanding is not actually driven by the prosody tokens, and the central claim would collapse.","supporting_citations":[{"cited_title":"Giveness, contrasitiveness, definiteness, subjects, topics, and point of view","cited_arxiv_id":null,"evidence_quote":"Supplies the contrastive-focus phenomenon that the clarification experiments are designed to test."}],"review_version":2}