{"id":"2f27e7fd-f2a4-4686-934c-fc7f0ea9e7c8","arxiv_id":"2507.03912","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Combining frozen speech SSL/Whisper features with phoneme-level BERT features improves automatic Japanese prosody label prediction over either input alone.","lead":"This paper combines speech foundation model features (HuBERT or Whisper) with phoneme-level BERT text features to automatically annotate Japanese prosody labels such as pitch accents and phrase breaks. On the CSJ corpus, the best combined model reaches 89.8% accent accuracy, 93.2% high-low pitch accuracy, and 94.3% break-index accuracy.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Fusion gains over speech-only input are 0.4–1.5 percentage points and are reported without error bars or repeated runs; the claimed superiority over either input alone may rest on run-to-run noise.","rationale":"The reader's weakest-assumption pick was the VAE-based phoneme alignment used to average speech features (Section 5.3). That concern is real for transferring the method to other aligners, but it is not the most direct threat to the central relative claim: both the speech-only and fusion conditions use the same alignment, so misalignment would affect both arms without necessarily invalidating the comparison. The more load-bearing issue is statistical robustness. The headline numbers are single-run accuracies on a small evaluation set, and the margin of fusion over the strongest single-modality baseline is very small: 0.8 points for ACC, 0.4 for HL, and 1.5 for BI. Without repeated runs or confidence intervals, the abstract's claim that the combination 'enhanced prediction accuracy' over either input alone is not yet distinguished from optimizer/sampling noise. The proposed test, five seeds plus paired bootstrap intervals on the fixed evaluation split, would settle whether the margins are stable. If the margins hold, the central claim is credible; if not, the paper should be revised to claim only improvement over text-only prediction or to add uncertainty quantification. This does not change the reader's CONDITIONAL verdict, since the same missing evidence was part of the reader's rationale.","tokens_in":12326,"tokens_out":5856,"duration_ms":67658,"concrete_test":"Retrain the annotation model from scratch with 5 random seeds for the three key conditions in Table 1 (HuBERT + PnG BERT, HuBERT with no linguistic input, and PnG BERT with no acoustic input), keeping data split and hyperparameters fixed. Report mean and standard deviation of ACC/HL/BI/PAU on the fixed evaluation set, and compute paired bootstrap 95% confidence intervals for fusion-minus-speech-only and fusion-minus-linguistic-only differences. If the ACC or HL interval for fusion minus HuBERT-only includes zero, the abstract's claim that fusion beats either input alone is not supported by the presented evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that concatenating speech and linguistic foundation-model features beats either modality alone. In Table 1, the most relevant contrast is HuBERT-base alone vs. HuBERT-base + PnG BERT: ACC 0.890 vs. 0.898, HL 0.928 vs. 0.932, BI 0.928 vs. 0.943, PAU 0.987 vs. 0.987. The gains are only 0.4–1.5 percentage points, and for HL the gap is 0.4 points. The evaluation set contains 896 utterances from 8 talks, every reported number comes from a single run, and no standard deviation, confidence interval, seed variation, or significance test is given. Since the annotation model is randomly initialized and trained for 100k steps, run-to-run variation of this magnitude is plausible. The no-acoustic PnG BERT baseline is much lower, but the abstract's 'either a speech or linguistic input alone' includes the speech-only condition, where the improvement is small. If the fusion-vs-speech gap is noise, the headline claim is not established at the precision claimed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automatic prosody annotation model for Japanese that concatenates frozen features from a speech foundation model (SSL such as HuBERT or Whisper encoder) with frozen features from a phoneme-input linguistic foundation model (PnG BERT or PL-BERT), and trains a small CNN annotation head with multi-task losses for accent symbols, high-low accents, break indices, and pause presence. The speech features are phoneme-averaged using VAE-based alignment. Experiments on CSJ core data report accuracy and macro F1 on a held-out split, with headline numbers of 89.8% for accent labels, 93.2% for high-low accents, and 94.3% for break indices using HuBERT-base plus PnG BERT. Additional experiments compare several SSL and Whisper models, and the paper includes confusion matrices, an analysis of layer weights, and a qualitative example of predicted labels.","tokens_in":12449,"tokens_out":5167,"duration_ms":49879,"significance":"The paper is carefully designed in several respects: target labels are external X-JToBI manual annotations, the evaluation is on a held-out split, the foundation models are frozen so that only the annotation head and layer-combination weights are fitted, and the ablations over acoustic and linguistic input types are systematic. If the fusion result is real, the approach is practically useful for building prosody-controllable TTS without large amounts of prosodically labeled data and provides a clean recipe for combining speech and linguistic foundation models. However, the quantitative evidence for the central claim is fragile: all reported numbers come from a single training run on a small evaluation set, and the fusion gains over speech-only input are small. The paper's value therefore depends on whether the main effect survives repeated-run or resampled evaluation.","major_comments":[{"comment":"The central claim that the combination beats either modality alone rests on differences of only 0.4–1.5 accuracy points over the speech-only baseline. For example, with HuBERT-base, adding PnG BERT changes ACC from 0.890 to 0.898, HL from 0.928 to 0.932, and BI from 0.928 to 0.943, while the PAU score is unchanged. Every number is from a single run on an 896-utterance evaluation set, with no standard deviation, confidence interval, seed variation, or significance test. Since the annotation model is randomly initialized and trained for 100k steps, run-to-run variation of this magnitude is plausible. The paper should report results over multiple training seeds or provide bootstrap/talk-level confidence intervals and a paired significance test (e.g., McNemar or a talk-wise paired test) for the fusion-versus-speech contrast. Without this, the abstract's statement that the combination 'enhanced the prediction accuracy compared to using either a speech or linguistic input alone' is not established at the claimed precision. The same limitation affects the model rankings in Section 5.6 and Tables 2 and 3, where differences as small as 0.001–0.01 are interpreted as meaningful.","section":"Section 5.4, Table 1"},{"comment":"All acoustic features are computed by phoneme-averaging frame-level SSL or Whisper features using phoneme durations from a single VAE-based alignment method. The paper provides no analysis of how alignment errors affect the fusion result, even though the entire acoustic pathway depends on these boundaries. If the aligner is systematically biased, the speech-model contribution could be distorted, and the reported fusion gains might not transfer to another alignment method or to cases where phoneme transcriptions are not available. Please add a sensitivity analysis with an alternative aligner or an oracle alignment, or at least report alignment error rates on the evaluation set, to support the generality of the proposed phoneme-averaging approach.","section":"Section 5.3"}],"minor_comments":[{"comment":"The text contains the typo 'V AE-based alignment'; the extra space before 'AE' should be removed.","section":"Section 5.3"},{"comment":"The figure caption labels PnG BERT as 'phoneme+text', but the text and Figure 1 define PnG BERT as using phonemes and graphemes; please use consistent terminology.","section":"Figure 3"},{"comment":"The phrase 'SSL-based linguistic models such as BERT' is confusing because the paper elsewhere uses 'SSL' to refer to speech self-supervised models; BERT can be described as a pretrained transformer model rather than an 'SSL-based linguistic model'.","section":"Section 2"},{"comment":"The paper states that all labels are mora-level, but BI and PAU are boundary-level events; please clarify how boundary labels are attached to mora-core phonemes when computing losses.","section":"Section 5.2"},{"comment":"The layer-weight optimization is not described; please state whether the weights are softmax-normalized or otherwise constrained, and add this detail to the experimental conditions.","section":"Section 5.7, Figure 10"},{"comment":"The paper does not compare with prior automatic prosody annotation systems (e.g., [7], [8], [12]) on the same data; adding such a comparison would help position the practical contribution, even though the internal fusion comparison is the main claim.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is statistical: the headline fusion gains are small and unreplicated. The experimental design is otherwise sound and there is no apparent circularity, since the labels are external manual annotations and evaluation is held out. I would not reject; if the authors provide repeated-run or bootstrap evidence that the fusion-versus-speech difference is stable, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid, incremental empirical paper. It concatenates frozen speech SSL/Whisper features with phoneme-input BERT features (PnG BERT, PL-BERT) to predict four X-JToBI-derived prosody label types on CSJ. That exact combination is new in the literature, though both ingredients are established. The experiments are carefully laid out: external manual labels, held-out split, frozen foundation models, multi-task loss, ablations across acoustic inputs (HuBERT, Whisper, melspec, F0) and linguistic inputs (PnG BERT, PL-BERT, one-hot, none), plus confusion matrices and layer-weight analysis. That is more thorough than most Interspeech-scale papers, and the writing is clear.\n\nWhat it does well: the confusion matrices actually support the narrative—PnG BERT helps word-boundary decisions (BI 0 vs 1, filled pauses), HuBERT helps boundary pitch movements and pause presence. The layer-weight plot is a nice sanity check. The comparison across many SSL and Whisper sizes (Tables 2–3) is useful for practitioners. No circularity: labels are external, evaluation is held-out.\n\nSoft spots, in proportion: the headline claim—fusion beats either modality alone—rests on a contrast where the gain over the speech-only HuBERT baseline is 0.4–1.5 points (ACC 0.890→0.898, HL 0.928→0.932, BI 0.928→0.943, PAU tied). Every number comes from a single run on 896 utterances from 8 talks, with no error bars, no significance test, and no comparison to prior prosody-annotation systems (e.g., the Whisper-based sequence model of Shirahata et al. [8]). Run-to-run noise of that magnitude is plausible for a randomly initialized 6-layer CNN trained 100k steps. So the abstract slightly overstates the strength of evidence; the table is honest, but the claim is not established at the precision implied. The dependence on the VAE alignment from the author's own prior work is also a real transfer risk: if alignment is systematically off, the phoneme-averaged acoustic features degrade, and the fusion gain may not reproduce with a different aligner. That is a legitimate concern, not a fatal one.\n\nBottom line: the paper is a useful, honest incremental contribution for Japanese TTS prosody annotation. It deserves a serious referee. The revision should add repeated runs or error bars, compare against at least one prior system, and ideally test with an off-the-shelf aligner. I'd bring it to reading group only if someone is working on prosody labeling; otherwise it's a cite-for-benchmark paper.","headline":"A clean, incremental fusion study for Japanese prosody labeling; the central fusion benefit over speech-only features is small and unquantified, but the paper is honest and deserves refereeing.","tokens_in":13064,"tokens_out":1868,"would_cite":true,"duration_ms":19599,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Combining frozen speech and linguistic foundation models improves automatic Japanese prosody label prediction beyond either modality alone, reaching 89.8% accuracy on accent labels, 93.2% on high-low accents, and 94.3% on break indices.","keywords":["prosody labeling","automatic prosody annotation","pitch accent","break index","PnG BERT","PL-BERT","HuBERT","Whisper"],"falsifier":"Run the same fused system on the same CSJ evaluation set but replace the VAE-based alignment with a standard forced aligner (or with oracle phoneme boundaries from CSJ); if the accuracy gap between the fused and speech-only systems narrows substantially, the fusion benefit depends on the alignment method rather than on the foundation models themselves.","tokens_in":12040,"feed_emoji":"🎙️","tokens_out":5348,"duration_ms":50043,"temperature":0.7,"pith_summary":"This paper proposes an automatic prosody annotation system that predicts phoneme-level pitch accents, high-low accents, break indices, and pause presence from Japanese speech. Its central claim is that concatenating features from a frozen speech foundation model (HuBERT, wav2vec2, WavLM, or Whisper) with features from a phoneme-input linguistic foundation model (PnG BERT or PL-BERT) gives higher label prediction accuracy than either modality alone. On the Corpus of Spontaneous Japanese, the best combination reaches 89.8% accuracy for accent symbols, 93.2% for high-low accents, and 94.3% for break indices. The intended use is to generate reliable prosodic labels automatically for training prosody-controllable text-to-speech systems.","feed_headline":"Fusing speech and text models tops Japanese prosody labeling","feed_subtitle":"Combining HuBERT audio with PnG BERT text hits 94.3% break-index and 89.8% accent accuracy.","key_machinery":"The mechanism is a frozen-feature fusion pipeline: a speech waveform passes through a frozen SSL or Whisper encoder, frame-level hidden states are averaged over phoneme intervals using VAE-based alignment durations, and the resulting phoneme-level acoustic vector is concatenated with the corresponding phoneme-level representation from a frozen phoneme-input BERT model. A weighted sum over encoder layers, learned during training, selects informative layers, and a small CNN annotation model predicts four prosodic label types under multi-task learning.","core_discovery":"The paper establishes that speech and linguistic foundation models carry complementary information for Japanese prosody annotation. Acoustic features, obtained by averaging SSL or Whisper encoder hidden layers over phoneme intervals, capture boundary pitch movements, break-index confusions, and pause presence that text alone cannot provide. Linguistic features from PnG BERT or PL-BERT improve word-boundary distinctions, filled pauses, and disfluencies. Combining the two, with a weighted sum of hidden layers feeding a six-layer CNN, outperforms either modality alone and also beats traditional mel-spectrogram and F0 features by a clear margin.","pith_inferences":["If the same fusion pattern generalizes to other languages, concatenating speech and linguistic foundation models could become a default recipe for prosody labeling; the paper only tests Japanese, so this remains an open extrapolation.","The learned layer weights indicate later HuBERT layers (8-10) carry most prosodic information; a testable extension is whether this layer-selection pattern transfers across corpora and languages.","Because all foundation models are frozen, the approach is parameter-efficient; fine-tuning the acoustic or linguistic models might boost accuracy further but would complicate training and risk overfitting the limited labeled data."],"forward_implications":["Automatic prosodic labels accurate enough for training prosody-controllable TTS can be derived from waveform plus text, reducing reliance on expert manual annotation.","Speech foundation models trained on Japanese yield higher accuracy than English or multilingual SSL models, so language-matched pretraining matters for prosody labeling.","Phoneme-input BERT alone predicts accent symbols at 82.5% accuracy, so text-only prosody prediction for TTS also benefits from linguistic foundation models.","Boundary pitch movements and pause presence are largely acoustic in nature; speech features are necessary for these labels, while linguistic models mainly help word-boundary and filled-pause decisions."],"supporting_citations":[{"why":"Supplies the VAE-based alignment that yields phoneme durations for averaging acoustic features.","marker":"[35]"},{"why":"Provides PnG BERT, the phoneme-plus-grapheme linguistic foundation model used in the main experiments.","marker":"[23]"},{"why":"Provides PL-BERT, the phoneme-only linguistic BERT used as an alternative text-side input.","marker":"[24]"},{"why":"Whisper's encoder serves as the ASR-based acoustic foundation model in the comparison.","marker":"[15]"},{"why":"HuBERT is the primary SSL acoustic model whose hidden layers are fused with linguistic features.","marker":"[27]"},{"why":"wav2vec2.0 is an SSL baseline acoustic model evaluated in the model-comparison table.","marker":"[16]"},{"why":"WavLM is another SSL acoustic baseline evaluated in the comparison.","marker":"[28]"},{"why":"Prior automatic prosody annotation work with Whisper that this paper differentiates from.","marker":"[8]"},{"why":"Corpus of Spontaneous Japanese supplies the speech and manually annotated prosodic labels.","marker":"[26]"},{"why":"X-JToBI defines the break-index annotation scheme converted to mora-level labels.","marker":"[42]"}],"fun_headline_variants":["Fusing speech and text models lifts Japanese prosody","Audio plus language features beat single-model prosody","Merging SSL and BERT features leads Japanese prosody","Speech and text fusion hits high accuracy on prosody"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The accuracy of the speech-side features rests on the VAE-based phoneme alignment being correct, because frame-level acoustic features are averaged over the aligned phoneme intervals; if those durations are systematically wrong, the acoustic contribution is distorted and the reported fusion gain may not survive.","fun_headline_variants_meta":{"raw":{"variants":["Fusing speech and text models lifts Japanese prosody","Audio plus language features beat single-model prosody","Merging SSL and BERT features leads Japanese prosody","Speech and text fusion hits high accuracy on prosody"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000184,"raw_usage":{"total_tokens":1256,"prompt_tokens":824,"completion_tokens":432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":440,"completion_tokens_details":{"reasoning_tokens":368}},"tokens_in":440,"tokens_out":432,"duration_ms":5097,"temperature":1.0,"reasoning_tokens":368,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T19:59:40.337910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same fused system on the same CSJ evaluation set but replace the VAE-based alignment with a standard forced aligner (or with oracle phoneme boundaries from CSJ); if the accuracy gap between the fused and speech-only systems narrows substantially, the fusion benefit depends on the alignment method rather than on the foundation models themselves.","supporting_citations":[{"cited_title":"V AE-based phoneme alignment using gradient an- nealing and SSL acoustic features,","cited_arxiv_id":null,"evidence_quote":"Supplies the VAE-based alignment that yields phoneme durations for averaging acoustic features."},{"cited_title":"wav2vec 2.0: A framework for self-supervised learning of speech representa- tions,","cited_arxiv_id":null,"evidence_quote":"Provides PnG BERT, the phoneme-plus-grapheme linguistic foundation model used in the main experiments."},{"cited_title":"Pre-trained text embeddings for enhanced text-to- speech synthesis,","cited_arxiv_id":null,"evidence_quote":"Provides PL-BERT, the phoneme-only linguistic BERT used as an alternative text-side input."},{"cited_title":"Audio- conditioned phonemic and prosodic annotation for building text- to-speech models from unlabeled speech data,","cited_arxiv_id":null,"evidence_quote":"Whisper's encoder serves as the ASR-based acoustic foundation model in the comparison."},{"cited_title":"Im- proving prosody modelling with cross-utterance BERT embed- dings for end-to-end speech synthesis,","cited_arxiv_id":null,"evidence_quote":"HuBERT is the primary SSL acoustic model whose hidden layers are fused with linguistic features."},{"cited_title":"Low-resourced phonetic and prosodic feature estimation with self-supervised-learning-based acoustic modeling,","cited_arxiv_id":null,"evidence_quote":"wav2vec2.0 is an SSL baseline acoustic model evaluated in the model-comparison table."},{"cited_title":"WavLM: Large-scale self-supervised pre-training for full stack speech processing,","cited_arxiv_id":null,"evidence_quote":"WavLM is another SSL acoustic baseline evaluated in the comparison."},{"cited_title":"Mora-level prosody prediction for text-to-speech us- ing japanese BERT without accentual labels,","cited_arxiv_id":null,"evidence_quote":"Prior automatic prosody annotation work with Whisper that this paper differentiates from."},{"cited_title":"Improving the prosody of RNN-based English text-to-speech synthesis by incorporating a BERT model,","cited_arxiv_id":null,"evidence_quote":"Corpus of Spontaneous Japanese supplies the speech and manually annotated prosodic labels."},{"cited_title":"X-JToBI: an extended J-ToBI for spontaneous speech,","cited_arxiv_id":null,"evidence_quote":"X-JToBI defines the break-index annotation scheme converted to mora-level labels."}],"review_version":1}