Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training

Jan Cernocky; Petr Schwarz; Sathvik Udupa; Shinji Watanabe

REVIEW 4 major objections 6 minor 45 references

Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that streaming neural audio codec features, combined with a label-delay training scheme, reduce cutoff errors of spoken-dialogue endpointers by 42.7% (single-stream) and 37.5% (two-stream) at a fixed median latency of…

desk verdict A credible incremental contribution to streaming endpointing, but the headline cutoff reductions depend on an under-specified evaluation protocol that needs a clear validation-set threshold story. read the letter →

arxiv 2506.07081 v2 pith:L4XVFA5U submitted 2025-06-08 cs.SD eess.AS

Sathvik Udupa , Shinji Watanabe , Petr Schwarz , Jan Cernocky This is my paper

classification cs.SDeess.AS

keywords streamingendpointerneuralaudiocodeclabeldelaytrainingturn-takingpredictionlatency-accuracytrade-offmulti-turndialoguecutofferrorspeechlargelanguagemodel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that a streaming endpointer for multi-turn dialogue can be made substantially more accurate at the same response speed by swapping Mel-spectrogram features for pretrained neural audio codec (NAC) tokens and by shifting the training labels a few frames into the future. Such an improvement matters because endpointers decide when the system starts responding: fewer cutoff errors at equal latency means less interrupting the user, and lower latency means snappier dialogue. The headline results at a fixed median latency of 160 ms are a 42.7% relative cutoff-error reduction for the single-stream model and 37.5% for the two-stream model over the Mel-spectrogram baseline without delay. The paper further claims that the same endpointer, sharing codec features with a pretrained codec-based speech language model, reduces that model's median response time by about 1200 ms and its cutoff error by 35%.

What carries the argument

Two mechanisms carry the argument. The first is the streaming neural audio codec as a feature extractor: pretrained codecs (EnCodec, AudioDec, Mimi) quantize audio into discrete codes, and the paper feeds the corresponding codebook vectors into an LSTM endpointer, with causal average pooling to lower high native frame rates; the observation that silence frames show high codebook entropy motivates why these features can separate speech from mid-silence. The second is label-delay training: given labels $Y=(y_1,\dots,y_T)$ and a delay $\tau$, the training target is shifted so that the model predicts the label from $\tau$ frames earlier, with $\tau$ padding labels prepended and the loss masked on those frames; this teaches delayed, high-confidence predictions without adding any inference-time latency. The architectures are a single-stream LSTM that sees a merged user-plus-system signal with system-activity embeddings, and a two-stream LSTM that processes user and system audio through separate channels with shared feature layers.

What would settle it

On a held-out test set with thresholds fixed from a validation split, if the Mimi-plus-label-delay model no longer shows roughly a 43% and 38% relative cutoff-error reduction at 160 ms median latency when both systems are matched at that latency, the central claim would be refuted.

Watch

Extended reading notes

Core claim

On its own terms, the central claim is that pretrained streaming neural audio codecs, especially the Mimi codec with its acoustic-semantic tokens, are a better acoustic representation for streaming endpointing than spectrum-based features, and that a label-delay training objective removes most of the remaining premature-trigger errors. At a median latency of 160 ms, the single-stream Mimi endpointer trained with delay $\tau=2$ reaches a 7.01% cutoff error versus 12.23% for the Mel-spectrogram baseline with no delay, a 42.7% relative reduction; the two-stream Mimi endpointer with $\tau=1$ reaches 4.76% versus 7.61%, a 37.5% relative reduction. The error analysis shows that without delay, cutoffs concentrate on short mid-silences within a user turn, and delay training suppresses those short-pause mistakes. The integration result is that a pretrained speech LLM can be controlled by the endpointer's turn-end prediction through special tokens, cutting median response latency from about 1640 ms to 412 ms and the cutoff rate from 46.5% to 21.2% at sampling temperature 0.7.

Load-bearing premise

The comparison hinges on assuming that matching two endpointers at the same median latency by sweeping the decision threshold is fair, with early triggers counted only as cutoff errors and excluded from the latency statistics.

Editorial extensions

If this is right

If the central claim is correct, replacing Mel-spectrograms with Mimi codec features alone gives about a 30% relative reduction in cutoff error at 160 ms median latency, before any label delay is applied.
Label delay is a parameter-free training change that improves cutoff error for both feature types and both stream configurations, with the best delay differing by architecture ($\tau=2$ single-stream, $\tau=1$ two-stream).
The two-stream architecture is the stronger configuration at a fixed median latency, cutting the best single-stream cutoff error from 7.01% to 4.76% at 160 ms, at the cost of a larger model and slightly higher tail latency.
Because the endpointer and the speech LLM can share the same Mimi encoder, integrating the endpointer with a pretrained codec-based speech LLM needs no LLM retraining and yields faster, less intrusive responses; in simulation, median response latency drops from 1640 ms to 412 ms and cutoff rate from 46.5% to 21.2% at temperature 0.7.
Lower codec frame rates (20-25 Hz) and more quantizer codebooks improve the latency-error trade-off, which is a concrete design rule for future NAC-based endpointers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The label-delay idea is not logically tied to codec features, so a natural extension is to test it with self-supervised speech features or ASR-derived embeddings to isolate how much of the gain comes from the delay versus the representation.
Because the evaluation counts early triggers as cutoff errors and excludes them from latency percentiles, a stricter extension would report full latency distributions including negative latencies and thresholds selected only on development data; this would clarify whether the 42.7% and 37.5% reductions hold outside the chosen operating-point comparison.
The shared-codec integration suggests a deployment pattern in which one frozen codec encoder serves the endpointer, ASR, and the speech LLM; the paper demonstrates this only with a single LLM, so a broader test across codec-based models would show how general the 1200 ms latency gain is.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Bottom line: this is a solid, incremental endpointing paper whose central claim is plausible but whose headline numbers rest on an evaluation choice the paper doesn't fully justify. The two ideas—using neural audio codec features for streaming endpointing, and training with label delay—are new to this task and are presented clearly. The experiments cover single- and two-stream architectures, several codecs, and a Moshi integration that, while simulated, shows a real benefit from sharing Mimi features between endpointer and speech LLM. The label-delay trick is a simple adaptation of delayed-token training, and the paper is honest about its simplicity.

The main soft spot is the evaluation protocol. Section III-C says the decision threshold is swept from 0.7 to 0.99, and Tables II and III report operating points at fixed ep50 of 120/160 ms, but the paper never says the thresholds were chosen on a held-out validation set. If they were selected on the SpokenWOZ test set, the 42.7% and 37.5% relative cutoff reductions are maxima over a swept decision rule, not a fixed deployable operating point. The second part of the protocol is also concerning: negative latencies are excluded from ep50/ep90 and counted only as cutoffs. Because label-delayed models are explicitly trained to fire later, removing their early triggers from the latency median can make them look equal-latency at the same threshold sweep while their premature firings are hidden inside the cutoff rate. That asymmetry could systematically favor the proposed method. This isn't a manufactured flaw; it's the load-bearing premise for the headline claim.

Minor issues: no confidence intervals or significance tests, and the Moshi evaluation is a simulation with 700 segments, not a real duplex interaction. The code is promised but not yet available.

For a reader in spoken dialogue systems, the paper is useful: the NAC comparison alone is a good empirical contribution, and the label-delay idea is worth trying. The evaluation needs to be tightened before the central claim is accepted. I'd send it to review, with a request for validation-set threshold selection, confidence intervals, and code.

Referee Report

4 major / 6 minor

Summary. This paper proposes two additions to streaming speech endpointing: using pretrained streaming neural audio codec (NAC) features (EnCodec, AudioDec, Mimi) as LSTM inputs, and a label-delay training objective that shifts frame labels by τ frames and masks the padded prefix. The authors evaluate single-stream and two-stream LSTM architectures on SpokenWOZ, reporting at fixed median latencies (ep50 = 120/160 ms) reductions in cutoff error: 42.7% relative for single-stream and 37.5% for two-stream versus the Mel-spectrogram baseline at 160 ms, with the largest gains from Mimi features plus label delay. A final experiment integrates the Mimi-based endpointer with the Moshi speech LLM, reporting reduced median response latency and cutoff rate. Code and data preparation are promised for release.

Significance. Assuming the evaluation protocol is sound, the work is a useful empirical contribution: it identifies Mimi NAC features as a strong replacement for Mel-spectrogram features for endpointing, introduces a simple training-time label-delay technique with no inference-time cost, and validates trends across two architectures, three codecs, multiple frame rates/codebooks, and several τ values. The paper is honest about limitations (e.g., the error analysis shows that some errors shift to longer mid-silences) and the code-release commitment is a strength for reproducibility. I found no circularity in the derivation; threshold and τ sweeps are standard hyperparameter selection. The main limitation is that the headline comparison protocol is not fully specified: if thresholds are selected on the test set and ep50 is computed only over positive latencies, the magnitude of the claimed gains is not established. For these reasons, the contribution is promising but needs a methodological fix before the central claim is accepted.

major comments (4)

[III-C, Tables II–III] The decision-threshold sweep is not described as held out. The text says thresholds are swept from 0.7 to 0.99, and Tables II–III report operating points at ep50 = 120/160 ms, but it never states that the thresholds achieving those operating points were chosen on a validation set. The same issue applies to the choice of τ (τ = 2 for single-stream, τ = 1 for two-stream), which also appears to be selected per architecture. If these choices were made on the SpokenWOZ test set, the reported relative reductions (42.7% and 37.5%) are maxima over a swept decision rule and may not represent a fixed deployable threshold. Please specify the validation protocol, report the exact threshold values used for each entry, and, ideally, report the operating-point curves with confidence bands.
[III-C, first paragraph] The definition of ep50 as the median of positive latency values only, with early triggers excluded from latency and counted only as cutoff errors, makes equal-ep50 comparisons unfair across models with different early-trigger rates. A model that triggers early on many turns has those turns removed from its latency distribution, so its conditional median can be artificially low; because label-delayed models are trained to shift predictions later, this censoring can systematically favor them. Please report the signed latency distribution (or at least the early-trigger rate separately) and show that the headline cutoff reductions hold when latency is measured over all turns, not only the late-triggering subset.
[IV-A, IV-B, Tables II–III] All comparisons are point estimates from a single test set, with no confidence intervals or significance tests. The headline 42.7% and 37.5% relative reductions could be within run-to-run or test-set noise. Please add bootstrap confidence intervals over test dialogues, or repeated training runs with different seeds, for at least the key comparisons (Spec τ = 0 vs. Mimi τ = 2/τ = 1 at both ep50 operating points).
[III-D, Table IV] The Moshi integration is evaluated through a simulation in which the baseline detects the end of the query from the known duration of the simulated query segment, while the proposed system uses the predicted endpointer trigger. This is not a like-for-like comparison of end-to-end systems: the baseline has oracle endpoint timing but lacks the suppression/acceleration injections. Please clarify exactly what the baseline models, report the endpointer's trigger latency and whether the <pad>/<unk> injections sometimes cause delayed or missed responses, or temper the practical-utility claim accordingly.

minor comments (6)

[III-A] The word 'offical' should be 'official'.
[II-C, Eq. (1)] The equation omits the range of t; please specify t = 1, ..., T and clarify that the padding label k is used only in the first τ frames.
[IV-A-2] The text states that at a 480 ms worst-case latency the Mimi-based model reduces median latency by 40 ms, but Table II does not directly show a 40 ms median-latency difference at that operating point; please identify the comparison or correct the sentence.
[IV-D, Table IV] The text says median latency dropped from 1640 ms to 412 ms at temperature 0.7, while Table IV reports 1690 ms to 412 ms; please make the text and table consistent.
[Throughout] Spelling is inconsistent (e.g., 'optimising' in Section III-C vs. 'optimize' elsewhere, 'V oice Activity Detection' in the Introduction); please harmonize.
[Figure 3] The caption says the endpoint is denoted by ⋆, but the figure is not described with enough detail to see how the <pad>/<unk> injections relate to the displayed signal; please add a reference or explanation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the reported endpointing gains are empirical comparisons on a held-out test set, not derivations from fitted constants or load-bearing self-citations.

full rationale

The paper's central claims are empirical: it trains LSTM endpointers on SpokenWOZ using frozen, externally pretrained neural codec features (EnCodec, AudioDec, Mimi), applies a label-shifting training scheme (Eq. 1), and measures ep50, ep90, and ep-cutoff at swept decision thresholds. The label delay tau and decision thresholds are hyperparameters selected through standard model selection, and the headline 42.7%/37.5% relative cutoff reductions are read directly from measured test-set metrics in Tables II and III, not computed from the same quantities used to define the model or the metric. No equation in the paper defines its output in terms of its input, and no 'prediction' reduces by construction to a fitted constant. The external benchmarks (SpokenWOZ test set, pretrained codecs, Moshi LLM) are independent of the authors' proposed method. The cited potential issue of selecting thresholds on the test set and the ep50 convention excluding negative latencies concerns evaluation fairness and potential overfitting, not circularity; per the hard rules, that concern belongs under correctness risk, not circularity. Thus the derivation chain is self-contained as an empirical study, and no circularity score above zero is warranted.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim depends on several dataset and feature assumptions, plus hyperparameters (tau, threshold) chosen from experiments. No invented physical entities.

free parameters (5)

label delay tau = 2 (single-stream Mimi), 1 (two-stream Mimi)
Chosen per architecture from experiments; directly drives the reported error reduction.
decision threshold = swept 0.7-0.99; selected for ep50 120/160 ms
Sets the operating point; if selection is on test data, comparisons may be optimistic.
LSTM hidden dimensions = 324 (3-layer) for Spec/EnCodec/AudioDec; 512 (2-layer) for Mimi
Capacity matched to feature dimension; affects results but not the main comparison.
codebook count = 8 for AudioDec and Mimi; ablations shown
More codebooks improve features; chosen based on ablations in Figure 5.
learning rate and epochs = 0.001, 50 epochs
Standard Adam config; no extensive tuning reported.

assumptions (5)

domain assumption Frozen pretrained NAC features retain enough speech and silence information for endpointing
Supported only by entropy observation in Figure 2; no causal proof.
domain assumption Resampling 8 kHz SpokenWoZ audio to 24 kHz preserves information relevant to endpointing
Required for all NAC models; not separately validated.
domain assumption SpokenWoZ with Silero VAD trimming is a representative multi-turn endpointing benchmark
Single dataset; results may not transfer to other domains.
ad hoc to paper Label delay training with padding labels and masked loss is a valid training objective
Introduced in Section II-C; no theoretical justification beyond empirical gains.
domain assumption Moshi simulation with 700 pairs and special-token injection approximates real duplex conversation
Not a live user study; assumes token injection has the same effect in real interactions.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training." pith.science (2026). https://pith.science/paper/L4XVFA5U

@misc{pith2026250607081,
  author       = {Pith},
  title        = {Pith review of: Streaming Endpointer for Spoken Dialogue using Neural Audio Codecs and Label-Delayed Training},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L4XVFA5U}},
  note         = {Machine review of arXiv:2506.07081}
}

read the original abstract

Accurate, low-latency endpointing is crucial for effective spoken dialogue systems. While traditional endpointers often rely on spectrum-based audio features, this work proposes real-time speech endpointing for multi-turn dialogues using streaming, low-bitrate Neural Audio Codec (NAC) features, building upon recent advancements in neural audio codecs. To further reduce cutoff errors, we introduce a novel label delay training scheme. At a fixed median latency of 160 ms, our combined NAC and label delay approach achieves significant relative cutoff error reductions: 42.7% for a single-stream endpointer and 37.5% for a two-stream configuration, compared to baseline methods. Finally, we demonstrate efficient integration with a codec-based pretrained speech large language model, improving its median response time by 1200 ms and reducing its cutoff error by 35%.

Figures

Figures reproduced from arXiv: 2506.07081 by the authors.

**Figure 1.** Endpointing model architectures. (a) Single-stream endpointer: Processes a single, merged audio input. It incorporates system activity information [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Frame level codebook entropy for AudioDec neural audio codec. High [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Our duplex simulation for Moshi speech LLM. Endpoint predicted [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Single stream: Mel-Spectrogram-based endpointer (Spec) compared [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

**Figure 5.** Figure 5: Single stream: Metrics on the AudioDec NAC-based endpointer, NQ [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Single stream: Endpointer performance without label delay ( [PITH_FULL_IMAGE:figures/full_fig_p005_6.png]

**Figure 8.** Figure 8: Two-stream: Endpointer performance without label delay ( [PITH_FULL_IMAGE:figures/full_fig_p006_8.png]

**Figure 7.** Figure 7: Endpoint triggered at 0.5 threshold with and without delay training for single stream endpointer. ⋆ represents the estimated endpoint location, while the vertical dotted line indicates the ground truth endpoint [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 36 canonical work pages

[1]

WavChat: A Survey of Spoken Dialogue Models,

S. Ji, Y . Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Cheng et al. , “WavChat: A Survey of Spoken Dialogue Models,” arXiv preprint arXiv:2411.13577 , 2024

arXiv 2024
[2]

A review of subjective scales measuring the user experience of voice assistants,

L. I. D. Faruk, M. D. Babakerkhell, P. Mongkolnam, V . Chongsupha- jaisiddhi, S. Funilkul, and D. Pal, “A review of subjective scales measuring the user experience of voice assistants,” IEEE Access , 2024

work page 2024
[3]

Gemini: a family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

arXiv 2023
[4]

OpenAI-gpt-4o,

“OpenAI-gpt-4o,” https://openai.com/index/hello-gpt-4o/, accessed: 2025-01-10

work page 2025
[5]

Improved End- of-Query Detection for Streaming Speech Recognition,

M. Shannon, G. Simko, S.-Y . Chang, and C. Parada, “Improved End- of-Query Detection for Streaming Speech Recognition,” in Interspeech. ISCA, Aug. 2017, p. 1909–1913

work page 2017
[6]

V oice activity projection: Self-supervised learning of turn-taking events,

E. Ekstedt and G. Skantze, “V oice activity projection: Self-supervised learning of turn-taking events,” in Interspeech 2022 , 2022, pp. 5190– 5194

work page 2022
[7]

Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics ,

S. Arora, Z. Lu, C.-C. Chiu, R. Pang, and S. Watanabe, “Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics ,” in ICLR, 2025

work page 2025
[8]

Root causes of lost time and user stress in a simple dialog system,

N. G. Ward, A. G. Rivera, K. Ward, and D. G. Novick, “Root causes of lost time and user stress in a simple dialog system,” in Interspeech 2005, 2005, pp. 1565–1568

work page 2005

Show all 45 references

[9]

A statistical model-based voice activity detection,

J. Sohn, N. S. Kim, and W. Sung, “A statistical model-based voice activity detection,” IEEE SPL , vol. 6, no. 1, pp. 1–3, 1999

1999
[10]

A Convolutional Neural Network Smartphone App for Real-Time V oice Activity Detection,

A. Sehgal and N. Kehtarnavaz, “A Convolutional Neural Network Smartphone App for Real-Time V oice Activity Detection,”IEEE Access, vol. 6, pp. 9017–9026, 2018

2018
[11]

Temporal modeling using dilated convolution and gating for voice-activity-detection,

S.-Y . Chang, B. Li, G. Simko, T. N. Sainath, A. Tripathi, A. van den Oord, and O. Vinyals, “Temporal modeling using dilated convolution and gating for voice-activity-detection,” in ICASSP. IEEE, 2018, pp. 5549–5553

2018
[12]

Robust end-of-utterance detection for real-time speech recognition applications,

R. Hariharan, J. Hakkinen, and K. Laurila, “Robust end-of-utterance detection for real-time speech recognition applications,” in ICASSP, vol. 1. IEEE, 2001, pp. 249–252

2001
[13]

Combining acoustic embeddings and decoding features for end-of-utterance detection in real-time far-field speech recognition systems,

R. Maas, A. Rastrow, C. Ma, G. Lan, K. Goehner, G. Tiwari, S. Joseph, and B. Hoffmeister, “Combining acoustic embeddings and decoding features for end-of-utterance detection in real-time far-field speech recognition systems,” in ICASSP. IEEE, 2018, pp. 5544–5548

2018
[14]

Dynamic speech endpoint detection with regression targets,

D. Liang, H. Su, T. Singh, J. Mahadeokar, S. Puri, J. Zhu, E. Thomaz, and M. Seltzer, “Dynamic speech endpoint detection with regression targets,” in ICASSP. IEEE, 2023, pp. 1–5

2023
[15]

SoundStream: An End-to-End Neural Audio Codec,

N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasac- chi, “SoundStream: An End-to-End Neural Audio Codec,” IEEE/ACM Transactions on Audio, Speech, and Language Processing , vol. 30, p. 495–507, 2022

2022
[16]

High Fidelity Neural Audio Compression,

A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High Fidelity Neural Audio Compression,” Transactions on Machine Learning Research , 2023

2023
[17]

Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec,

Y .-C. Wu, I. D. Gebru, D. Markovi ´c, and A. Richard, “Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec,” in ICASSP, 2023, pp. 1–5

2023
[18]

Moshi: a speech-text foundation model for real-time dialogue,

A. D ´efossez, L. Mazar ´e, M. Orsini, A. Royer, P. P ´erez, H. J ´egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” arXiv, no. arXiv:2410.00037, Oct. 2024

2024 arXiv
[19]

Codec-SUPERB: An In-Depth Analysis of Sound Codec Models,

H. Wu, H.-L. Chung, Y .-C. Lin, Y .-K. Wu, X. Chen, Y .-C. Pai, H.-H. Wang, K.-W. Chang, A. H. Liu, and H. yi Lee, “Codec-SUPERB: An In-Depth Analysis of Sound Codec Models,” 2024

2024
[20]

ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech,

J. Shi, J. Tian, Y . Wu, J. weon Jung, J. Q. Yip, Y . Masuyama, W. Chen, Y . Wu, Y . Tang, M. Baali, D. Alharhi, D. Zhang, R. Deng, T. Srivastava, H. Wu, A. H. Liu, B. Raj, Q. Jin, R. Song, and S. Watanabe, “ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs ...

2024 arXiv
[21]

Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations,

K. Dhawan, N. R. Koluguri, A. Juki ´c, R. Langman, J. Balam, and B. Ginsburg, “Codec-ASR: Training Performant Automatic Speech Recognition Systems with Discrete Speech Representations,” in Inter- speech, 2024, pp. 2574–2578

2024
[22]

High-Fidelity Simultaneous Speech-To-Speech Translation,

T. Labiausse, L. Mazar ´e, E. Grave, P. P ´erez, A. D ´efossez, and N. Zeghidour, “High-Fidelity Simultaneous Speech-To-Speech Translation,” 2025. [Online]. Available: https://arxiv.org/abs/2502.03382

2025 arXiv
[23]

Self-supervised speech representation learning: A review,

A. Mohamed, H.-y. Lee, L. Borgholt, J. D. Havtorn, J. Edin, C. Igel, K. Kirchhoff, S.-W. Li, K. Livescu, L. Maaløe et al. , “Self-supervised speech representation learning: A review,” IEEE Journal of Selected Topics in Signal Processing , vol. 16, no. 6, pp. 1179–1210, 2022

2022
[24]

Hilcodec: High fidelity and lightweight neural audio codec,

S. Ahn, B. J. Woo, M. H. Han, C. Moon, and N. S. Kim, “Hilcodec: High fidelity and lightweight neural audio codec,” arXiv preprint arXiv:2405.04752, 2024

2024 arXiv
[25]

End-to-end speech endpoint detection utilizing acoustic and language modeling knowledge for online low- latency speech recognition,

I. Hwang and J.-H. Chang, “End-to-end speech endpoint detection utilizing acoustic and language modeling knowledge for online low- latency speech recognition,” IEEE access , vol. 8, pp. 161 109–161 123, 2020

2020
[26]

Joint endpointing and decoding with end-to-end models,

S.-Y . Chang, R. Prabhavalkar, Y . He, T. N. Sainath, and G. Simko, “Joint endpointing and decoding with end-to-end models,” in ICASSP. IEEE, 2019, pp. 5626–5630

2019
[27]

Turn-Taking Prediction for Natural Conversational Speech,

S.-Y . Chang, B. Li, T. Sainath, C. Zhang, T. Strohman, Q. Liang, and Y . He, “Turn-Taking Prediction for Natural Conversational Speech,” in Interspeech, 2022, pp. 1821–1825

2022
[28]

Towards fast and accurate streaming end-to-end ASR,

B. Li, S.-y. Chang, T. N. Sainath, R. Pang, Y . He, T. Strohman, and Y . Wu, “Towards fast and accurate streaming end-to-end ASR,” in ICASSP. IEEE, 2020, pp. 6069–6073

2020
[29]

Streaming Automatic Speech Recognition with Re-blocking Processing Based on Integrated V oice Activity Detection

Y . Sudo, M. Shakeel, K. Nakadai, J. Shi, and S. Watanabe, “Streaming Automatic Speech Recognition with Re-blocking Processing Based on Integrated V oice Activity Detection.” in INTERSPEECH, 2022, pp. 4641–4645

2022
[30]

Two-Pass Endpoint Detection for Speech Recognition,

A. Raju, A. Khare, D. He, I. Sklyar, L. Chen, S. Alptekin, V . A. Trinh, Z. Zhang, C. Vaz, V . Ravichandranet al., “Two-Pass Endpoint Detection for Speech Recognition,” in ASRU. IEEE, 2023, pp. 1–8

2023
[31]

Towards Accurate and Real-Time End-of-Speech Estimation,

Y . Fan, C. Vaz, D. He, J. Heymann, V . A. Trinh, Z. Zhang, and V . Ravichandran, “Towards Accurate and Real-Time End-of-Speech Estimation,” in ICASSP, 2023, pp. 1–5

2023
[32]

Text Injec- tion for Capitalization and Turn-Taking Prediction in Speech Models,

S. Bijwadia, S.-Y . Chang, W. Wang, Z. Meng, and H. Zhang, “Text Injec- tion for Capitalization and Turn-Taking Prediction in Speech Models,” in Interspeech, 2023, pp. 1409–1413

2023
[33]

Multilingual turn-taking prediction using voice activity projection,

K. Inoue, B. Jiang, E. Ekstedt, T. Kawahara, and G. Skantze, “Multilingual turn-taking prediction using voice activity projection,” in Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) , N....

2024
[34]

Yeah, un, oh: Continuous and real-time backchannel prediction with fine-tuning of voice activity projection,

K. Inoue, D. Lala, G. Skantze, and T. Kawahara, “Yeah, un, oh: Continuous and real-time backchannel prediction with fine-tuning of voice activity projection,” in Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Ling...

2025
[35]

Turn-Taking and Backchannel Prediction with Acoustic and Large Language Model Fusion,

J. Wang, L. Chen, A. Khare, A. Raju, P. Dheram, D. He, M. Wu, A. Stol- cke, and V . Ravichandran, “Turn-Taking and Backchannel Prediction with Acoustic and Large Language Model Fusion,” in ICASSP, 2024, pp. 12 121–12 125

2024
[36]

End-to-end automatic speech recognition integrated with CTC-based voice activity detection,

T. Yoshimura, T. Hayashi, K. Takeda, and S. Watanabe, “End-to-end automatic speech recognition integrated with CTC-based voice activity detection,” in ICASSP. IEEE, 2020, pp. 6999–7003

2020
[37]

Text- free prosody-aware generative spoken language modeling,

E. Kharitonov, A. Lee, A. Polyak, Y . Adi, J. Copet, K. Lakhotia, T. A. Nguyen, M. Riviere, A. Mohamed, E. Dupoux, and W.-N. Hsu, “Text- free prosody-aware generative spoken language modeling,” in ACL, S. Muresan, P. Nakov, and A. Villavicencio, Eds., Dublin, Ireland, May 2022...

2022
[38]

Simple and controllable music generation,

J. Copet, F. Kreuk, I. Gat, T. Remez, D. Kant, G. Synnaeve, Y . Adi, and A. D ´efossez, “Simple and controllable music generation,” NeurIPS, vol. 36, 2024

2024
[39]

SpokenWOZ: a large-scale speech-text benchmark for spoken task-oriented dialogue agents,

S. Si, W. Ma, H. Gao, Y . Wu, T.-E. Lin, Y . Dai, H. Li, R. Yan, F. Huang, and Y . Li, “SpokenWOZ: a large-scale speech-text benchmark for spoken task-oriented dialogue agents,” in NeurIPS, 2024

2024
[40]

Silero V AD: pre-trained enterprise-grade V oice Activity Detector (V AD), Number Detector and Language Classifier,

S. Team, “Silero V AD: pre-trained enterprise-grade V oice Activity Detector (V AD), Number Detector and Language Classifier,” https: //github.com/snakers4/silero-vad, 2024

2024
[41]

PyTorch: an imperative style, high- performance deep learning library,

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. K ¨opf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: an imperative style, high-...

2019
[42]

Transformers: State-of-the-art natural language processing,

T. Wolf, L. Debut, V . Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, J. Davison, S. Shleifer, P. von Platen, C. Ma, Y . Jernite, J. Plu, C. Xu, T. L. Scao, S. Gugger, M. Drame, Q. Lhoest, and A. M. Rush, “Transformers: State-of-the-art nat...

2020
[43]

Modeling turn-taking in human-to-human spoken dialogue datasets us- ing self-supervised features,

E. Morais, M. Damasceno, H. Aronowitz, A. Satt, and R. Hoory, “Modeling turn-taking in human-to-human spoken dialogue datasets us- ing self-supervised features,” in ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) . IEEE, 2023, pp. 1–5

2023
[44]

Llm- enhanced dialogue management for full-duplex spoken dialogue sys- tems,

H. Zhang, W. Li, R. Chen, V . Kothapally, M. Yu, and D. Yu, “Llm- enhanced dialogue management for full-duplex spoken dialogue sys- tems,” arXiv preprint arXiv:2502.14145 , 2025

2025 arXiv
[45]

Minmo: A multimodal large language model for seamless voice interaction,

Q. Chen, Y . Chen, Y . Chen, M. Chen, Y . Chen, C. Deng, Z. Du, R. Gao, C. Gao, Z. Gao et al., “Minmo: A multimodal large language model for seamless voice interaction,” arXiv preprint arXiv:2501.06282 , 2025

2025 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

WavChat: A Survey of Spoken Dialogue Models,

S. Ji, Y . Chen, M. Fang, J. Zuo, J. Lu, H. Wang, Z. Jiang, L. Zhou, S. Liu, X. Cheng et al. , “WavChat: A Survey of Spoken Dialogue Models,” arXiv preprint arXiv:2411.13577 , 2024

arXiv 2024

[2] [2]

A review of subjective scales measuring the user experience of voice assistants,

L. I. D. Faruk, M. D. Babakerkhell, P. Mongkolnam, V . Chongsupha- jaisiddhi, S. Funilkul, and D. Pal, “A review of subjective scales measuring the user experience of voice assistants,” IEEE Access , 2024

work page 2024

[3] [3]

Gemini: a family of highly capable multimodal models,

G. Team, R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican et al., “Gemini: a family of highly capable multimodal models,”arXiv preprint arXiv:2312.11805, 2023

arXiv 2023

[4] [4]

OpenAI-gpt-4o,

“OpenAI-gpt-4o,” https://openai.com/index/hello-gpt-4o/, accessed: 2025-01-10

work page 2025

[5] [5]

Improved End- of-Query Detection for Streaming Speech Recognition,

M. Shannon, G. Simko, S.-Y . Chang, and C. Parada, “Improved End- of-Query Detection for Streaming Speech Recognition,” in Interspeech. ISCA, Aug. 2017, p. 1909–1913

work page 2017

[6] [6]

V oice activity projection: Self-supervised learning of turn-taking events,

E. Ekstedt and G. Skantze, “V oice activity projection: Self-supervised learning of turn-taking events,” in Interspeech 2022 , 2022, pp. 5190– 5194

work page 2022

[7] [7]

Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics ,

S. Arora, Z. Lu, C.-C. Chiu, R. Pang, and S. Watanabe, “Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics ,” in ICLR, 2025

work page 2025

[8] [8]

Root causes of lost time and user stress in a simple dialog system,

N. G. Ward, A. G. Rivera, K. Ward, and D. G. Novick, “Root causes of lost time and user stress in a simple dialog system,” in Interspeech 2005, 2005, pp. 1565–1568

work page 2005