TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

Georgi Tinchev; Hao Zhang; Laureano Moro-Velazquez; Thomas Thebaud; Venkatesh Ravichandran

Reviewed by Pith at T0; open to challenge.

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

TurnNat scores turn-taking naturalness by negative log-likelihood of observed voice-activity states under a model trained on natural conversations.

2026-07-03 21:20 UTC pith:LACMJKAM

load-bearing objection TurnNat introduces a likelihood-based metric for turn-taking naturalness using NLL from a causal VAD predictor plus a human-validated perturbation benchmark. the 2 major comments →

arxiv 2607.01345 v2 pith:LACMJKAM submitted 2026-07-01 cs.CL cs.AI

TurnNat: Automatic Evaluation of Turn-Taking Naturalness in Dyadic Spoken Dialogue

Hao Zhang , Thomas Thebaud , Georgi Tinchev , Venkatesh Ravichandran , Laureano Moro-Velazquez This is my paper

classification cs.CL cs.AI

keywords turn-taking naturalnessspoken dialogue evaluationvoice activity predictionlikelihood-based metricdyadic dialogueperturbation benchmarkautomatic evaluationfull-duplex dialogue

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Turn-taking naturalness is central to full-duplex spoken dialogue systems, yet automatic evaluation has been limited because existing methods rely on human judgments or narrow timing metrics that cannot compare different kinds of failures. TurnNat trains a causal model on natural conversations to predict future two-speaker voice-activity states, then treats the negative log-likelihood of the actual observed states as a measure of timing atypicality. These frame-level scores are pooled over turn-taking boundary units taken from utterance onsets and offsets, and the mean and tail values are aggregated into a dialogue-level naturalness score. The authors created a controlled benchmark of paired natural and perturbed clips that human raters judged for naturalness, and experiments show the scores successfully flag unnatural perturbations of many kinds. A sympathetic reader would care because the method supplies a single, scalable metric that works across heterogeneous timing problems without repeated human evaluation.

Core claim

TurnNat is a likelihood-based framework that trains a causal turn-taking prediction model on natural conversations, computes the negative log-likelihood of observed future two-speaker voice-activity states to quantify timing atypicality, pools the values over turn-taking boundary units extracted from utterance boundaries, and aggregates mean and tail scores into a dialogue-level naturalness score; experiments on a controlled perturbation benchmark validated by human judgments show that the resulting scores identify unnatural turn-taking across heterogeneous timing failures.

What carries the argument

A causal turn-taking prediction model trained on natural conversations that estimates future two-speaker voice-activity states, with negative log-likelihood of the observed states serving as the measure of timing atypicality.

Load-bearing premise

That negative log-likelihood of future voice-activity states under a model trained on natural conversations measures timing atypicality in a way that aligns with human naturalness judgments.

What would settle it

A collection of dialogue clips in which human raters judge the timing as natural yet TurnNat assigns high atypicality scores, or clips in which perturbed timing receives lower scores than the corresponding natural versions.

If this is right

Dialogue systems can be compared and improved on turn-taking quality using a single automatic score instead of separate human studies for each timing issue.
Overlaps, gaps, and other timing perturbations can be evaluated and ranked on the same numerical scale.
Training or fine-tuning of full-duplex dialogue models can incorporate the TurnNat score as a direct optimization target.
The approach can be applied to any new two-channel dialogue corpus once the prediction model is retrained on its natural examples.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the underlying prediction model generalizes well, the same scoring procedure could be transferred to new languages or domains by retraining only the voice-activity predictor.
TurnNat scores could be combined with other automatic dialogue metrics such as semantic appropriateness to produce a broader evaluation suite.
Real-time computation of the likelihood during live conversations might allow a system to adjust its own turn-taking behavior on the fly.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

TurnNat introduces a likelihood-based metric for turn-taking naturalness using NLL from a causal VAD predictor plus a human-validated perturbation benchmark.

read the letter

TurnNat gives a unified automatic way to score turn-taking naturalness using negative log-likelihood from a causal voice-activity predictor, and they back it with a human-validated perturbation benchmark.

The new part is the shift away from separate hand-crafted timing features toward a single likelihood score that can catch different kinds of timing problems. Training the model on natural two-speaker sequences and then measuring how unlikely the actual future states are makes sense for detecting atypical turn-taking. Aggregating at onsets and offsets with mean and tail scores is a practical aggregation step. The benchmark of paired clips with controlled perturbations is the part that lets them test whether the scores match human judgments.

The main soft spot is how tightly the NLL tracks actual human naturalness ratings. If the correlation is only moderate or if certain perturbation types are missed, the metric would need more work before it replaces listening tests. The paper would be stronger with more detail on the prediction model architecture and any ablations on the TBU pooling.

This is for teams building full-duplex spoken dialogue systems who want something more scalable than repeated human evaluations. A reader focused on dialogue system evaluation would get concrete ideas from the method and the benchmark construction. It deserves a serious referee because the core idea is coherent and the evaluation setup is testable.

Referee Report

2 major / 0 minor

Summary. The manuscript proposes TurnNat, a likelihood-based framework for automatic evaluation of turn-taking naturalness in two-channel spoken dialogue. A causal model is trained on natural conversations to predict future two-speaker voice-activity states; negative log-likelihood (NLL) of observed future states, pooled over turn-taking boundary units (TBUs) at onsets/offsets and aggregated into mean and tail scores, serves as the naturalness metric. A controlled perturbation benchmark of paired natural/perturbed clips is constructed and human-validated. The central claim is that experiments on this benchmark demonstrate TurnNat successfully identifies unnatural perturbations across heterogeneous timing failures.

Significance. If the empirical results hold, the work would supply a unified, model-based automatic metric capable of comparing heterogeneous timing failures without task-specific rules or direct human rating at inference time. The human-validated perturbation benchmark itself constitutes a reusable contribution for controlled evaluation of turn-taking models.

major comments (2)

[Abstract] Abstract: the claim that 'Experiments on this benchmark show that TurnNat successfully identifies unnatural turn-taking perturbations across heterogeneous timing failures' is unsupported by any quantitative results, tables, figures, error analysis, or baseline comparisons in the manuscript description. This absence is load-bearing for the central empirical claim.
[Abstract / Experiments] The evaluation rests on the assumption that NLL of future VAD states under a model trained on natural data aligns with human naturalness judgments for perturbed dialogues; while a human-validated benchmark is mentioned, no details are supplied on perturbation generation, inter-rater agreement, or correlation between TBU NLL scores and human ratings.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive feedback. We address the major comments point by point below. We agree that the abstract and evaluation sections require strengthening with additional quantitative support and methodological details, and we will revise the manuscript accordingly.

read point-by-point responses

Referee: [Abstract] Abstract: the claim that 'Experiments on this benchmark show that TurnNat successfully identifies unnatural turn-taking perturbations across heterogeneous timing failures' is unsupported by any quantitative results, tables, figures, error analysis, or baseline comparisons in the manuscript description. This absence is load-bearing for the central empirical claim.

Authors: We agree that the abstract claim would be stronger if accompanied by quantitative evidence. The manuscript body contains the relevant experimental results, tables, and figures, but these are not referenced or summarized in the abstract. We will revise the abstract to include key quantitative outcomes (e.g., differences in mean and tail TBU NLL scores between natural and perturbed conditions, along with statistical significance) to directly support the claim. revision: yes
Referee: [Abstract / Experiments] The evaluation rests on the assumption that NLL of future VAD states under a model trained on natural data aligns with human naturalness judgments for perturbed dialogues; while a human-validated benchmark is mentioned, no details are supplied on perturbation generation, inter-rater agreement, or correlation between TBU NLL scores and human ratings.

Authors: We acknowledge the need for greater transparency on the benchmark construction and validation. In the revised manuscript we will expand the relevant section to include: (1) the precise procedure used to generate the controlled perturbations, (2) inter-rater agreement statistics from the human validation study, and (3) quantitative correlation results between the aggregated TBU NLL scores and the collected human naturalness ratings. These additions will make the alignment between model likelihood and human judgments explicit. revision: yes

Circularity Check

0 steps flagged

No significant circularity identified

full rationale

The paper defines TurnNat as a likelihood-based scoring method: a causal model is trained on natural two-speaker VAD sequences, after which NLL of observed future states (aggregated over TBUs) serves as the atypicality measure. This score is then tested for its ability to flag artificially perturbed clips on a human-validated benchmark. No derivation chain, equation, or self-citation reduces the central claim to a tautology or fitted input by construction; the evaluation relies on external human judgments rather than internal re-use of the same fitted values. The approach is therefore self-contained against the supplied benchmark.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Review is based solely on the abstract; full text, methods, and data unavailable.

axioms (1)

domain assumption A causal turn-taking prediction model trained on natural conversations can estimate future two-speaker voice-activity states whose negative log-likelihood measures timing atypicality.
This is the core premise enabling the NLL-based score.

pith-pipeline@v0.9.1-grok · 5710 in / 1193 out tokens · 24605 ms · 2026-07-03T21:20:36.206575+00:00 · methodology

0 comments

read the original abstract

Turn-taking naturalness is central to full-duplex spoken dialogue systems, yet its automatic evaluation remains limited. Existing evaluations often rely on human judgments or behavior-specific timing metrics, making it difficult to compare heterogeneous timing failures within a unified framework. We propose TurnNat, a likelihood-based framework for automatic turn-taking naturalness evaluation in two-channel spoken dialogue. A causal turn-taking prediction model trained on natural conversations estimates future two-speaker voice-activity states, and the negative log-likelihood (NLL) of the observed future activity measures timing atypicality. TurnNat pools frame-level NLLs over turn-taking boundary units (TBUs) extracted from utterance onsets and offsets, and aggregates mean and tail TBU scores into a dialogue-level naturalness score. We further construct a controlled perturbation benchmark of paired natural and perturbed dialogue clips, validated by human naturalness judgments. Experiments on this benchmark show that TurnNat successfully identifies unnatural turn-taking perturbations across heterogeneous timing failures.

Figures

Figures reproduced from arXiv: 2607.01345 by Georgi Tinchev, Hao Zhang, Laureano Moro-Velazquez, Thomas Thebaud, Venkatesh Ravichandran.

**Figure 1.** Figure 1: Overview of the TurnNat framework. TurnNat first extracts VAD-based turn-taking boundary units from the two-channel dialogue, then uses a causal [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models
cs.CL 2026-07 conditional novelty 6.0

An open-source benchmark for speech-to-speech models shows that current systems produce intelligible audio but diverge from human conversational behavior in latency, dialect consistency, emotional entrainment, and prosody.