Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents

Shwetak Patel; Vidya Srinivas; Vikram Iyer; Zachary Englhardt

REVIEW 4 major objections 6 minor 5 references

Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents

T0 review · 4 major / 6 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 360M-parameter on-device model can start speaking immediately and weave in streamed knowledge from a large backend model, keeping time-to-first-response under 200ms and raising NaturalQuestions accuracy from 10% to 46–52%.

desk verdict The conversational-infill idea is genuinely new and the TTFT results are real, but the arXiv abstract overclaims the body (7 models, user study, 6.3% gap) and the Pareto claim rests on time-to-first-token, not time to a useful answer. read the letter →

arxiv 2511.07397 v2 pith:4A7FXM56 submitted 2025-11-10 cs.CL

Vidya Srinivas , Zachary Englhardt , Shwetak Patel , Vikram Iyer This is my paper

classification cs.CL

keywords conversationalinfillvoiceagentsstreamingknowledgetransferon-deviceinferencelatencyhidingmodelcollaborationNaturalQuestionssyntheticdialogue

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether voice agents must choose between responding in milliseconds and reasoning with a large language model. It answers with 'conversational infill,' a task in which a small on-device model generates the user-facing conversation while a slower, more capable backend streams knowledge chunks that the small model folds into its speech as they arrive. Training a 360M model on a synthetic, entailment-filtered corpus produces ConvFill, which sustains sub-200ms time-to-first-response—comparable to standalone small models—and improves question-answering accuracy on NaturalQuestions from 10% to 46–52% depending on the backend, although it does not match the backend's own 69–80%. The paper positions this as a new point on the latency-capability frontier: voice agents could combine the immediacy of on-device models with the world knowledge of cloud foundation models, hiding any high-latency reasoner behind a responsive talker.

What carries the argument

The load-bearing device is the interleaved knowledge-stream conditioning format. At each turn the model is prompted with the user utterance, a rolling list of backend knowledge chunks and special <|sil|> silence tokens (injected every 1 second when no chunk has arrived), and its own previously generated conversational phrases; it is trained to predict the next phrase. This lets the small model generate immediately, using silence tokens as placeholders for 'thinking time' and knowledge chunks as factual anchors, without needing the backend to produce a full answer before speech begins. The training corpus is synthetic: GPT-4o generates 5,997 goal-directed conversations across six domains with

What would settle it

Re-run the evaluation with the knowledge stream replaced by unrelated sentences (or by silence throughout). If ConvFill's NaturalQuestions accuracy stays near the 10% baseline, the gain demonstrably comes from the streamed knowledge; if accuracy rises substantially without informative chunks, the effect is an artifact of fine-tuning on the synthetic corpus, not the proposed knowledge transfer.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a small model can produce fluent, contextually grounded conversational phrases in real time while receiving external knowledge from a backend model, and that this ability can be learned from data. The task is defined formally: the small model sees the user utterance, a stream of knowledge chunks and silence tokens, and its own previous phrases, and must generate the next phrase. Its implementation, ConvFill (fine-tuned SmolLM2-360M), achieves 0.16–0.17s mean time-to-first-token with three different backends (GPT-5, Claude Sonnet 4.5, Gemini 2.5 Pro)—a 4–67× speedup over the backends alone—while lifting NaturalQuestions accuracy from the base model's 10%

Load-bearing premise

The method's reliability depends on the GPT-4o-generated synthetic conversations being a faithful stand-in for real goal-directed dialogue; if the way knowledge chunks and silences are arranged in training does not match how real backends stream, the learned infill behavior will not generalize.

Editorial extensions

If this is right

Voice agents can be assembled from a fast local talker and an arbitrary backend reasoner; swapping backends (GPT-5, Claude, Gemini) changed accuracy but not the ~170ms time-to-first-response, so responsiveness becomes independent of the reasoner chosen.
The system particularly benefits thinking and agentic models, whose long reasoning latencies (up to 10.9s for Gemini in the paper) are hidden behind immediate conversational speech.
A 360M talker cannot absorb all streamed knowledge: ConvFill's accuracy trails the backend by 17–31 points and per-turn entailment is only 28–36%, so the practical ceiling for this architecture is set by the talker's capacity and training signal, not the backend.
The task formulation gives a concrete recipe—synthetic multi-domain dialogue, entailment-filtered chunks, and a 1-second silence cadence—that can be reused to train other small models for conversational infill.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the paper evaluates on text tokens before TTS, the full speech pipeline will add synthesis latency; whether the promised responsiveness survives end-to-end is an open question the paper does not address.
An ablation that replaces the knowledge stream with random or empty chunks would separate the effect of the streamed knowledge from the effect of fine-tuning on the synthetic corpus; without it, part of the 36–42 point accuracy gain could be attributed to the fine-tuning itself.
If the synthetic-to-real transfer holds, the approach could be adapted to other real-time interaction domains—agentic assistants, tutoring, customer service—by regenerating the corpus with a domain-specific persona list; the paper does not test this transfer.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Here is the short version: the conversational-infill idea is genuinely new, the measured TTFT numbers are believable, and the accuracy gain over the base model is real. But the arXiv abstract overstates the body in three load-bearing ways, and the central 'Pareto' claim is supported only if you define latency as time-to-first-token rather than time-to-information.

What is actually new: the task formulation. A small model generates conversational phrases for a single turn, conditioned on a stream of knowledge chunks from a backend model and on a silence token that is emitted every second when no chunk has arrived. The small model sees only the current turn; the backend sees the full history. That specific interleaved setup, with learned silence-token handling, is not present in Talker-Reasoner, phased responses, or Mini-Omni-Reasoner. The implementation looks competent: the synthetic corpus is filtered with an MNLI entailment check, and the training/inference format in Figure 2 is coherent.

Credit where earned: Table 1 shows ConvFill at 0.16-0.17s TTFT across three backends, versus 0.74-10.9s for the backends alone. That is a real, consistent latency improvement. Table 2 shows accuracy moving from 10% for SmolLM2-360M to 46-52% with ConvFill, and the paper explicitly acknowledges the 69-80% backend ceiling and the 28-35% entailment rates in Section 6. Those numbers are reported honestly.

The problems: the metadata abstract says 'within 6.3%' and 'seven widely used small language models' and cites a user study (n=18). None of that is in the body. The body trains one 360M model, the gaps are 20-32 points, there is no user study, and dataset size is 5,997 conversations / 46,151 turns, not 290,571 examples. These are not cosmetic; they are the paper's central claims. Also, code/data are promised but not available, so nothing can be independently checked.

The stress-test concern holds up. The sub-200ms TTFT is time to the first filler phrase, not to answer content. The QA accuracy in Table 2 is computed on responses that incorporate streamed backend chunks, so the final answer cannot be complete before the backend produces those chunks. For Gemini-2.5-Pro, backend mean TTFT is 10.9s; even with chunking, the user hears substantive content only after the backend starts generating. The paper never reports end-to-end time to a complete or correct answer. So 'decoupling response latency from capability' is not demonstrated for information delivery; it only works if the user is satisfied with the filler. On a time-to-answer metric, ConvFill sits on roughly the same latency curve as the backend and is 20-32 points less accurate.

Who this is for: people building real-time voice agents, and reviewers at systems/HCI venues who want a solid idea that needs a major rewrite. My recommendation: send it to peer review rather than desk-reject; it deserves referee time. But the current version should come back with the abstract reconciled to the body, the user study either added or removed, an honest time-to-answer/latency-to-content analysis, and the artifacts released.

Referee Report

4 major / 6 minor

Summary. The paper proposes a new task, conversational infill, in which a small on-device language model immediately generates conversational phrases while a large cloud-based backend model streams knowledge chunks that the small model incorporates into its ongoing response. The authors generate a synthetic multi-domain training corpus with GPT-4o, filter it with an MNLI entailment model, and fine-tune SmolLM2-360M (ConvFill) on the resulting turns. They evaluate ConvFill on 250 NaturalQuestions samples with three backend models (GPT-5, Claude Sonnet 4.5, Gemini-2.5-Pro), reporting time-to-first-token (TTFT), question-answering accuracy, and turn-level entailment. The paper claims that ConvFill achieves sub-200ms TTFT, improves accuracy from 10% to 46–52%, and that conversational infill 'decouples response latency from model capability.' The abstract additionally claims a gap of only 6.3% to frontier models, a 290,571-example dataset, seven small models, and a live user study with n=18, but these claims do not appear in the body.

Significance. If the central claim were established, conversational infill would be a practically valuable technique: it would let a voice agent provide immediate, contextually appropriate speech while a larger model computes, improving the latency-capability tradeoff for conversational AI. The synthetic dataset generation pipeline and the framing of streaming knowledge incorporation are potentially reusable. However, the paper's headline claims are not supported by the reported experiments. The accuracy gap to backend models is 20–32 percentage points, not 6.3%. The latency metric is time-to-first-token, which measures the delay until a filler phrase, not until the answer is delivered; the substantive answer still awaits the backend's streamed chunks. The entailment rates are 28–35%, and the authors' reinterpretation of most neutral classifications as acceptable conversational variation is not backed by a quantitative human evaluation. The abstract promises a seven-model comparison and a user study that are absent from the body. These are load-bearing issues, not presentation concerns.

major comments (4)

[Abstract; Table 2; Sections 4.1, 4.2, 5.2] The abstract states that ConvFill closes the accuracy gap 'to within 6.3%' of frontier models, that the dataset has 290,571 examples, that the task is learned across seven small models (135M–1.7B), and that a user study (n=18) ranks ConvFill on par with frontier models. The body reports a 5997-conversation / 46,151-turn dataset, only SmolLM2-360M as the infill model, no user study, and Table 2 accuracy gaps of 20 points (Gemini), 28 points (Claude), and 32 points (GPT-5). These contradictions are not cosmetic; the abstract's central quantitative claims are unsupported by the manuscript's experiments.
[Section 4.3; Section 5.2, Table 1] The 'decoupling of response latency from model capability' rests on TTFT. But Table 1's sub-200ms TTFT is the time to generate a conversational phrase, not the time to any answer content. The inference pipeline (Section 4.3) starts ConvFill and the backend in parallel, and ConvFill receives a silence token every d=1s until a knowledge chunk arrives. For Gemini-2.5-Pro, the backend mean TTFT is 10.9s; no complete answer can be delivered before the backend's chunks are streamed. The paper reports no end-to-end time-to-complete-answer or time-to-correct-information. On a time-to-answer metric, ConvFill would likely fall on the same latency-capability curve as the backend, with an additional accuracy loss (Table 2). The claimed Pareto improvement is therefore an artifact of defining latency as time-to-first-token.
[Sections 5.2, 6, Table 3] The grounding claim is only weakly supported. Entailment rates in Table 3 are 28.1–35.5%, with 59.2–64.2% neutral. Section 6 dismisses most neutral classifications as acceptable conversational variation based on 'manual inspection,' but no protocol, inter-annotator agreement, or quantitative human evaluation is reported. Without such validation, the entailment numbers cannot support the claim that ConvFill preserves backend knowledge faithfully. Additionally, the NQ accuracy in Table 2 reports point estimates on 250 samples with no confidence intervals, so the 36–42 percentage-point improvement over the base model is not statistically characterized.
[Section 4.1; Section 3] The training premise is that GPT-4o-generated synthetic dialogues, with responder_thoughts as ground-truth knowledge/silence streams and an MNLI entailment filter, provide a faithful and sufficient signal for learning conversational infill. The paper does not validate this premise against real dialogue, nor does it test whether the learned mapping generalizes beyond the six synthetic domains. The 1-second silence cadence (d=1) is a hand-set engineering parameter. If the synthetic structure does not reflect real backend streaming behavior and user turns, the reported NQ accuracy and low-latency behavior may be artifacts of the training distribution rather than a general capability. At minimum, an evaluation on real conversational data or a human quality assessment of the synthetic corpus is needed.

minor comments (6)

[Section 5.2] 'accuracy improvements of 36-42%' should be '36–42 percentage points' (e.g., 10% to 46% is a 36-point increase, not a 360% relative improvement). The same wording appears in the conclusion.
[Section 4.1] 'responder_thoughts concise concise thoughts' contains a duplicated word; please fix.
[Section 5.2] 'ConvFill repsonds only relying on its context' is a typo for 'responds.'
[Section 4.2] The text says 'Further specifications can be found in our code, which will be released at a later time,' but the abstract and GitHub link promise released code. Please clarify the availability status.
[Table 3] The sentence 'Percentages of neutral ratings for those models are entailment rates only remain at 64.2%, 59.2%, and 59.9%, respectively' is ungrammatical and should be rewritten.
[Section 3] The notation is dense and occasionally undefined (e.g., 'i∈t' and 'li−d'). Please provide a notation table or a more precise formulation of the streaming protocol.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ConvFill's training, evaluation, and latency measurements are not reduced to their own inputs.

full rationale

The derivation chain is self-contained against independent benchmarks. ConvFill is trained on a synthetic corpus generated by GPT-4o (Section 4.1) and evaluated on NaturalQuestions (Section 5.1), a public QA benchmark, with three different backend model families. The accuracy gains (Table 2) are not fitted to the evaluation set: no NQ data is used in training, and no parameter is optimized against the reported metrics. The entailment filter used to curate the training data (DeBERTaV3-MNLI) is also used as an evaluation tool, but it is an external public checkpoint, and the reported entailment rates are low (28.1-35.5%), so the evaluation result is not forced by the filtering construction. The sub-200 ms TTFT is a structural consequence of starting a small local model immediately while the backend streams knowledge; this is a design property of the measured quantity rather than a circular derivation. The paper's own Section 3 definition requires len(S)+len(L)=len(C), which merely formalizes the task and does not encode the NQ accuracy result. No self-citations are load-bearing; no uniqueness theorem or imported ansatz is invoked. There are correctness concerns (e.g., the abstract's 'within 6.3%' claim does not match Table 2, and TTFT does not measure time-to-substantive-answer), but these are not circularity.

Assumptions & free parameters 1 free parameters · 4 assumptions · 1 invented entities

All central-claim dependencies are listed: one hand-set latency parameter, four domain/ad-hoc assumptions about the synthetic data and entailment evaluation, and one invented special token. The paper introduces no physical or theoretical entities.

free parameters (1)

silence interval d = 1 second
Section 4.3 chooses d=1s for when to feed a <|sil|> token if no backend chunk has arrived; no ablation shows how TTFT or accuracy changes with d.

assumptions (4)

domain assumption Textual entailment (Williams et al. 2017) is the correct relation between a streamed knowledge chunk and the infill model's conversational phrase (ei -> ci for ei in L).
Defines the task in Section 3; assumes a knowledge chunk must strictly entail the phrase, ignoring conversational paraphrase, hedging, and elaboration that are normal in dialogue.
ad hoc to paper GPT-4o-generated synthetic conversations with responder_thoughts as ground-truth knowledge/silence streams are a faithful proxy for real goal-directed dialogue.
Section 4.1: the entire training set is synthetic; no human validation or real conversational data is used, so transfer to real users is assumed.
ad hoc to paper DeBERTaV3-MNLI's neutral label can be manually reinterpreted as acceptable conversational variation rather than grounding failure.
Section 6: paper discounts 59-64% neutral entailment ratings based on informal manual inspection, with no quantitative support.
domain assumption The silence token cadence d=1s and the two-thread streaming queue architecture faithfully capture realistic backend latency distributions.
Section 4.3; API/server latency in real deployments varies widely; a single fixed cadence may not represent worst-case reasoning models.

invented entities (1)

<|sil|> silence token
purpose: Represents periods in the stream when the backend has not produced a knowledge chunk; the talker must generate a conversational filler without external knowledge.
Introduced in Sections 3-4 as a new special token added to the tokenizer/embeddings. It is a system design artifact rather than a physical entity; no external prediction is tied to it and no ablation is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents." pith.science (2026). https://pith.science/paper/4A7FXM56

@misc{pith2026251107397,
  author       = {Pith},
  title        = {Pith review of: Thinking While Speaking: Inference-Time Knowledge Transfer for Responsive and Intelligent Conversational Voice Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4A7FXM56}},
  note         = {Machine review of arXiv:2511.07397}
}

read the original abstract

Voice agents face a fundamental tension: the reasoning, retrieval, and tool use that make foundation models capable are iterative and slow, while conversational interaction demands responses on a millisecond timescale. Smaller, real-time models meet the latency bar but cannot match foundation models on complex tasks, leaving current voice agents to trade away either responsiveness or capability. We introduce conversational infill, where a small talker model both immediately generates contextually grounded responses to hide the latency of an external reasoner model and fluently integrates streamed reasoner knowledge into its responses during inference. We curate a 290,571-example synthetic dataset spanning six domains and demonstrate that this task is learnable across seven widely used small language models ranging from 135M to 1.7B parameters. Our system implementation, ConvFill, sustains millisecond-level time-to-first-response while closing the accuracy gap to within 6.3% of the corresponding frontier reasoner performance. In a live user study (n=18) with talker deployments running on an Apple M2 SoC, participants rank ConvFill on par with frontier models overall, prefer it for retrieval-heavy tasks, and rate it significantly more responsive. These results show that conversational infill unlocks a new point on the latency-capability Pareto frontier, offering a practical path toward voice agents that are both responsive and highly capable. Code, models, and datasets are available at https://github.com/vysri/conversational-infill.

Figures

Figures reproduced from arXiv: 2511.07397 by the authors.

**Figure 1.** Conversational Infill. ConvFill operates on the turn level while the backend model operates on the conversation level. The backend model outputs silence or a knowledge chunk (blue) and ConvFill incorporates these chunks along with conversational infill to generate the response to the user (orange). Existing approaches to real-time voice assistants fall broadly into one of two categories. Cascaded systems combine sep… view at source ↗

**Figure 2.** Infill generation format. During training, ConvFill sees the user utterance, previous streamed knowledge chunks, and previous conversational phrases in an interleaved manner. ConvFill is trained to predict the last red utterance—its own last conversational phrase conditioned on the last external knowledge chunk (blue) and the phrase history. able (noa) version of DeBERTaV3 (He et al., 2023) finetuned on the MNLI dat… view at source ↗

**Figure 3.** Conversational infill inference example. The user asks about moving to a new city. ConvFill generates a response in an interleaved and streaming manner, in each conversational phrase (purple), referencing an external knowledge chunk (blue). sational interfaces. Turn-level entailment measures if the ConvFill model is able to preserve and use information from a large knowledge model in a manner that is consistent wit… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

5 extracted references · 3 linked inside Pith

[4]

InProceedings of the 40th Inter- national Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 19274–19286

Fast inference from transformers via spec- ulative decoding. InProceedings of the 40th Inter- national Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 19274–19286. PMLR. Zijian Lew, Joseph B Walther, Augustine Pang, and Wonsun Shin. 2018. Interactivity in online chat: Conversational contingency and response lat...

arXiv 2018
[5]

Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien-Ming Huang

Online speculative decoding.arXiv preprint arXiv:2310.07177. Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien-Ming Huang. 2025. User in- teraction patterns and breakdowns in conversing with llm-powered voice assistants.International Journal of Human-Computer Studies, 195:103406. Mykola Maslych, Mohammadreza Katebi, Christo- pher Lee, Yah...

arXiv 2025
[2023]

ArXiv:2111.09543

DeBERTaV3: Improving DeBERTa us- ing ELECTRA-Style Pre-Training with Gradient- Disentangled Embedding Sharing.arXiv preprint. ArXiv:2111.09543. Brendan Iribe, Ankit Kumar, and The Sesame Team

arXiv
[2024]

Blog post

Crossing the uncanny valley of voice. Blog post. Yuin Jeong, Juho Lee, and Younah Kang. 2019. Explor- ing effects of conversational fillers on user perception of conversational agents. InExtended Abstracts of the 2019 CHI Conference on Human Factors in Com- puting Systems, CHI EA ’19, page 1–6, New York, NY , USA. Association for Computing Machinery. Tom ...

2019
[2025]

Original-date: 2023-06-26T19:39:32Z

Ollama. Original-date: 2023-06-26T19:39:32Z. Yash Akhauri, Anthony Fei, Chi-Chih Chang, Ahmed F AbouElhamayed, Yueying Li, and Mohamed S Ab- delfattah. 2025. Splitreason: Learning to offload reasoning.arXiv preprint arXiv:2504.16379. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Le- andro von Werra, and Thomas Wolf. 2024. Smollm - blazingly fast and rema...

arXiv 2023

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [4]

InProceedings of the 40th Inter- national Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 19274–19286

Fast inference from transformers via spec- ulative decoding. InProceedings of the 40th Inter- national Conference on Machine Learning, volume 202 ofProceedings of Machine Learning Research, pages 19274–19286. PMLR. Zijian Lew, Joseph B Walther, Augustine Pang, and Wonsun Shin. 2018. Interactivity in online chat: Conversational contingency and response lat...

arXiv 2018

[2] [5]

Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien-Ming Huang

Online speculative decoding.arXiv preprint arXiv:2310.07177. Amama Mahmood, Junxiang Wang, Bingsheng Yao, Dakuo Wang, and Chien-Ming Huang. 2025. User in- teraction patterns and breakdowns in conversing with llm-powered voice assistants.International Journal of Human-Computer Studies, 195:103406. Mykola Maslych, Mohammadreza Katebi, Christo- pher Lee, Yah...

arXiv 2025

[3] [2023]

ArXiv:2111.09543

DeBERTaV3: Improving DeBERTa us- ing ELECTRA-Style Pre-Training with Gradient- Disentangled Embedding Sharing.arXiv preprint. ArXiv:2111.09543. Brendan Iribe, Ankit Kumar, and The Sesame Team

arXiv

[4] [2024]

Blog post

Crossing the uncanny valley of voice. Blog post. Yuin Jeong, Juho Lee, and Younah Kang. 2019. Explor- ing effects of conversational fillers on user perception of conversational agents. InExtended Abstracts of the 2019 CHI Conference on Human Factors in Com- puting Systems, CHI EA ’19, page 1–6, New York, NY , USA. Association for Computing Machinery. Tom ...

2019

[5] [2025]

Original-date: 2023-06-26T19:39:32Z

Ollama. Original-date: 2023-06-26T19:39:32Z. Yash Akhauri, Anthony Fei, Chi-Chih Chang, Ahmed F AbouElhamayed, Yueying Li, and Mohamed S Ab- delfattah. 2025. Splitreason: Learning to offload reasoning.arXiv preprint arXiv:2504.16379. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Le- andro von Werra, and Thomas Wolf. 2024. Smollm - blazingly fast and rema...

arXiv 2023