Computer Vision and Pattern Recognition · arXiv:2609.13814 v3 · submitted 12 Sep 2026
Realtime-Venus: A full-duplex interaction system with asynchronous delegation
The paper says
“Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context.”
more of the abstract
We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 15 Sep 2026 · of version 1 of the paper; version 3 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. Solid engineering, plausible results, but the central empirical claims rest on unreleased code and data, and the synthetic pipeline is an unvalidated black box. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Section 6 and Section 7.3-7.6
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The entire training corpus (2.8M samples) is synthetically generated via a pipeline that includes scenario planning and speech realization (Section 6). The paper reports no contamination check between this synthetic data and the test sets of the evaluated benchmarks (StreamingBench, OVO-Bench, OmniPro, Daily-Omni, OmniVideoBench, LVOmniBench, and the audio benchmarks).
more of objection 1
If the scenario planning stage was seeded from publicly available benchmark examples or similar natural conversation patterns, the models could learn benchmark-specific patterns rather than general capabilities, directly invalidating the comparative leadership claims (Tables 2, 4, 5). The risk is amplified by the scale of the corpus and the lack of data release. The authors should either release the synthetic data or a representative subset for external scrutiny, or at minimum provide a systematic contamination analysis showing that no benchmark test cases appear in the training data (e.g., via n-gram overlap or embedding similarity) and that model performance does not degrade on a held-out non-benchmark evaluation set.
The paper says, at Section 7.6
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The delegate benchmark is internally constructed and not publicly released. Its task categories are described at a high level, but there is no information about the number of examples, how reference labels were produced, or how representative the scenarios are of real-world delegation needs.
more of objection 2
Because the system is trained on synthetic data that presumably includes analogous delegation scenarios, the benchmark may exhibit circularity. The authors should release the delegate benchmark or at least provide a detailed description of its construction, size, difficulty distribution, and inter-annotator agreement, and ideally evaluate on an external, independently constructed delegation test set.
The paper says, at Section 7 overall
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
No confidence intervals, error bars, or statistical significance tests are reported for any of the benchmark results. For models whose outputs are stochastic, point estimates alone are insufficient to support the claim that one model 'leads' another, especially when margins are small (e.g., 70.2 vs. 67.9 on StreamingBench, or 78.0 vs. 77.5 on MMAU).
more of objection 3
The authors should report multiple runs with standard deviations or at minimum conduct a significance test (e.g., bootstrap or McNemar) for the key comparisons where a leadership claim is made.
6 minor objections
The paper says, at Section 4.3
Pith says: minor objection 1 of 6
Equation (2) uses a duration estimate D(Y_{k,1:n}) that is not defined. Please clarify how the duration is computed (e.g., by a separate duration model, by average phoneme rate, or from training data statistics).
The paper says, at Section 3.2
Pith says: minor objection 2 of 6
Equation (1) uses a subscript k for chunks but the notation includes eta_k which is said to be runtime feedback but never formally defined or instantiated in the evaluation setup. Please clarify what concrete feedback signals are used.
show 4 more
The paper says, at Section 5.1
Pith says: minor objection 3 of 6
The look-back horizon Delta is listed as a free parameter in the axiom ledger but its value is never stated anywhere in the paper. Please specify the value used in the experiments.
The paper says, at Table 2
Pith says: minor objection 4 of 6
The table caption states that all scores are on a 0-100 scale but ProactiveVideoQA reports PAUC and OmniPro reports probe-mode accuracy. Please clarify the range of PAUC and whether probe-mode accuracy is on the same scale.
The paper says, at Reference list
Pith says: minor objection 5 of 6
Several entries are missing URL access dates or version identifiers. For reproducibility, please ensure that all arxiv links include the version (e.g., v1, v2).
The paper says, at General
Pith says: minor objection 6 of 6
Figures 1 and 2 are reproduced at very low resolution in the PDF. Please ensure that all figures are clearly legible and that each is accompanied by a self-contained caption that explains what is being plotted.
What would settle it
Compute n-gram overlap (e.g., 5-gram Jaccard similarity) between the synthetic training data and the test splits of the evaluated benchmarks (StreamingBench, OVO-Bench, OmniPro, Daily-Omni, OmniVideoBench, LVOmniBench, MMAU, MMAU-Pro, Llama Questions). Report the maximum per-benchmark overlap percentage. Also perform embedding-level nearest-neighbor search (e.g., Sentence-BERT) to detect paraphrased leakage. If any benchmark shows >5% exact multi-gram overlap or >10% semantic nearest-neighbor similarity with training data, the benchmark results cannot be considered valid for that task.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 15 Sep 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This is a well-engineered full-duplex system. The dual-loop runtime that separates live interaction from asynchronous delegation is the real contribution, and the unified streaming serialization with private control tokens is cleanly designed. The benchmark results — especially the streaming video scores and the full-duplex continuation rates — are competitive and suggest the system works. I'd believe it.
That said, there are significant soft spots. First, no model weights, no training code, no data release. That alone blocks independent verification. Second, the delegate benchmark is internal and not public, which means the delegation routing claims cannot be reproduced. Third, the synthetic training data pipeline (2.8M samples) is described in detail, but the paper reports no contamination check against the evaluation benchmarks. The stress-test concern about benchmark contamination is plausible in principle, though nothing in the paper suggests it happened. Still, without a check it's a load-bearing uncertainty. Fourth, no confidence intervals or significance tests on any metric — for a paper that claims leadership on 6 of 8 video benchmarks, that's a notable omission. Fifth, the novelty claim about being the first full-duplex omni model with asynchronous backend invocation is contested by DuplexOmni and JoyAI-VL-Interaction; the paper acknowledges them but the phrasing is a bit strong.
None of this makes the work fraudulent. The architecture is thoughtful and the engineering effort is substantial. The soft spots are about reproducibility and rigor, not correctness.
This paper is for researchers building real-time spoken/multimodal systems. It deserves a serious referee who will push for code/data release and contamination checks. I would accept peer review.
The strongest objection
The central claim—that Realtime-Venus-Omni leads online models on six of eight video benchmarks—depends on the integrity of those benchmarks. Section 6 describes a pipeline that generates 2.8 million synthetic training samples via scenario planning and speech realization. The paper reports no contamination check between this synthetic data and the test sets of the evaluated benchmarks (StreamingBench, OVO-Bench, OmniPro, Daily-Omni, OmniVideoBench, LVOmniBench, and the audio benchmarks). If the scenario planning stage was seeded from publicly available benchmark examples or similar natural conversation patterns, the models could learn benchmark-specific patterns rather than general capabilities. This would directly invalidate the comparative results. The risk is amplified by the scale of the synthetic corpus (≈70% video, 30% audio) and the lack of data release, which prevents external scrutiny. The reader flagged synthetic data realism as a weakness; contamination is the specific mechanism by which that weakness could undermine the benchmark claims.
The reader's weakest_assumption identified the synthetic data pipeline as insufficiently validated but focused on realism rather than contamination. My concern is a specific causal path by with that weakness could invalidate the central claim. The reader's verdict of CONDITIONAL already covers reproducibility; adding contamination would not change the verdict ow, but it would justify a stronger condition for upgrading. The paper does not perform or mention contamination analysis, which is standard practice in large-scale generative training. This is a concrete, testable threat that should be addressed before unconditional acceptance.
The referee's report
If the reported results are reliable, the system makes a meaningful engineering contribution by demonstrating how asynchronous delegation can be integrated into a full-duplex conversational frontend without degrading interaction quality. The unified streaming formulation and the dual-loop runtime are clean design choices. However, the significance is substantially tempered by two concerns that affect the central benchmark claims: (i) the training data is entirely synthetic and no contamination analysis against the evaluation benchmarks is provided, creating a risk that the reported leadership is inflated by benchmark-specific patterns; (ii) the delegate benchmark (Section 7.6) is internally constructed and not publicly available, limiting independent verification. The paper is otherwise well-structured and covers a wide range of evaluation scenarios.
Pith's reading of the paper (a machine's summary, not the authors')
Realtime-Venus is a system that combines a native full-duplex conversational model with an asynchronous background execution framework, allowing the assistant to perceive, speak, and delegate tasks to external tools without pausing the conversation. The authors claim that a shared causal timeline that aligns user inputs, model outputs, and background events enables the model to decide when to listen, speak, or delegate while maintaining continuous interaction. If true, this design would allow real-time voice assistants to seamlessly incorporate tool use, web search, and complex reasoning into ongoing conversations, overcoming a key limitation of turn-based systems.
What carries the argument. Unified stream serialization: each second-long chunk aligns the user stream (audio features, plus video for the omni model), the assistant stream (control tokens <|listen|>/<|speak|>, response text, speech tokens, and optional <delegate> spans), and the background stream (asynchronous <backend> results). The control tokens decide whether to listen, speak, or yield the floor; the delegate span triggers an external task whose result later re-enters the background stream. A dual-loop runtime separates the latency-sensitive interaction loop from the capability loop that executes registered tools and returns prepared replies.
Core claim. The paper claims that a unified stream serialization—where user audio/video, assistant text/speech, and private delegation requests and results are interleaved as three synchronized streams on a one-second-per-chunk timeline—enables a single 9B model to jointly learn interaction control, response generation, and asynchronous delegation. This design lets the frontend continue listening and speaking while the harness executes external tasks in the background, and results are reintroduced at the next available input boundary for the frontend to speak them at the right conversational moment.
The premise it rests on. The synthetic data pipeline (Section 6) that generates over 2.8 million samples of full-duplex trajectories and delegation workflows is diverse and realistic enough to teach the model to generalize to natural, unscripted conversations with genuine disfluencies, ambiguities, and complex multi-step user intents.
If the paper is right (Pith's inferences, not the authors')
Real-time voice assistants can incorporate external tool calls and retrieval without interrupting the conversational flow, enabling natural mixed-initiative dialogue.
A single 9B model trained with offline understanding, duplex trajectories, and delegation data can handle both perception and asynchronous execution, suggesting that smaller models can rival larger turn-based systems on streaming benchmarks.
The shared causal timeline simplifies the integration of delayed results into an ongoing conversation: the frontend controls when to speak a prepared reply, while the harness determines its content, preserving both timing and coherence.
Memory-augmented streaming with a pretrained-free retrieval module extends hour-scale video understanding, allowing the system to answer questions about events that occurred minutes earlier within a long session.
The data pipeline for generating synthetic full-duplex and delegation trajectories (scenario planning, speech realization, temporal alignment) provides a recipe for training other interaction systems without requiring expensive human annotation of overlapping speech and interruptions.
The paper relies entirely on synthetic training data for duplex and delegation behaviors. The claim of generalizability would be much stronger if the model were evaluated on real-world recorded conversations with natural disfluencies, overlapping speech, and task complexity not present in the synthetic pipeline.
The trade-off between high continuation rates (0.97 for backchannels) and lower interruption response rates (0.75 vs. Joy-Duplex's 0.88) suggests that the model may be conservative about stopping its own speech. This could frustrate users who expect the assistant to yield quickly when interrupted.
The Delegation Benchmark results show a clear modality asymmetry: the audio-only model delegates routine requests aggressively (specificity 39%), while the omni model misses many requests that truly need delegation (recall 68%). A combined routing strategy might outperform either single model.
The fact that the paper compares only online models in its main leaderboards but includes offline models only for reference indicates that the claimed state-of-the-art is within a specific deployment protocol (real-time streaming), not across all possible inference settings.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Natural interaction in digital and physical environments requires continuous perception and timely responses. Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. A dual-loop runtime coordinates live interaction with background reasoning and tool execution. Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.38
“We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.06 · human participants 0.15 · controlled comparison 0.65 · randomized assignment 0.01 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Natural interaction in digital and physical environments requires continuous perception and timely responses. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| Spoken dialogue relies on acoustic and linguistic cues, while video interaction also requires grounding the conversation in evolving visual context. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| We present Realtime-Venus, a proactive full-duplex interaction system with two separately trained 9B models: Realtime-Venus-Omni for audio-visual interaction and Realtime-Venus-Audio for spoken interaction. | contribution 0.95 | 0.92 | 0.09 | asserted 1.00 |
| Each model serves as a complete conversational frontend, integrating continuous perception, conversational control, and native speech generation through a shared causal timeline for user inputs, model outputs, and delegation events. | method 0.88 | 0.84 | 0.07 | asserted 1.00 |
| A dual-loop runtime coordinates live interaction with background reasoning and tool execution. | method 0.96 | 0.77 | 0.04 | asserted 1.00 |
| Foreground interaction continues while Realtime-Venus-Harness executes tasks asynchronously and returns results for integration into the ongoing dialogue. | method 0.98 | 0.74 | 0.05 | asserted 1.00 |
| Both models follow a common post-training recipe combining offline understanding, proactive full-duplex trajectories, and delegation workflows. | method 0.98 | 0.83 | 0.05 | asserted 1.00 |
| Among the evaluated online models, Realtime-Venus-Omni achieves the highest scores on six of eight video benchmarks, including StreamingBench (70.2%), OVO-Bench (64.7%), and Daily-Omni (81.3%). | result 1.00 | 0.95 | 0.16 | asserted 1.00 |
| Across eight audio understanding and spoken question answering benchmarks, Realtime-Venus-Audio leads the compared models on MMAU (78.0%), MMAU-Pro (63.2%), Llama Questions (83.8%), and Speech CMMLU (67.8%), while matching the best VoiceBench AlpacaEval score of 4.81. | result 1.00 | 0.94 | 0.12 | asserted 1.00 |
| On Full-Duplex-Bench v1.5, Realtime-Venus-Audio responds to 75% of user interruptions and achieves continuation rates of 97%, 88%, and 86% under backchannels, other-directed speech, and background speech, respectively, exceeding Gemini 3.1 Live and GPT-4o on all three continuation metrics. | result 1.00 | 0.95 | 0.15 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,890 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Cited by 6 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Jarvis: Coordination Deserves a Brain of Its Own in Agentic Interactive Models cs.SD · Oct 2026
- DeltaStreamer: Towards Human-Like Agentic Streaming Video Understanding cs.CV · Oct 2026
- FLUID: Flexible, Evidence-Grounded Situated Task Guidance cs.AI · Nov 2026
- DuplexAgent-RSI: Recursive Harness Improvement for Full-Duplex Voice Agent Collaboration cs.AI · Oct 2026
- APEX-Voice: Can Voice Agents Complete Professional Workflows Through Full-Duplex Interaction cs.AI · Sep 2026
- Asynchronous Robot Policies cs.RO · Nov 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment