{"id":"40f7c798-d59b-4260-a9d3-5357aa9dec15","arxiv_id":"2607.23023","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"OmniMate generates real-time streaming video, speech, and sound effects for interactive avatars while preserving the same face and voice across long multi-turn conversations.","lead":"OmniMate is a system that generates a talking avatar's video, speech, and sound effects live, switching smoothly between listening and acting during a conversation. It aims to keep the same face and voice through long, open-ended chats with virtual characters.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPC's load-bearing component is the LLM duration estimate; its accuracy is never measured, so the claimed seamless execution/listening transitions are unsupported.","rationale":"The reader's weakest_assumption is the same: the GPC progress schedule depends on the LLM-estimated execution duration, which is neither measured nor controlled. The manuscript itself flags the failure mode in Limitations, so this is not manufactured; it is the admitted weak point of the main claimed contribution. The 30/60/240s robustness rows are not a substitute for testing duration-estimation error. I do not see a stronger internal inconsistency; the architecture and experiments are coherent, and the reported FPS/TTFF, if correct, support the real-time synthesis claim. The correct disposition remains CONDITIONAL: the paper should not be rejected, but the central interactive claim is conditional on demonstrating that LLM duration estimates are accurate enough (or that GPC degrades gracefully under error). No other concern changes this verdict.","tokens_in":11692,"tokens_out":5633,"duration_ms":60975,"concrete_test":"Build a controlled benchmark of 100–200 short interaction instructions. For each instruction, record the LLM-estimated duration; obtain an oracle duration from forced alignment or human transcription of the eventual response; run OmniMate twice, once with the estimated duration and once with the oracle duration. Measure word omission/repetition and state-transition timing (e.g., fraction of responses ending within ±10% of schedule, number of truncated/repeated words). If the estimated-duration condition degrades these metrics by more than a pre-registered margin relative to oracle, the GPC's load-bearing assumption fails and the central interactivity claim is unsupported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"OmniMate's central novelty is GPC's duration-aware state control. Eq. 1–2 compute progress 1..1000 from T_v/T_a, which are derived from the LLM-estimated duration; tokens past this point get progress 0 and, via Eq. 5, listening-prompt conditioning. Training progress values, however, come from ground-truth token counts of teacher-generated sequences, so at inference the controller is fed an unvalidated prediction. The paper reports no metric for duration-estimation error, no ablation sweeping estimate error, and no end-to-end interaction latency; the only relevant statement is the limitation that 'inaccurate duration estimation may occasionally cause word omissions or repetitions.' The 30s/60s/240s rows in Table 2 vary total generation length, not the mismatch between predicted and actual duration, so they do not exercise this failure mode. Because seamless execution→listening transitions and complete responses are the claimed advance, this is the least secure load-bearing condition. It is not an internal inconsistency, but it is an unsupported dependency on an external component.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"OmniMate proposes a streaming framework built on LTX2.3 that jointly generates video, speech, and sound effects for interactive avatars. It introduces a Generation Progress Controller (GPC), which injects progress embeddings computed from an LLM-predicted execution duration to control execution/listening transitions, and a Multi-Reference Conditioning Module (MRCM), which concatenates multiple reference image latents and optionally replaces audio latents with reference speech to preserve visual and speaker identity. Training uses a DMD-distilled teacher to construct state-transition trajectories, followed by ODE initialization and self-forcing DMD distillation. The paper reports quantitative comparisons on an interaction-oriented adaptation of VerseBench, showing the highest FPS, the lowest TTFF, and strong audio-visual consistency over 30s–240s segments.","tokens_in":12007,"tokens_out":5677,"duration_ms":60319,"significance":"If the system operates as claimed, the two architectural ideas—explicit progress conditioning and multi-reference identity injection—are useful steps toward open-ended interactive audio-visual avatars. The paper provides a detailed training pipeline and attempts to compare with recent offline and streaming models. However, the evidence in the current manuscript is not commensurate with the strength of the central claims. The real-time claim covers only the generation model, the GPC's dependence on LLM duration estimates is not validated, and the benchmark is self-constructed without release or error bars. The core ideas are plausible and the writing is clear, but the manuscript needs substantial empirical additions before the claims are supported.","major_comments":[{"comment":"The real-time claim is not demonstrated end-to-end. FPS and TTFF in Table 1 measure only the streaming generation model on an H100 GPU; the method overview states that user speech is transcribed by an ASR system and processed by an LLM before generation. No end-to-end interaction latency or system latency breakdown is reported. The abstract's claim of 'real-time interactive avatar generation' and 'low-latency state transitions' therefore applies only to the generator, not the full interactive loop. Please report the complete ASR→LLM→generation pipeline latency, or explicitly state that the real-time claim is generator-only.","section":"Method Overview / Table 1"},{"comment":"The load-bearing GPC mechanism depends on an unvalidated LLM-estimated execution duration. At inference, T_v and T_a in Eqs. (1)–(2) are derived from this predicted duration, while training progress values are computed from ground-truth teacher-generated token counts. The paper never measures the accuracy of the LLM duration estimate, nor its effect on response completeness and execution/listening transitions. The 30s/60s/240s rows in Table 2 vary total generation length, not the mismatch between predicted and actual duration. The Limitations section itself concedes that 'inaccurate duration estimation may occasionally cause word omissions or repetitions.' Because seamless transitions and complete responses are a central claimed contribution, please add a quantitative evaluation of duration-prediction error and an ablation that deliberately perturbs the predicted duration to show gracefu","section":"Equations (1)–(2) / Generation Progress Controller"},{"comment":"The evaluation evidence is insufficient to support the strong quantitative conclusions. Tables 1 and 2 report no error bars, confidence intervals, significance tests, or number of trials, and many metric differences are small (e.g., AQ 6.22 vs 5.92 in Table 1; V-ID 0.957 vs 0.952). The benchmark is an 'interaction-oriented adaptation of VerseBench' constructed by the authors, but no benchmark samples, annotation details, or evaluation code are provided. This makes it difficult to assess whether the reported advantages are stable or an artifact of a self-constructed test set. Please provide variance estimates, exact metric definitions, and make the benchmark and evaluation code publicly available.","section":"Tables 1 and 2 / Evaluation Data Construction"},{"comment":"The FPS/TTFF comparison is confounded by resolution and inference settings. The experimental setup states that OmniMate is evaluated at 480p (with DMD distillation at 384p), while baselines include MOVA-360P and other methods that are likely at their native resolutions with different chunk sizes and sampling steps. The paper does not report the resolution, number of steps, or chunk configuration for each compared method in Table 1. Thus the claimed 'highest streaming efficiency' may reflect favorable settings rather than architectural advantage. Please report matched-resolution comparisons, or provide a latency/quality Pareto plot across resolutions and steps for all methods.","section":"Table 1 / Experimental Setup"}],"minor_comments":[{"comment":"The caption labels (b) as GPC and (c) as MRCM, but the figure body shows '(b) MRCM' and '(c) GPC' with swapped tags. This should be corrected.","section":"Figure 2"},{"comment":"The acronym DMD is used without definition at first occurrence in the Method Overview. Define 'distribution matching distillation' or add it to the Related Work introduction.","section":"Method Overview"},{"comment":"The rows 'OmniMate(30s)', 'OmniMate(60s)', and 'OmniMate(240s)' are not explicitly described in the text. Clarify that these are the full model evaluated at different generation durations, and state what changes (e.g., the LLM-estimated duration) between them.","section":"Table 2"},{"comment":"Equation (4) defines m_i = 1 for i < τ and then states 'm_i = 1 denotes the execution state'. The notation should be made consistent: currently the condition is m_i = 1 when i < τ, which matches, but the sentence is slightly confusing and could be rewritten for clarity.","section":"Eq. (4)"},{"comment":"The paper references an appendix for further details on metrics and data construction, but the appendix content is not included in the submitted text. The references to 'VerseBench' also vary in capitalization ('Versebench' vs 'VerseBench'). Please ensure the appendix is available and standardize the capitalization.","section":"Dataset / Evaluation"}],"recommendation":"major_revision","confidential_remarks":"The manuscript does not clearly differentiate itself from the authors' own prior InteractiveAvatar (arXiv:2606.22905), referenced in the Related Work, which also targets real-time streaming avatar generation with consistency and intent-awareness. Given the substantial overlap in problem setting, the revision should explicitly discuss the differences and novelty relative to that prior work. Also, because the evaluation benchmark is self-constructed and not released, I would ask the editor to require benchmark and code release with the revision, or at least a detailed public description that allows independent reproduction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is a real system-level contribution: OmniMate combines joint audio-video-sound generation with explicit progress-conditioned state transitions (GPC) and multi-reference identity conditioning (MRCM) into one streaming framework. That combination is new relative to the cited real-time baselines, and the training pipeline—especially the state-transition data construction and the use of a DMD-distilled teacher to avoid prompt leakage—shows careful engineering. The ablations are the strongest part of the paper: they convincingly show that removing GPC or MRCM hurts the reported metrics, and the qualitative figures support the claim that these components do something. I believe the method is plausible and internally consistent.\n\nThe soft spots are exactly where the reader and the stress-test point. First, the GPC's behavior at inference depends on the LLM's estimated execution duration, which is never measured for accuracy. Training progress values come from ground-truth token counts; inference progress comes from an unvalidated prediction. The paper acknowledges this in the Limitations, so it is not hidden, but the acknowledgment does not make the dependency less load-bearing. The experiments do not exercise the failure mode: the 30s/60s/240s rows vary total length, not the mismatch between predicted and actual duration. That is a real gap and the paper should address it, either with a duration-estimation error metric or an ablation that perturbs the estimate.\n\nSecond, the evaluation is weaker than the claims. The benchmark is self-constructed, no code or data are released, and the FPS/TTFF comparison appears to mix resolutions and inference settings. The real-time claim also abstracts away ASR and LLM latency, which matters for an interactive system. These are not fatal—they are standard issues in this area—but they mean the numbers should be read as indicative, not definitive.\n\nOverall, this is a solid paper for a specific audience: people building interactive avatars or streaming audio-video models. It is not a paradigm shift, but it is a credible step forward. The central argument holds up as far as it can be checked without code and data. I would not cite it in my own work this year because I do not work directly on this subfield and the lack of reproducibility makes it hard to build on, but I would bring it to a reading group discussion. A serious editor should send it to peer review; the method deserves scrutiny and the gaps are fixable. My recommendation: engage with it, request the missing evaluation details and ideally code, and let the authors tighten the duration-estimation story.","headline":"Worth a referee, but the load-bearing interactive claim sits on an unvalidated LLM duration estimate and an evaluation that is thinner than the results suggest.","tokens_in":689,"tokens_out":1215,"would_cite":false,"duration_ms":31452,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"OmniMate generates real-time interactive avatars that switch between speaking and listening while preserving identity across long conversations.","keywords":["real-time generation","audio-visual synthesis","interactive avatars","streaming diffusion","identity preservation","generation progress controller","multi-reference conditioning","state transitions"],"falsifier":"Feed the system instructions of systematically varied lengths and compare the actual response durations to the LLM-predicted durations; if a significant fraction of responses show perceptible word omissions or repetitions due to duration mismatch, the progress-conditioning mechanism does not robustly guarantee coherent responses as claimed.","tokens_in":11660,"feed_emoji":"🎭","tokens_out":2282,"duration_ms":21144,"temperature":0.7,"pith_summary":"The paper tries to establish that open-ended, real-time streaming generation of audio-visual avatars is achievable by explicitly modeling generation progress and providing persistent multi-reference identity cues. The key problem is that the generation horizon is unknown and identity drifts over long interactions. OmniMate's Generation Progress Controller (GPC) lets the model know how far along it is in a response, enabling seamless transitions between execution and listening states based on an LLM-estimated duration. The Multi-Reference Conditioning Module (MRCM) uses multiple reference images and a reference speech segment to keep appearance and voice consistent across many turns. If true, this would make interactive avatars feel natural, responsive, and stable in real-time use.","feed_headline":"One framework streams real-time avatar audio-video with steady identity","feed_subtitle":"OmniMate keeps avatars looking and sounding the same across long conversations while switching between speaking and listening.","key_machinery":"Two modules carry the argument. GPC maps latent-aligned progress values (1 to 1000 during execution, 0 during listening) into a sinusoidal time embedding added to the latents, giving the model explicit awareness of how much of a response remains and enabling duration-aware chunk generation. MRCM concatenates multiple reference image latents with noisy video latents along the temporal dimension (with negative RoPE to avoid disturbing temporal positions) and replaces warm-up audio latents with reference speech latents during denoising, providing persistent visual and speaker identity cues. Training relies on a DMD-distilled bidirectional teacher to construct state-transition ODE trajectories,","core_discovery":"The central discovery is that a diffusion-based streaming model can be trained to condition each generated chunk on an explicit progress signal (GPC) and on multiple reference images plus reference audio (MRCM), yielding real-time joint audio-visual generation with state transitions and long-term identity preservation. The model achieves high generation speed (27.64 FPS) and low time-to-first-frame (3.49s) while maintaining strong identity scores and the best speech accuracy among compared methods.","pith_inferences":["The approach suggests that accurate duration estimation is a critical bottleneck; learning to predict response length from instruction semantics (rather than relying solely on an LLM) could further reduce word omissions and repetitions.","Treating identity preservation as a multi-reference conditioning problem, rather than extending temporal context, may generalize to other streaming generative tasks where long-term consistency matters.","The state-transition framework (execution vs listening) could be extended to other interactive content generation, such as live narration or real-time game avatars, where the generation horizon is unknown.","The reliance on a distilled teacher for constructing supervision trajectories hints that the method's effectiveness may scale with the teacher's temporal consistency; a stronger teacher could yield sharper transitions."],"forward_implications":["Real-time interactive avatars can sustain multi-turn conversations with low latency and seamless execution-to-listening transitions.","Long-form generation (tested up to 240 seconds) shows no significant degradation in visual quality, audio quality, or cross-modal consistency.","The framework builds on an existing joint audio-video foundation model with only lightweight adaptation, avoiding the need for massive new training data or architectures.","Explicit progress conditioning prevents premature termination and repetitive speech during open-ended interactions.","Using multiple reference images and a reference speech segment preserves identity under unseen poses and viewpoints over extended dialogues."],"fun_headline_variants":["Real-time avatar streams with long-term identity lock","One model: live audio-visual avatars that never drift","Progress-controlled streaming keeps avatars consistent","OmniMate: open-ended avatar chat at 27 FPS","Steady identity in real-time avatar conversations"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The system relies on the LLM's estimated execution duration being accurate; if the estimate is wrong, responses get truncated or repeated, breaking the natural interaction.","fun_headline_variants_meta":{"raw":{"variants":["Real-time avatar streams with long-term identity lock","One model: live audio-visual avatars that never drift","Progress-controlled streaming keeps avatars consistent","OmniMate: open-ended avatar chat at 27 FPS","Steady identity in real-time avatar conversations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000157,"raw_usage":{"total_tokens":1050,"prompt_tokens":728,"completion_tokens":322,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":247}},"tokens_in":472,"tokens_out":322,"duration_ms":4397,"temperature":1.0,"reasoning_tokens":247,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T03:49:28.815752+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Feed the system instructions of systematically varied lengths and compare the actual response durations to the LLM-predicted durations; if a significant fraction of responses show perceptible word omissions or repetitions due to duration mismatch, the progress-conditioning mechanism does not robustly guarantee coherent responses as claimed.","supporting_citations":[],"review_version":1}