{"id":"04c51d52-00cc-4411-8970-b27c79607614","arxiv_id":"2602.07106","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Ex-Omni is an OLLM that natively generates speech and ARKit-52 3D facial animation in one pass by using discrete speech units as temporal scaffolding and gated semantic injection.","lead":"Ex-Omni extends omni-modal large language models so they can output 3D facial animation together with speech, not just text or audio. A generalist should read it because it is one of the first openly described systems to couple LLM reasoning with talking-avatar motion, though its main quality metric is partly self-referential.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline S2F advantage is teacher-referential: Audio2Face-3D generates the training labels (Stages III/IV) and also serves as the LVE reference; independent human evidence is too small to carry the synchronization claim.","rationale":"The reader's weakest-assumption analysis correctly identifies the teacher-referential design as the central vulnerability: Audio2Face-3D is used both to synthesize the blendshape supervision (Section 4, Stages III and IV) and as the LVE reference (Section 5.1). This creates a systematic advantage for Native Ex-Omni, which is explicitly optimized against that teacher, over cascaded baselines that are not. The comparison in Table 2 is therefore not a neutral measurement of audio-visual synchronization or naturalness. The paper explicitly acknowledges this possible bias in Section 5.2 and Appendix A.3 and attempts to mitigate it with a human A/B study. That study is a real independent signal, but it is small (8 evaluators, 20 pairs each, no significance testing) and cannot by itself support the strong quantitative claim in the abstract. The remaining issues noted by the reader — unsupported latency comparison in Table 10, the mixed TQGF ablation in Table 6, and the absence of released code/data — are secondary but reinforce the need for conditional acceptance rather than full acceptance. The central architecture and training recipe are coherent, the external VoiceBench and Seed-TTS-Eval results are plausible, and the paper does not claim formal verification. The correct verdict remains conditionally accept, conditioned on an independent-reference evaluation or a statistically robust human study, plus a real cascaded latency comparison and release of artifacts.","tokens_in":19820,"tokens_out":4250,"duration_ms":41004,"concrete_test":"Re-run Table 2 with a reference that was not used to generate any training labels. Specifically, score all methods on the same prompts using (a) a held-out real mocap S2F test set (e.g., VOCASET) or (b) an independently trained S2F teacher (e.g., FaceFormer or another open-weights model), computing LVE or lip-vertex distance against that reference. If Native Ex-Omni does not consistently beat the best cascaded baseline under the new reference, the headline advantage is a teacher-fit artifact. As a secondary check, add a significance test (e.g., paired bootstrap or Wilcoxon) to the human A/B data in Table 3.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing assumption is that Table 2's LVE measurement is an unbiased proxy for 'audio-visual synchronization' when Audio2Face-3D is the reference. Section 4 shows Audio2Face-3D generated the blendshape labels for Stage III (10K TTS&Face) and Stage IV (59.34K S2S&Face, 20K TTS&Face). Section 5.1 then evaluates all methods by LVE against Audio2Face-3D. Native Ex-Omni is trained to imitate this exact teacher; cascaded EmoTalk/UniTalker baselines are not. Therefore the large advantage on A2F-Bench (3.866 vs. best cascaded 6.530) and CommonEval (4.754 vs. 6.527) is at least partly a fit-to-teacher artifact. The paper acknowledges this in Section 5.2 and Appendix A.3, but its defense — that Audio2Face-3D is trained on professional capture data — does not remove the confound: an independent S2F model could produce equally natural but different motion and be unfairly penalized. The human A/B study (8 evaluators × 20 pairs) is the only independent evidence, but it has no significance test or confidence interval, and it only compares Native Ex-Omni against Ex-Omni-based cascaded variants. Table 10's latency comparison also lacks cascaded baselines, so the 'lower latency than cascaded pipelines' component of the claim is unsupported. Thus the paper's central quantitative claim is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Ex-Omni, an open-source omni-modal large language model (OLLM) that natively generates speech and 3D facial animation (ARKit-52 blendshapes) from text/speech instructions. The core design decouples semantic reasoning from temporal generation: an LLM (Qwen3-8B) produces text hidden states, a speech unit generator uses token-as-query gated fusion (TQGF) to predict discrete speech units, and a non-autoregressive facial decoder predicts blendshape coefficients from speech-unit-derived queries plus speech-generator context. The authors introduce a multi-stage training recipe (ASR alignment, TTS pretraining, speech-face co-training, joint fine-tuning) and a dataset called InstructEx/InstructS2SF; the face labels are synthesized with NVIDIA Audio2Face-3D. Experiments report competitive speech understanding (VoiceBench), reasonable TTS (Seed-TTS-Eval), a large LVE advantage over cascaded S2F baselines, a small human A/B preference study, ablations of the proposed components, and latency measurements. The paper claims better audio-visual synchronization and lower face-generation latency than cascaded pipelines while preserving OLLM capabilities.","tokens_in":20120,"tokens_out":6362,"duration_ms":60089,"significance":"If the empirical claims held, this would be a useful early open-source contribution: it demonstrates a plausible way to extend OLLMs with native speech-aligned 3D facial animation, with a clear separation of semantic and temporal modeling. The architecture is well motivated, the training stages are detailed, and the ablations in Table 6 (e.g., the effect of the velocity loss and of speech-context conditioning) are informative. The dataset, once released, could support further work. However, the central quantitative evidence for the headline synchronization advantage is currently self-referential (the evaluation reference is also the label generator), and the supporting human and latency evidence is too thin to independently carry the claim. The contribution is therefore promising and likely salvageable, but the paper as submitted does not yet establish its central claim.","major_comments":[{"comment":"The headline S2F/T2F advantage is measured by LVE against Audio2Face-3D, the same model that generated the blendshape training labels in Stages III and IV (10K TTS&Face and 59.34K S2S&Face + 20K TTS&Face). Native Ex-Omni is thus compared with a reference it was trained to imitate, while cascaded EmoTalk/UniTalker baselines are not. The large margins (CommonEval 4.754 vs 6.527; A2F-Bench 3.866 vs 6.530) may substantially reflect teacher reproduction rather than general facial-animation quality. The acknowledgement in §5.2 that Audio2Face-3D is a strong proxy does not remove this confound. Please evaluate on held-out professional mocap data not produced by the teacher, or provide independent perceptual evidence on the actual Table 2 comparisons with inferential statistics.","section":"§5.1 and §4, Table 2"},{"comment":"The human A/B study uses 8 evaluators and 20 pairs per comparison, with no confidence intervals or significance tests. Moreover, the comparison is only Native Ex-Omni vs Ex-Omni-based cascaded variants (Ex-Omni+EmoTalk, Ex-Omni+UniTalker), so the strongest Table 2 cascaded baselines (e.g., Qwen2.5-Omni+UniTalker) are absent. This does not yet establish that native generation beats cascaded OLLM pipelines perceptually. Report per-comparison sample sizes, a paired test (e.g., Wilcoxon), and include at least one competitive non-Ex-Omni cascaded baseline.","section":"Table 3 and §5.1"},{"comment":"The abstract claims 'lower face-generation latency than cascaded pipelines', but Table 10 reports only Ex-Omni (RTF 2.158, Speech TTFT 0.029 s, Face Latency 0.012 s) with no cascaded comparison. 'Face Latency' appears to measure decoder time after speech units are available, whereas a cascaded system's face latency must include the OLLM speech-generation stage plus the downstream S2F model. Without measuring the same metric on EmoTalk/UniTalker cascades, the latency claim is unsupported. Either add direct comparisons or qualify the claim.","section":"Table 10 and §A.4"},{"comment":"Stage III is described as training the speech generator on TTS data paired with blendshape annotations, but Table 7 freezes the Speech Generator (lr = 0) and lists only the Facial Decoder with lr = 1e-3. This is internally contradictory and matters for reproducing the co-training protocol. Clarify which modules are updated in Stage III; if the speech generator is frozen, revise the prose accordingly.","section":"§3.5 vs Table 7"}],"minor_comments":[{"comment":"The dataset name is inconsistent: the abstract calls it 'InstructS2SF-1200K' while §1 and §4 call it 'InstructEx'. Also, the abstract's '1200K samples' is not obviously consistent with Table 1, whose listed counts sum to well over 1.2M; please define exactly which subset has 1200K samples.","section":"Abstract/§1 and §4"},{"comment":"The caption says 'D3 and D6 denotes UniTalker-B-D3 and UniTalker-B-D3'; the second should presumably be 'UniTalker-B-D6'.","section":"Table 3 caption"},{"comment":"'Therfore' should be 'Therefore'.","section":"Eq. (14)"},{"comment":"'EX-Omni' in the figure caption/architecture label is inconsistent with the paper's 'Ex-Omni' naming.","section":"Figure 2"},{"comment":"'Latency ana' appears to be an incomplete word; use 'Latency analysis'.","section":"Table 10 caption"},{"comment":"The translation of A2F-Bench is attributed to 'GPT-4o (Fu et al., 2025)', but Fu et al. 2025 is VITA-1.5; the GPT-4o system card is Hurst et al. 2024. The citation should be corrected.","section":"§5.1"}],"recommendation":"major_revision","confidential_remarks":"The authors are transparent about the teacher/reference bias in §5.2 and Appendix A.3, which is to their credit. However, the paper's central claim currently rests on that contaminated metric; the human and latency evidence is too thin to substitute. If independent evaluation is supplied, the contribution is likely publishable; without it, the headline should be substantially weakened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Ex-Omni is a real systems contribution: it is one of the first open-source OLLMs that natively outputs speech plus ARKit-52 3D facial animation, and the speech-unit scaffolding plus TQGF design are sensible engineering choices. The InstructEx dataset (1.2M samples, staged training) is also a tangible asset. The paper is honestly written — it flags the teacher-bias issue in §5.2 and Appendix A.3, and lists limitations in A.5. That level of candor counts for something.\n\nWhat is genuinely new is the task integration, not the components. The authors admit that TQGF follows published gated attention (Qiu et al., 2025), the speech units follow GLM-4-Voice, and periodic positional encoding follows UniTalk. The contribution is the system and the dataset, not new math.\n\nNow the soft spots, in proportion. The central S2F evaluation is teacher-referential: Audio2Face-3D generated the training blendshape labels in Stages III and IV and is also the fixed LVE reference in §5.1. The large LVE margins in Table 2 (e.g., 3.866 vs 6.530 on A2F-Bench) therefore mostly measure how well the model imitates its teacher. The defense that Audio2Face-3D is a strong professional-quality proxy does not remove the confound: an independent S2F model could produce equally natural but different motion and be unfairly penalized. The paper explicitly acknowledges this in §5.2 and Appendix A.3, which is good, but acknowledgment is not a fix.\n\nThe human A/B study is independent but small: 8 evaluators, 20 pairs per comparison, no significance testing, and it only compares Native Ex-Omni against Ex-Omni-based cascaded variants. That supports the direction but is not enough to carry the headline claim on its own.\n\nThe latency claim is weaker still: Table 10 reports only Ex-Omni's own RTF, speech TTFT, and face latency, with no cascaded baselines. The abstract's \"lower face-generation latency than cascaded pipelines\" is therefore unsupported. Also, the TQGF ablation is mixed: removing it improves Ex-A2F-EN (3.184 vs 3.377) while slightly worsening A2F-Bench (3.682 vs 3.667). The paper interprets this as a balance across languages and notes higher overhead, but that is not a clean win. Minor point: the abstract calls the dataset \"InstructS2SF-1200K\" while the body calls it \"InstructEx.\"\n\nMy take: the paper deserves a serious referee. The integrated capability is real, and the training recipe is detailed enough to reproduce. But the evaluation needs rework before the synchronization and latency claims can be accepted. A revised version should use an independent reference (or a much larger, significance-tested human study), add actual cascaded latency comparisons, and clarify what the LVE numbers do and do not mean. If the authors deliver that, this becomes a useful baseline for the subfield. If not, the paper remains a promising but unvalidated system.","headline":"A solid systems contribution — new integrated capability for OLLMs — but the headline synchronization claim is teacher-referential and the human study is too small; worth a serious referee.","tokens_in":20690,"tokens_out":3448,"would_cite":true,"duration_ms":32528,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ex-Omni claims that an omni-modal language model can natively generate speech-synchronized 3D facial animation by decoupling semantic reasoning from dense temporal generation.","keywords":["omni-modal large language models","3D facial animation","blendshape coefficients","speech-to-face generation","token-as-query gated fusion","speech units","multi-stage training","audio-visual synchronization"],"falsifier":"Take a small set of real, professionally captured speech-and-blendshape sequences that no model has been trained on, run the native model and the best cascaded pipeline on them, and compare lip vertex error against those real recordings; if the native advantage disappears, the claimed benefit is teacher-imitation rather than generalizable joint generation. A cheaper version is a larger blinded human study that scores full-face naturalness, not just lip-sync, with more than eight evaluators.","tokens_in":19625,"feed_emoji":"🗣️","tokens_out":4473,"duration_ms":43811,"temperature":0.7,"pith_summary":"Ex-Omni sets out to give omni-modal large language models (OLLMs) a missing output: 3D facial animation that moves in sync with generated speech. The paper's central claim is that the mismatch between the LLM's discrete, token-level semantics and the dense, smooth time structure of facial motion can be overcome by decoupling the two. The model uses discrete speech units as temporal scaffolding and a token-as-query gated fusion mechanism to decide when and how semantic information enters the face generator. It also contributes a large multi-stage dataset of text, speech, and synthetically labeled facial animation. The payoff, if correct, is a single open-source model that speaks and makes a face at the same time, with better lip-sync and lower marginal face-generation latency than cascaded speech-then-face pipelines, while keeping speech understanding competitive.","feed_headline":"One LLM now generates speech and 3D facial animation together","feed_subtitle":"Speech units anchor the timing, so a single model can lip-sync a 3D avatar while it speaks.","key_machinery":"The load-bearing design is the unified token-as-query gated fusion (TQGF): in every fusion step, the incoming token sequence always acts as the query and upstream semantic representations act as key/value context, with a sigmoid gate learned from the query deciding how much semantic conditioning enters at each frame. Around it sit two supporting choices: discrete speech units predicted autoregressively by a small speech generator give the face a stable temporal grid, and the facial decoder predicts all 52 blendshape coefficients in parallel with a hybrid frame-wise and velocity loss, using periodic positional encodings that bias toward rhythmic mouth motion.","core_discovery":"On its own terms, the paper establishes that an omni-modal LLM can natively generate ARKit-52 blendshape coefficients together with speech, in a non-autoregressive facial decoder, rather than routing through a separate speech-then-animation pipeline. The key move is to stop asking the LLM to emit dense motion directly: the LLM reasons, a lightweight speech generator produces discrete speech units, and those units become the temporal skeleton on which the face decoder builds, with gated cross-attention selectively injecting semantic context. The paper reports that this joint native generation is preferred by human evaluators for lip-speech synchronization and scores lower lip vertex error tha","pith_inferences":["The headline quantitative gain is measured against the same teacher model that produced the training labels, so part of the gap likely reflects imitation fidelity; a fair reader should weight the human A/B study at least as heavily as the lip-vertex numbers.","A natural extension would be to replace synthetic teacher labels with real motion-capture data for a subset and test whether the native advantage survives outside the teacher's distribution.","The same gated bridge could let LLMs drive other continuous outputs, e.g., gesture, gaze, or prosody, where hidden semantic states are too coarse to supervise directly.","The long-form speech truncation observed in the paper suggests the speech-unit budget, not the face decoder, is the next bottleneck for full conversational use."],"forward_implications":["Speech and 3D facial animation can be produced by one model rather than a speech model followed by a separate face model.","The decoupling principle (semantic tokens as scaffolding, gated injection for dense outputs) likely transfers to other dense temporal modalities such as hand gestures or body motion.","Because blendshape coefficients are identity-agnostic, the same trained decoder can drive any avatar rig, not just the training template.","Cascaded pipelines of this kind derive most of their facial quality from the downstream face model; native generation avoids that information bottleneck.","The reported sub-20 millisecond marginal face latency means adding a face to an omni-model need not add a perceptible delay."],"fun_headline_variants":["LLM now animates a 3D face while speaking","Omni-modal LLM generates speech and 3D facial motion natively","Speech units anchor timing for 3D facial animation in LLMs","Ex-Omni: single model for speech plus lip-synced 3D avatar","From LLM reasoning to 3D facial animation in one shot"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The evaluation assumes that the audio-to-face teacher model that created the facial training labels is a valid reference for natural facial animation, so the central quantitative advantage partly measures how well the model imitates its own teacher.","fun_headline_variants_meta":{"raw":{"variants":["LLM now animates a 3D face while speaking","Omni-modal LLM generates speech and 3D facial motion natively","Speech units anchor timing for 3D facial animation in LLMs","Ex-Omni: single model for speech plus lip-synced 3D avatar","From LLM reasoning to 3D facial animation in one shot"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000214,"raw_usage":{"total_tokens":1256,"prompt_tokens":733,"completion_tokens":523,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":477,"completion_tokens_details":{"reasoning_tokens":426}},"tokens_in":477,"tokens_out":523,"duration_ms":5489,"temperature":1.0,"reasoning_tokens":426,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:43:25.884361+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small set of real, professionally captured speech-and-blendshape sequences that no model has been trained on, run the native model and the best cascaded pipeline on them, and compare lip vertex error against those real recordings; if the native advantage disappears, the claimed benefit is teacher-imitation rather than generalizable joint generation. A cheaper version is a larger blinded human study that scores full-face naturalness, not just lip-sync, with more than eight evaluators.","supporting_citations":[],"review_version":1}