{"id":"681a470e-abe8-42d2-89c7-6e666a0eccd1","arxiv_id":"2508.19320","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":7,"one_line_summary":"A new autoregressive video-generation framework for interactive digital humans, with a 64x compression autoencoder and a diffusion renderer, claims real-time multimodal control but is only demonstrated for audio input.","lead":"MIDAS is a system for generating live, interactive digital-human videos by predicting video frames with a language model and rendering them with a diffusion model. The paper claims real-time multimodal control (audio, pose, text), but the experiments only test audio-driven generation and give no quantitative measurements.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central capability claims are unsubstantiated: multimodal control (pose/text) is deferred, and real-time/low-latency operation has no measured latency or throughput numbers.","rationale":"The reader's verdict is REJECT and I agree. My strongest concern is closely related but slightly broader: the paper's central claim is not just about the DC-AE; it's about real-time multimodal interactive generation. Section 4.2 explicitly defers pose and text conditioning ('we leave pose and text conditioning as future work'), so 'multimodal control' is experimentally absent. The paper also contains no runtime numbers whatsoever. The absence of latency/throughput measurements is more load-bearing than reconstruction metrics alone because even a perfect DC-AE does not establish real-time operation. However, the reader's weakest_assumption (DC-AE fidelity) is an important sub-premise, hence 'partial' agreement. The paper does have some independent value: a clear architecture, a large dataset (claimed but not released), and a sensible noise-injection strategy, so the rejection is for insufficient evidence rather than internal inconsistency. I see no ad hominem issue; the critique is about missing evidence.","tokens_in":10397,"tokens_out":1504,"duration_ms":13705,"concrete_test":"Run the claimed system (or its reported components: Qwen2.5-3B AR backbone + PixArt-alpha diffusion head with 4 steps, 60 tokens/frame, 6-frame chunks) on a representative H800 GPU, and report end-to-end latency per generated frame (audio encoding + AR forward + diffusion denoising + VAE decoding) and sustained FPS over at least 60 seconds of streaming generation. If per-frame latency exceeds the 83 ms frame budget (12.5 fps) or 40 ms (25 fps) claimed in Section 3.4 during the long-video post-training, the 'real-time' central claim fails. Also measure DC-AE reconstruction PSNR/SSIM on held-out portrait video to verify the 64x compression premise.","verdict_should_be":"REJECT","load_bearing_attack":"The abstract and introduction claim interactive multimodal control (audio, pose, text) and low-latency real-time streaming generation. Section 4.2 explicitly states pose and text conditioning are left to future work, so the only demonstrated control modality is audio. Thus the headline 'multimodal control' claim is not supported by experiments. More importantly, 'real-time' and 'low-latency' are asserted throughout (Sections 3.3, 3.4, 4.1) but no latency, throughput, FPS, or end-to-end delay figures appear anywhere. The architecture's token budget per frame (60 tokens) for a 3B LLM backbone plus a 0.5B diffusion head with 4 denoising steps may or may not stream in real time; the paper provides no measured wall-clock evidence. Given the central claim is 'real-time interactive multimodal control,' the absence of any quantitative runtime evaluation is a load-bearing gap. Additionally, the DC-AE 64x compression is load-bearing (it determines token count and quality), but no reconstruction fidelity (PSNR/SSIM) or identity preservation metrics are reported. Section 3.2.1's 5-frame streaming history is a design choice with no ablation. The reader correctly identified the unmeasured DC-AE premise; I would add that even if DC-AE fidelity were acceptable, the 'real-time' claim remains unsupported by any measurements.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MIDAS, an autoregressive video generation framework for interactive digital humans. It claims to accept multimodal conditions (audio, pose, text), to operate in real time via streaming next-frame prediction, and to achieve long-horizon stability via a 64× compressed latent space and a diffusion head. The method combines a Qwen2.5-3B LLM backbone, a custom Deep Compression Autoencoder with causal temporal modules, a noise-injection exposure-bias mitigation scheme, and a ~20,000-hour dialogue dataset. Experiments are qualitative: duplex conversation, multilingual singing, and a Minecraft-based interactive world model. The paper does not report quantitative evaluations.","tokens_in":10807,"tokens_out":2173,"duration_ms":17680,"significance":"If the claims were substantiated, the work would be significant: a unified LLM-based autoregressive architecture for streaming multimodal digital-human synthesis with real-time latency would address a practically important gap. The design choices (LLM backbone with minimal modification, flow-matching diffusion head, deep compression VAE) are sensible and the dataset construction is a practical contribution. However, the reported evidence is entirely qualitative; core quantitative claims—latency, reconstruction fidelity, identity preservation, and multimodal controllability—are not measured. The paper should be credited for being explicit in Section 4.2 that pose and text conditioning are future work, and for providing the architecture details necessary to reproduce the pipeline, but the absence of metrics leaves the central claims unverified.","major_comments":[{"comment":"The abstract and introduction claim 'interactive multimodal control' including pose and text, but Section 4.2 explicitly states 'we leave pose and text conditioning as future work.' Only audio-driven results are shown. The multimodal-control claim is therefore unsupported by the experiments, and the headline capability is not demonstrated.","section":"Abstract/§1/§4.2"},{"comment":"The paper repeatedly asserts 'real-time,' 'low-latency,' and 'streamable' operation, yet no measured latency, throughput, FPS, or end-to-end delay is reported. With a 3B LLM, a 0.5B diffusion head, 60 tokens/frame, and 4 denoising steps, real-time operation is non-obvious; a runtime benchmark is essential to the paper's central claim.","section":"§3.3, §3.5, §4.1"},{"comment":"The Deep Compression Autoencoder with 64× spatial compression and a 5-frame causal history is load-bearing: it determines token count, reconstruction quality, and temporal consistency. No reconstruction metrics (PSNR/SSIM/LPIPS), identity-similarity scores, or ablations of the 5-frame window are provided. The entire qualitative pipeline rests on this unmeasured premise.","section":"§3.2.1 and §4"},{"comment":"The claim of 'up to one hour' and '4 minutes without significant drift' is stated without a protocol, a definition of drift, or associated quantitative evidence. A qualitative still-frame figure does not support a temporal-stability claim of this strength.","section":"§3.5/§4.2"}],"minor_comments":[{"comment":"Equation (2) defines x_noisy = σ·ε + (1−σ)·x with 'σ' as noise level, but 'σ' conventionally denotes standard deviation; the text should clarify the corruption schedule and that the same noise level is sampled per frame.","section":"§3.4"},{"comment":"The description of the DC-AE training is terse: '8-frame temporal window,' 'asymmetric replicate padding,' 'RoPE-based attention layers' are mentioned without details. Add a table or precise layer configuration.","section":"§3.2.1"},{"comment":"The dataset is a major contribution claim, but no statistics are given for the number of identities, language distribution, or hours per source; please provide a dataset table.","section":"§3.1"},{"comment":"Reference [11] is cited as both OmniHuman-1 (a) and (b) with identical content but different years; unify and correct.","section":"References"},{"comment":"Report the exact number of tokens per frame and the spatial latent resolution for 384×640 input, since 'up to 60 tokens' is ambiguous.","section":"§4.1"}],"recommendation":"reject","confidential_remarks":"The paper is a technical report with no quantitative evaluation. The core claims—multimodal control, real-time operation, and compression fidelity—are asserted rather than measured. These gaps are not easily fixable within the manuscript's scope: pose/text conditioning is explicitly deferred, and no runtime or reconstruction measurements exist. Rejection is appropriate; a resubmission with quantitative benchmarks, ablations, and measured latency would be worth considering."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful core here is the pipeline design: a Qwen2.5-3B LLM trained for next-frame prediction over 64x-compressed causal video latents, plus a lightweight diffusion render head, chunked streaming, and noise-injection to fight exposure bias. That is a sensible combination, and it borrows from the right recent work (MAR-style diffusion conditioning, FAR-style truncation, Valevski's noise corruption). The DC-AE with causal streaming and a 5-frame history is also a reasonable engineering choice, and the paper is honest that longer histories would likely help. The 20,000-hour dataset is a real resource, though we only get the pipeline description, not the data itself.\n\nWhat the paper does not deliver is evidence for its own headline. Section 4.2 explicitly defers pose and text conditioning to future work, so 'multimodal interactive control' currently means audio only. The 'real-time' and 'low-latency' claims appear throughout Sections 3 and 4 but there is not a single latency, throughput, FPS, or end-to-end delay number. Even the qualitative results are sparse: a few figures, no baseline comparisons, no reconstruction metrics for the DC-AE. The 64x compression is load-bearing (it sets the token budget and the quality ceiling), and the paper gives no PSNR/SSIM, no identity-retention measurement, and no ablation of the 5-frame history or noise-level schedule. The free parameters list in the review (noise scale 0.5, chunk size, denoising steps) are all hand-set; that is fine for a technical report, but the paper should say so plainly.\n\nI think the reader's verdict is in the right ballpark: reject as stated, but not because the idea is wrong. The architecture is coherent and the direction (AR video generation on highly compressed latents with a diffusion decoder) is genuinely current and worth watching. The paper just has not yet done the measurement work to back its capability claims. The Minecraft 'world model' bit is a cute teaser but adds little evidence.\n\nWho is this for? Practitioners building real-time talking-head systems; they will find the recipe useful even without numbers. It deserves a serious referee if the authors add quantitative evaluation, especially latency and reconstruction fidelity. As it stands, I would not block it from being discussed internally, but I would not peer-review it in this state. If the authors submit this version, my recommendation is to ask for major revision: measure the autoencoder, measure latency, run at least one baseline, and scope the claims to audio-only.","headline":"System-design paper with a coherent recipe but the headline claims (multimodal control, real-time latency) outrun what the experiments actually show, so it reads as an early technical report rather than a substantiated system paper.","tokens_in":11267,"tokens_out":660,"would_cite":false,"duration_ms":8555,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MIDAS claims that a standard language model, fed with audio, pose, and text tokens, can predict compressed video frames causally and render them with a diffusion head, making interactive digital humans real-time and multimodal.","keywords":["digital human","autoregressive video generation","multimodal control","real-time interaction","audio-driven talking face","deep compression autoencoder","flow matching","large language model"],"falsifier":"Run a live duplex session and simultaneously log two numbers: wall-clock time per 480ms chunk, and DC-AE reconstruction PSNR/SSIM on held-out portraits. Real-time interaction fails if the first exceeds 480ms; long-horizon identity coherence fails if the second is below the level of standard talking-face autoencoders.","tokens_in":10328,"feed_emoji":"🎬","tokens_out":7470,"duration_ms":68926,"temperature":0.7,"pith_summary":"MIDAS sets out to prove that real-time interactive digital humans do not need a bespoke video-generation architecture: a standard autoregressive language model, lightly adapted, can predict the next chunk of compressed video from audio, pose, and text conditions, while a diffusion head renders each frame. The payoff is a system that can carry on a duplex conversation, switch languages mid-song, and even act as an interactive world model, all without pre-specifying the full control sequence. To make this tractable, the paper introduces a 64x-spatial-compression autoencoder that shrinks each frame to 60 tokens, and a 20,000-hour dialogue dataset. If the claims hold, interactive avatar video stops being a two-stage offline pipeline and becomes a streaming token-prediction problem.","feed_headline":"One language model animates avatars live, frame by frame","feed_subtitle":"MIDAS turns audio, pose, and text into streaming tokens, enabling duplex chat in 480ms chunks.","key_machinery":"The load-bearing object is the chunked, causal token sequence: in each 480ms chunk, condition tokens (audio, pose, text) precede the reference-image token and the target frame tokens, and a frame-level causal attention mask lets each frame attend only to conditions, its own intra-frame tokens, and earlier frames. This makes the standard LLM a streaming next-frame predictor. The second pillar is the Deep Compression Autoencoder (DC-AE), a 64x-spatial-compression VAE with causal temporal 3D convolutions and rotary-position attention, streamed frame-by-frame with a 5-frame history cache; it is what makes long-horizon autoregression affordable. The diffusion head is the third pillar: it renders","core_discovery":"The paper's central claim is that interactive digital-human synthesis can be reframed as causal next-frame latent prediction in a standard LLM. Modality encoders turn audio, pose, and text into condition tokens; a reference image and six target frames are converted into compact latent tokens by a deep-compression autoencoder with 64x spatial reduction and 60 tokens per frame. The LLM predicts the next frame's tokens from the condition tokens plus history, using a chunked causal attention mask, and a lightweight diffusion-transformer head denoises those tokens into frames in four steps. Training adds controlled Gaussian corruption to context frames so the model learns to recover from its own","pith_inferences":["Because conditions are attached per chunk rather than fixed for the whole sequence, the same machinery should let a user interrupt or redirect an avatar mid-conversation, such as changing emotion, language, or subject, without restarting generation; the paper demonstrates chunk-level audio driving but not this redirect scenario explicitly.","The 64x compression is the real hidden bottleneck: if reconstruction fidelity at that ratio degrades identity, no amount of autoregressive modeling or denoising can restore it. Reporting PSNR/SSIM for the DC-AE would directly bound how far the pipeline can scale.","The framework's claim to be a 'world model' is a natural extension of the same token stream: any signal that can be encoded as tokens, such as gaze, scene layout, or game-controller input, can in principle steer the predictor, making the method a candidate backbone for controllable simulators beyond human avatars."],"forward_implications":["Real-time duplex conversation works: avatars take turns and display listening behavior, with controls streamed in 480ms chunks rather than fixed for the whole video.","Long videos stay coherent: the noise-injection scheme plus 8-chunk truncation keeps generation stable for up to an hour of a single identity and four-minute multilingual clips without significant drift.","Multimodal control is additive: audio, pose, and text are encoded into the same token stream, so combinations of controls can be swapped in without retraining the backbone.","The same architecture transfers to non-human domains: directional tokens trained on Minecraft data produce an interactive world model with visual consistency and memory.","Lower latency follows directly from compression: 64x spatial reduction cuts the number of video tokens the LLM must predict, which is what makes streaming feasible."],"supporting_citations":[{"why":"Supplies the open-source LLM backbone the autoregressive model builds on without architectural modification.","marker":"Yang et al., 2024"},{"why":"Supplies the PixArt-α DiT architecture used for the diffusion head that renders predicted latents.","marker":"Chen et al., 2023"},{"why":"Supplies the MAR-style injection of autoregressive hidden states as conditioning into the diffusion process.","marker":"Li et al., 2024"},{"why":"Supplies the controlled noise-injection strategy the paper adapts to mitigate exposure bias.","marker":"Valevski et al., 2024"},{"why":"Supplies the frame-truncation strategy used to limit context to recent chunks during long inference.","marker":"Gu et al., 2025"},{"why":"Supplies the Whisper-VQ module that compresses each 80ms audio segment into a single token.","marker":"Radford et al., 2022"},{"why":"Supplies the T5 encoder used to embed text conditions.","marker":"Raffel et al., 2020"},{"why":"Supplies the TED-LRS dataset, one of the sources for the 20,000-hour dialogue training corpus.","marker":"Afouras et al., 2018"}],"fun_headline_variants":["LLM streams digital humans frame by frame","MIDAS turns audio, pose, text into live video","Autoregressive LLM drives real-time avatar video","MIDAS: one LLM for live, multimodal avatars"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The entire system assumes that the causal streaming 64x-compression autoencoder, using only a 5-frame history, retains enough facial identity and visual fidelity for high-quality synthesis, yet the paper reports no PSNR/SSIM or comparable reconstruction measurements.","fun_headline_variants_meta":{"raw":{"variants":["LLM streams digital humans frame by frame","MIDAS turns audio, pose, text into live video","Autoregressive LLM drives real-time avatar video","MIDAS: one LLM for live, multimodal avatars"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000808,"raw_usage":{"total_tokens":3375,"prompt_tokens":726,"completion_tokens":2649,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":2585}},"tokens_in":470,"tokens_out":2649,"duration_ms":17021,"temperature":1.0,"reasoning_tokens":2585,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:59:01.195465+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a live duplex session and simultaneously log two numbers: wall-clock time per 480ms chunk, and DC-AE reconstruction PSNR/SSIM on held-out portraits. Real-time interaction fails if the first exceeds 480ms; long-horizon identity coherence fails if the second is below the level of standard talking-face autoencoders.","supporting_citations":[],"review_version":1}