{"id":"8a63cac5-c65b-4cf4-a6e7-ea53b237a5db","arxiv_id":"2506.02975","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A training recipe that initializes different depth segments of one transformer from pretrained ViT, LLM, and DiT models, then jointly tunes them to do multimodal understanding and generation.","lead":"HaploOmni is a single transformer model that both understands and generates images and videos. It is trained cheaply by starting from existing vision, language, and video-generation models, and the paper reports competitive benchmark scores with far less training compute than comparable unified models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage 1 identity/distillation losses are never defined and the warmup is only ablated qualitatively; if these losses fail to preserve ViT/DiT knowledge, the reported gains may come from the strong Qwen2.5/CogVideoX priors rather than the proposed method.","rationale":"I agree with the reader's weakest_assumption and would keep the CONDITIONAL verdict. I checked whether a different concern is more load-bearing: the table/text numeric mismatches and the undefined feature pre-scaling are serious reporting defects, but they are correctable and do not by themselves decide the central claim. The warmup-preservation issue is deeper: the entire efficiency argument rests on Stage 1 transferring ViT/DiT knowledge into one autoregressive transformer, and the paper gives no way to test that transfer. The qualitative Fig. 5 ablation and loss curves in Fig. 6 are suggestive but not sufficient; the missing benchmark-level warmup ablation is exactly the experiment that would separate 'HaploOmni's method' from 'a strong base model fine-tuned on more data.' Since the idea is plausible and the request is for definitions and an ablation, not for a fundamentally impossible fix, CONDITIONAL remains the right verdict.","tokens_in":14365,"tokens_out":9611,"duration_ms":108335,"concrete_test":"Ask the authors to (1) publish the exact equations and hyperparameters for the Stage 1 identity and distillation losses, and (2) report the quantitative version of the 'W/O Multimodal Warmup' ablation already shown in Fig. 5(a), evaluated on all headline benchmarks (SEED, POPE, MVBench, VBench) under the same training budget and data. If removing Stage 1 leaves these scores statistically indistinguishable from the full model, or if the loss definitions reveal no mechanism that could preserve teacher knowledge, the central claim of efficient prior-knowledge transfer is not supported. A minimal independent check of the reported numbers should also resolve the SEED 74.6/74.0/74.8 and POPE 88.3/89.6/87.9 discrepancies by releasing evaluation logs.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (abstract; §4.3) is that HaploOmni's multimodal warmup stage reuses pretrained CLIP-ViT-L and CogVideoX-2B weights to build a high-performing single transformer at low cost. The claim requires Stage 1 (§3.3) to preserve the teacher models' visual and generative knowledge while adapting them to Qwen2.5's autoregressive decoding framework. The only mechanisms named, 'Identity Loss' and 'distillation loss,' are never defined: no loss equations, no targets, no weighting, and no list of updated parameters appear anywhere in the paper. The only ablation of Stage 1 is Fig. 5(a), a single qualitative 'cat' image, with no benchmark numbers; Fig. 6's loss curves pertain to connectors and pre-scaling, not to the warmup. If the warmup silently damages either the ViT pre-decoder or the DiT post-decoder, the reported gains (e.g., SEED 74.0, POPE 89.6, MVBench 52.9 versus VILA-U's 59.0, 85.8, 38.9 in Tables 1/2/6) could be explained by the strong Qwen2.5-7B base and CogVideoX initialization alone, leaving the proposed method non-contributory. This is the load-bearing gap: the paper's defining mechanism is unverifiable as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HaploOmni, a single-transformer architecture for unified image/video understanding and generation. The model is split into pre-decoder, base-decoder, and post-decoder components that are initialized from CLIP-ViT-L, Qwen2.5-7B, and CogVideoX-2B, respectively, and then trained in three stages: a multimodal warmup with identity and distillation losses, a connector alignment stage, and a final unified tuning stage. The authors introduce feature pre-scaling and a multimodal AdaLN module to handle cross-modal feature mismatches. They report competitive or state-of-the-art results among unified models on several image and video understanding and generation benchmarks while using substantially fewer GPU-hours than prior unified single-transformer models such as Chameleon and Janus.","tokens_in":14732,"tokens_out":4230,"duration_ms":46862,"significance":"If the claimed results are reproducible, the paper offers a practically valuable recipe: reuse strong pretrained vision, language, and diffusion models to assemble a single unified transformer at a fraction of the training cost of from-scratch unified models. The paper makes the efficiency story concrete with a GPU-hours comparison (5792 versus 856481 for Chameleon and 21504 for Janus) and commits to releasing code. The proposed multimodal warmup and multimodal AdaLN are architecturally interesting. However, the central mechanism of the warmup is not specified in verifiable form, the reported benchmark numbers are internally inconsistent, and the generation evaluation is incomplete. These gaps currently prevent the reader from assessing whether the reported gains come from the proposed method or from the strong Qwen2.5/CogVideoX priors alone.","major_comments":[{"comment":"The identity loss and distillation loss are named but never defined: there are no loss equations, no target definitions, no loss-weighting scheme, and no explicit list of parameters updated in each warmup step. This is load-bearing because the central claim is that warmup preserves the knowledge of the pretrained ViT and DiT teachers while adapting them to the LLM's autoregressive paradigm. The only ablation is Fig. 5(a), a single qualitative image. Please provide the exact loss definitions and a quantitative ablation, e.g., benchmark scores with and without the warmup stage and with and without teacher initialization.","section":"§3.3, Stage 1 (Multimodal Warmup)"},{"comment":"The paper reports three different sets of numbers for the same model: SEED is 74.8 in the text of §4.3, 74.6 in Table 1, and 74.0 in Table 2; POPE is 87.9 in the text, 88.3 in Table 1, and 89.6 in Table 2. These inconsistencies make it impossible to verify the central benchmark claims. Please reconcile the tables and the text, and state the exact evaluation protocol used for each reported number.","section":"Tables 1, 2, and §4.3"},{"comment":"The VBench evaluation reports only 5 of the 16 benchmark dimensions and gives no error bars or number of evaluation runs. The text claims HaploOmni 'exhibits strong performance across most evaluated aspects' and is 'competitive with pure generative models,' but the selected subset does not support that claim. Please report the full VBench suite, with multiple seeds or confidence intervals where applicable, and clarify the selection criterion for the reported dimensions.","section":"Table 3 and §4.3 (VBench)"},{"comment":"The definition of the multimodal AdaLN is not fully specified: Eq. (3) uses a learnable matrix W_Ada, while Algorithm 1 uses W_MAL, and the dimensions of the state matrix and the switch-score computation are not fully tied together. In addition, the motivating claim in §3.2 that noise-token amplitudes are 'about 10 times larger' than visual features is stated without supporting measurements. Please provide the exact dimensions, unify the notation, and quantify the amplitude mismatch that motivates feature pre-scaling.","section":"§3.2, Eq. (3) and Algorithm 1"}],"minor_comments":[{"comment":"There is a typo: 'This adjHaploOmniment enables compatibility' should read 'This adjustment enables compatibility.'","section":"§3.3"},{"comment":"The dimensions of W′, W_SN, and the indicator functions I_0 and I_1 are not specified; please clarify the shapes and how the switch scores interact with the learnable transformations.","section":"§3.2, Eq. (4)–(6)"},{"comment":"The sentence 'we relax the training for the post-decoder' is unclear; please clarify whether parameters are released or relaxed, and state exactly which parameters are trainable in each of the three connector-alignment steps.","section":"§A.3"},{"comment":"The HaploOmni-Block ablation reports only three benchmarks and does not state the training data, training steps, or evaluation protocol; please provide these details so the comparison with the 'standard block' is meaningful.","section":"Table 4"},{"comment":"References [28] and [29] are identical entries for SEED-Bench; please deduplicate.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is an empirical systems paper, and the main risk is that the proposed warmup mechanism is not actually validated as described. The inconsistent numbers across the text and tables suggest that the manuscript was rushed; I would encourage the editor to ask for the raw evaluation logs or a public evaluation harness. The heavy reliance on the authors' own benchmark lineage (SEED-Bench, HaploVLM) is not by itself a problem, but the missing loss definitions and incomplete generation evaluation make the current claims unverifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The training paradigm is the real contribution here. Partitioning a single transformer into pre/base/post decoders and initializing them from CLIP-ViT-L, Qwen2.5, and CogVideoX-2B respectively is new as a package, and the cost savings are substantial—5,792 GPU hours versus 856,481 for Chameleon is a big deal if it holds up. The feature pre-scaling and multimodal AdaLN are sensible engineering, and the loss curves in Fig. 6 give at least some evidence that those components help convergence.\n\nThe paper does several things well. The architecture is clearly described, the comparison with prior unified models is useful, and the qualitative results look reasonable. The claim of state-of-the-art among unified models on SEED and MVBench would be significant if the numbers are right.\n\nBut there are real problems. First, the identity loss and distillation loss in Stage 1 are never defined—no equations, no targets, no weighting. That is the load-bearing mechanism of the paper, and it is unverifiable as written. The only ablation is a single qualitative cat image, which is not evidence. The stress-test note is right: without a clear definition, we cannot tell whether the warmup preserves ViT/DiT knowledge or whether the strong Qwen2.5 base is doing all the work. Second, the paper reports three different numbers for the same model: SEED 74.8 in the text, 74.6 in Table 1, 74.0 in Table 2; POPE 87.9, 88.3, 89.6. That is the kind of inconsistency that makes a reviewer doubt the rest of the results. Third, VBench reports only 5 of 16 metrics with no error bars, and the ablation of multimodal AdaLN is also only qualitative.\n\nThese are not fatal objections to the idea itself—the approach is plausible and worth testing—but they are fatal to the paper in its current form. The authors promise code, but it is not released, and there is no error analysis.\n\nI would send this to peer review because the idea deserves serious scrutiny and could have real impact. But the reviewers should require the authors to define the losses, fix the numbers, report full benchmark results, and release the code before acceptance. As written, it is a promising sketch, not a verified result.","headline":"A genuinely novel training recipe for unified multimodal transformers, but the core warmup losses are undefined and the reported numbers are internally inconsistent, so the central claim is not yet verifiable.","tokens_in":15240,"tokens_out":1705,"would_cite":false,"duration_ms":21609,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a single transformer can both understand and generate images and videos by initializing its three parts from pretrained vision, language, and diffusion models, then aligning them with a multimodal warmup.","keywords":["unified multimodal model","single transformer","multimodal warmup","feature pre-scaling","multimodal AdaLN","video understanding","video generation","diffusion transformer"],"falsifier":"Train the same 9B Qwen2.5-based model on the same understanding and generation data with and without the multimodal warmup, and compare SEED-Bench, POPE, MVBench, and VBench scores; if the with-warmup version does not clearly beat the no-warmup version, the warmup is not the cause. A second check is to measure the cosine distance between CLIP-ViT-L outputs and the pre-decoder's visual outputs after warmup: if the distance grows sharply, the distillation loss failed to preserve visual knowledge.","tokens_in":14192,"feed_emoji":"🎬","tokens_out":9416,"duration_ms":94969,"temperature":0.7,"pith_summary":"This paper tries to show that a single-transformer architecture can handle both multimodal understanding and multimodal generation without paying the usual training cost of unified models. The idea is to build the transformer in three depth-wise parts, initialize the parts from specialized pretrained models — a vision transformer for visual encoding, an LLM for language, and a diffusion transformer for visual decoding — and then train the parts to work together through a multimodal warmup, connector alignment, and unified tuning. Two additional mechanisms, feature pre-scaling and multimodal AdaLN, are introduced to resolve cross-modal feature-scale differences that otherwise slow or destabilize training. If the claims hold, the result is a practical recipe for upgrading a strong text LLM into a unified image-and-video model at reduced GPU cost, with scores on benchmarks such as SEED-Bench, POPE, MVBench, and VBench that are competitive with or better than prior unified models.","feed_headline":"One transformer understands video and makes it after a warmup","feed_subtitle":"Warming up with pretrained vision and diffusion models beats prior unified models at a fraction of the training cost.","key_machinery":"The load-bearing machinery is the HaploOmni block, a transformer block whose normalization is a multimodal AdaLN: an adaptive layer-normalization that stores scale, shift, and gate parameters for both condition and noise tokens and selects them by a softmax switch score for each input token. Around this block the model uses a hybrid attention mask — causal attention for text and timestep tokens, bidirectional attention for visual and noise tokens — plus pre-connector and post-connector modules that map features into a common space, and feature pre-scaling that compensates for the roughly tenfold amplitude gap between noise tokens and ViT-distilled visual features. The training procedure does the real work: a multimodal warmup stage distills each specialized teacher into its decoder segment while teaching identity mapping, followed by connector alignment and end-to-end unified tuning with next-token-prediction loss and diffusion loss together.","core_discovery":"On the paper's own terms, the central discovery is that pretrained specialized models can be reused as the three segments of a single transformer decoder: a pre-decoder initialized from a ViT, a base-decoder from an LLM, and a post-decoder from a diffusion transformer. During multimodal warmup, the pre-decoder and post-decoder are fine-tuned with identity and distillation losses so they conform to the LLM's autoregressive inference paradigm while keeping their specialized knowledge. After connector alignment and unified tuning, the same parameters perform both understanding and generation; the paper reports state-of-the-art results among unified models on most tested benchmarks, including notable video understanding scores (47.1 EgoSchema, 52.9 MVBench) and strong video generation results on VBench.","pith_inferences":["Read as a recipe rather than an architecture search, the method implies that any strong LLM can be upgraded into a unified image-video model by attaching a warmed-up ViT pre-decoder and a warmed-up DiT post-decoder; the specifics of the base LLM may matter less than the alignment procedure.","A controlled next experiment would hold the Qwen2.5 base and the training data fixed and toggle multimodal warmup, feature pre-scaling, and multimodal AdaLN one at a time, measuring full benchmark deltas rather than loss curves alone; that would show which mechanism carries the reported gains.","The same warmup-plus-connector scheme could plausibly extend to audio or 3D modalities whenever a pretrained specialist encoder and a pretrained diffusion decoder exist, since the alignment machinery is modality-agnostic in form."],"forward_implications":["Unified image-video models can inherit most of their capability from existing pretrained checkpoints, so building one no longer requires training a large transformer from scratch.","A single 9B parameter model can serve understanding and generation with one set of weights, which would simplify deployment and reduce memory overhead relative to encoder-decoder-plus-diffusion pipelines.","The reported GPU-hours gap (5,792 for HaploOmni versus 856,481 for Chameleon and 21,504 for Janus) suggests that prior-knowledge initialization, if it holds up, is the main route to affordable unified models.","Video understanding and generation can share one transformer rather than needing separate video-specific models, as shown by the MVBench and VBench gains over VILA-U."],"supporting_citations":[{"why":"Supplies the Qwen2.5-7B base-decoder whose pretrained language knowledge forms the middle of the unified transformer.","marker":"[64]"},{"why":"Supplies the pretrained vision transformer used to initialize the pre-decoder and sets the visual feature scale that pre-scaling compensates for.","marker":"[14]"},{"why":"Supplies the pretrained diffusion transformer used to initialize the post-decoder and the VAE decoder used to turn latents into images and videos.","marker":"[66]"},{"why":"Provides the principal unified video-text baseline that HaploOmni compares against on video understanding and generation benchmarks.","marker":"[57]"},{"why":"Represents the prior single-transformer unified model with discretized visual tokens, which motivates the from-scratch-training gap this paper addresses.","marker":"[62]"},{"why":"Supplies the hybrid autoregressive-and-diffusion single-model precedent that justifies combining next-token-prediction and diffusion losses in one transformer.","marker":"[70]"},{"why":"Provides the VBench benchmark used to evaluate video generation quality across subject consistency, motion smoothness, and dynamic degree.","marker":"[22]"},{"why":"Provides the Janus unified model with separated understanding and generation tokenizers, a key comparison point for unified-model performance.","marker":"[55]"}],"fun_headline_variants":["One transformer to understand and generate video after warmup","Single transformer unifies video understanding and generation efficiently","HaploOmni: one transformer learns video understanding and generation","Warmup with pretrained models yields a single video transformer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the multimodal warmup can graft a pretrained vision encoder and a pretrained video-diffusion decoder onto an LLM using identity and distillation losses without eroding what either teacher knows, so the benchmark gains come from the alignment rather than from the strong Qwen2.5 backbone alone.","fun_headline_variants_meta":{"raw":{"variants":["One transformer to understand and generate video after warmup","Single transformer unifies video understanding and generation efficiently","HaploOmni: one transformer learns video understanding and generation","Warmup with pretrained models yields a single video transformer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000565,"raw_usage":{"total_tokens":2617,"prompt_tokens":825,"completion_tokens":1792,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":1725}},"tokens_in":441,"tokens_out":1792,"duration_ms":14637,"temperature":1.0,"reasoning_tokens":1725,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T11:10:59.412937+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same 9B Qwen2.5-based model on the same understanding and generation data with and without the multimodal warmup, and compare SEED-Bench, POPE, MVBench, and VBench scores; if the with-warmup version does not clearly beat the no-warmup version, the warmup is not the cause. A second check is to measure the cosine distance between CLIP-ViT-L outputs and the pre-decoder's visual outputs after warmup: if the distance grows sharply, the distillation loss failed to preserve visual knowledge.","supporting_citations":[{"cited_title":"In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition","cited_arxiv_id":null,"evidence_quote":"Provides the VBench benchmark used to evaluate video generation quality across subject consistency, motion smoothness, and dynamic degree."}],"review_version":1}