{"id":"a9182961-49e9-4ea1-96a8-08294e9d4e69","arxiv_id":"2608.12209","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A training framework that adds a removable image-generation branch to multimodal LLMs improves visual understanding benchmarks with zero inference-time cost.","lead":"A team at ByteDance trained multimodal AI models with an extra image-generation task during training, then removed the generation machinery at deployment. The resulting models scored higher on several visual understanding benchmarks with no extra compute at inference time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim of improved aggregate understanding rests on an undefined Overall metric; with mixed per-benchmark deltas in Table 1, different equal-weight aggregations could plausibly flip the sign.","rationale":"The paper has genuine strengths: the matched controls in Section 4.2 disentangle data, architecture, and objective; Table 9 provides repeated-run statistics; and the representation-level analyses in Section 4.5 give independent evidence for the qualitative mechanism. I did not elevate the reader's weakest assumption about NEP target fidelity to the main concern. Because the target is the same projected embedding space consumed by the understanding branch, even a coarse target would still train the model to retain the representations actually used at inference, and the RefCOCO probing result is direct evidence of spatial retention. The undefined Overall metric is more load-bearing: every headline comparison and ablation is read through it, and the mixed per-benchmark deltas make the aggregate sign genuinely weighting-dependent. The fix is cheap and empirical: define the aggregation, recompute, and check robustness. If the formula turns out to be standard and the sign is robust, the conditional verdict should stand; if not, the claim should be downgraded to per-benchmark transfer claims. I therefore keep the reader's CONDITIONAL verdict.","tokens_in":23134,"tokens_out":4928,"duration_ms":44988,"concrete_test":"Recompute Overall from Table 1 for GAS and baseline at 2B and 4B under three explicit aggregation rules: (i) per-benchmark max-normalized average, (ii) average rank across benchmarks, and (iii) average z-score across benchmarks, using all 16 benchmarks and, separately, the benchmarks named in each capability block. If GAS does not beat baseline under all three rules at both scales, the 'aggregate' claim is not robust to the undefined metric. Also request the exact Overall formula used in Tables 2-10; without it the claim is unverifiable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The manuscript's central claim ('improves aggregate multimodal understanding'; Abstract, Section 3.3) is carried by an 'Overall' score that appears in Tables 2, 3, 4, 5, 6, 9, and 10 but is never defined. Sections 3.1 and 3.2 list benchmarks but provide no aggregation formula: no per-benchmark normalization, no weighting scheme, and no statement of which of the 16 benchmarks enter the Overall or the Perception/Reasoning/Count&Spatial/Video capability aggregates. This is not a cosmetic omission because the per-benchmark evidence in Table 1 is mixed: at 4B, GAS is worse than baseline on BLINK (-0.9), DynaMath (-2.4), VisuLogic (-2.5), Video-MME (-0.6), and MVBench (-1.8), while winning on several others. The claimed aggregate gain is therefore a function of the unspecified weighting; equally reasonable aggregations (max-normalized mean, rank mean, z-score mean) can plausibly flip the sign. Since the matched-control tables use the same undefined Overall as their headline evidence, the central claim is not falsifiable from the manuscript as written. This is a missing-support issue rather than an internal inconsistency; the underlying data could still support the claim if the formula is reasonable, explicit, and robust to plausible alternatives.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents GAS, a generation-guided training framework for multimodal large language models. A decoupled Mixture-of-Transformers (MoT) generation branch is trained to predict, via a Next Embedding Prediction (NEP) loss, the projected ViT embeddings of target images conditioned on instruction and source image; at inference this branch is discarded. The authors claim that this auxiliary supervision improves aggregate multimodal understanding on 2B and 4B backbones, with the most reliable gains on perception and counting/spatial benchmarks, at zero inference overhead, and they support the claim with matched-budget controls, per-task ablations, repeated-run statistics, and representation-level diagnostics.","tokens_in":23399,"tokens_out":8328,"duration_ms":70051,"significance":"If the central claim holds, the paper is practically valuable: it demonstrates a training-only mechanism for transferring generation-style supervision into understanding at no inference-time cost, with a modest training overhead (approximately 11.6% additional GPU-hours). The paper is unusually thorough in its controlled comparisons: Table 3 disentangles data, architecture, and objective; Table 4 ablates generation task categories; Table 9 reports repeated-run statistics; and Table 10 compares against reconstructed objectives on the same backbone and data. These are genuine strengths. The main weakness is evaluative rather than mechanistic: the headline 'Overall' aggregate is never defined, and the NEP target representation is not validated to carry the fine-grained spatial information on which the mechanism depends. Both issues are fixable with additional reporting and analysis, so the core idea remains plausible conditional on those fixes.","major_comments":[{"comment":"The headline 'Overall' score, and the Perception/Reasoning/Count&Spatial/Video capability aggregates, are never defined. No normalization formula, weighting scheme, or explicit list of which of the 16 benchmarks enter each aggregate is provided. This is not merely a presentation issue: Table 1 shows mixed per-benchmark deltas (e.g., at 4B, GAS is below baseline on BLINK, DynaMath, VisuLogic, Video-MME, and MVBench), so the claimed aggregate gain depends on an unspecified weighting. Since the matched-control tables use the same undefined Overall as their primary evidence, the central claim 'improves aggregate multimodal understanding' is not falsifiable from the manuscript as written. Please define the aggregation explicitly, state which benchmarks enter each capability aggregate, and report robustness of the headline numbers to plausible alternative weighting schemes.","section":"Sections 3.2–3.3, Tables 2–6, 9, 10"},{"comment":"The NEP target is z_tgt = Projector(ViT(I_tgt)) with an EMA-stabilized projector, and the paper's mechanism story is that this objective injects fine-grained spatial precision into shared representations. However, the paper never verifies that the projected target embeddings actually retain region-level spatial detail. If the frozen ViT plus EMA projector compress away small-object or boundary information, the generation loss would reinforce a coarse or noisy target, and the reported spatial-perception gains would not follow from the stated mechanism. Please add a direct check, for example linear probing of the target embeddings on RefCOCO and ImageNet compared with raw ViT features, or a localization/retrieval probe on z_tgt.","section":"Section 2.1, Eqs. (1)–(2); Section 4.5"},{"comment":"The matched controls used to attribute the gain to NEP+MoT (understanding-only baseline, +11% data, same generation data with no target loss, MoT-only, and no-shift visual prediction) are all single runs, while only GAS is repeated: the 2.5B GAS mean is 48.46 ± 0.37 across three runs. The difference between GAS and the MoT-only control is 0.62 pp, which is less than two standard deviations of the repeated GAS runs, and no variance is reported for any control. The conclusion that GAS beats the matched controls therefore needs repeated baselines for at least the key controls, or a variance estimate, before the attribution claim is supported.","section":"Table 3 vs. Table 9"},{"comment":"The per-task ablation in Table 4 uses an 'Understanding only' baseline of 47.63, which is not the 47.25 understanding-only baseline of Table 3 but instead coincides with the 'MoT, no generation supervision' row. If the per-task rows were added to the MoT-only architecture rather than the standard baseline, the text should say so; otherwise the per-task deltas are computed against an inconsistent reference, and the 'complementary effect' claim (Count&Spatial 75.72 in the All row vs. 73.15 from the best single task) is not comparable to the main results.","section":"Tables 3 and 4"}],"minor_comments":[{"comment":"The text says VisuLogic is the 'lone exception' at 4B, but Table 1 also shows 4B regressions on BLINK (-0.9), DynaMath (-2.4), Video-MME (-0.6), and MVBench (-1.8); the sentence should be corrected to reflect the full set of per-benchmark regressions.","section":"Section 3.3"},{"comment":"In the LLaVA-v1.5 row, the entry '16.68.52' appears malformed and should be split into separate benchmark scores.","section":"Table 1"},{"comment":"The linear probing protocol is under-specified: please report the feature extraction layers, probe training data, optimizer, and whether the reported accuracies are on a held-out split.","section":"Section 4.5"},{"comment":"The description of the left panel says 'cosine similarity between per-layer visual tokens and input visual features' but does not state how the input features are defined or how per-token similarities are aggregated; please clarify.","section":"Figure 4"},{"comment":"The label 'General Perception (Multi-Image)' is misleading because MME and MMMU are not multi-image benchmarks; consider renaming the group.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is well-executed in its controlled comparisons, but the undefined Overall metric is a blocking issue because the per-benchmark evidence is genuinely mixed. If the authors provide an explicit aggregation formula and show robustness to plausible alternative weightings, the central claim may well survive; the NEP target check and repeated-control runs are also readily addable. I see no evidence of intentional misreporting, but the 'lone exception' sentence and the Table 4 baseline inconsistency need correction. I would be supportive after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a well-executed empirical paper with a genuinely useful set of controls, but its headline claim is currently unfalsifiable because the aggregate Overall score is never defined. That needs to be fixed before the conclusions can be trusted.\n\nWhat's new: the specific combination, instruction-conditioned next-embedding prediction on a decoupled MoT branch that is discarded at inference, with generation tasks chosen for cognitive correlation with understanding, is not in the prior work I've seen. The authors also do real work to show when generation helps: Table 3 separates extra data from the objective and the architecture, and the conclusion that neither the branch alone nor data exposure alone explains the gain is convincing. Repeated runs with means and standard deviations (Table 9), layer-injection ablation (Table 6), per-task ablation (Table 4), and the prompt-rewrite T2I control (Table 5) all point in a consistent direction. They even report a regression on VisuLogic at 4B and explain the training overhead explicitly. That's honest reporting.\n\nThe soft spot is not the method; it's the yardstick. The Overall score appears in Tables 2, 3, 4, 5, 6, 9, and 10 and drives the central claim, but the paper never says which of the 16 benchmarks enter it, how the capability aggregates are formed, or how scores are normalized. This matters because Table 1 is genuinely mixed: at 4B, GAS is worse than its baseline on five benchmarks (BLINK, DynaMath, VisuLogic, Video-MME, MVBench). An equally plausible weighting could flip the sign of the aggregate. That is not an internal inconsistency, but it means the headline is not verifiable from the manuscript.\n\nA smaller concern: the NEP target is Projector(ViT(I_tgt)) with an EMA of the projector, and the paper does not directly demonstrate that these targets retain the fine-grained spatial information the authors claim to inject. The RefCOCO linear probing is suggestive, but it is about the model's representations, not the target embeddings themselves. This is minor compared to the Overall issue.\n\nWho this is for: people who train MLLMs, particularly those interested in generation as auxiliary supervision. The method is practical and the analysis is informative. I'd send it to peer review, with the explicit requirement that the aggregation metric be defined and that the main table include baseline error bars. If the authors can show the Overall is robust to reasonable weighting choices, this becomes a solid conditional accept. If they can't, the claims need to be restated per-benchmark.\n\nMy recommendation: engage, but make the metric definition a hard requirement.","headline":"A carefully controlled empirical study whose headline 'improves aggregate understanding' claim rests on an undefined Overall score; the per-benchmark and ablation evidence is solid enough to deserve revision, not rejection.","tokens_in":23950,"tokens_out":3569,"would_cite":false,"duration_ms":31632,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training a multimodal model to predict projected image embeddings, through a decoupled generation branch, improves perception and spatial understanding, and discarding the branch at inference makes the gains cost-free.","keywords":["multimodal large language models","visual generation as auxiliary supervision","next embedding prediction","mixture-of-transformers","zero inference overhead","fine-grained spatial understanding","representation learning"],"falsifier":"Take the GAS training recipe and replace each target image's embedding with a spatially shuffled version of itself, keeping the global statistics intact; if counting and spatial scores still rise, the spatial content of the NEP target is not what carries the effect. Or probe the slowly-updated target projector with linear classifiers on small-object crops; if those target embeddings do not separate fine-grained spatial categories, the claimed injection of region-level detail is impossible.","tokens_in":22930,"feed_emoji":"👁️","tokens_out":13029,"duration_ms":103631,"temperature":0.7,"pith_summary":"GAS reworks visual generation from an output to a teacher. During training, a parallel generation branch receives the hidden states of a multimodal model and must autoregressively predict the projected embedding of a target image, given the text-and-image context; the loss is a cosine distance in the exact embedding space the language model consumes. Because the generation branch is discarded after training, the deployed model has exactly the same inference cost, latency, and memory as the baseline. The paper reports that this training-only supervision improves aggregate understanding on 2B- and 4B-parameter backbones, with the most reliable gains in perception, counting, and spatial reasoning, while controlled ablations indicate that the benefit comes from task-conditioned embedding prediction through a shared lower visual pathway, not from extra data or compute alone.","feed_headline":"Generation-guided training sharpens visual understanding for free","feed_subtitle":"An image-prediction loss during training lifts perception and spatial scores, and the generator is dropped at inference.","key_machinery":"The load-bearing mechanism is the pair NEP plus MoT decoupling. NEP replaces pixel or discrete-token targets with the continuous embedding of the target image, $\\hat{z}^{\\mathrm{tgt}}_i = f^{\\mathrm{gen}}(x_{\\mathrm{ctx}}, \\hat{z}^{\\mathrm{tgt}}_{<i}; \\Theta_{\\mathrm{gen}})$, optimized by the cosine loss $L_{\\mathrm{gen}} = \\frac{1}{N}\\sum_i \\left(1 - \\frac{\\hat{z}^{\\mathrm{tgt}}_i \\cdot z^{\\mathrm{tgt}}_i}{\\|\\hat{z}^{\\mathrm{tgt}}_i\\|\\|z^{\\mathrm{tgt}}_i\\|}\\right)$. To prevent the moving target projector from destabilizing training, the target embeddings come from an exponential moving average of the active projector. The MoT branch copies upper-layer weights from the understanding backbone, splits at $l_{\\mathrm{split}}\\approx L/2$, and lets generation gradients flow into the shared lower trunk while the upper understanding layers see only the text cross-entropy loss. That asymmetric gradient flow lets the generation loss inject fine-grained spatial pressure into representations that the understanding branch must consume, while keeping the understanding side's own optimization intact.","core_discovery":"At its center, the paper claims that autoregressive prediction of continuous visual embeddings—Next Embedding Prediction (NEP)—works as auxiliary supervision for visual understanding, provided it is routed through a Mixture-of-Transformers (MoT) decoupling. The target is the projected embedding of a target image, $z_{\\mathrm{tgt}} = \\mathrm{Projector}(\\mathrm{ViT}(I_{\\mathrm{tgt}}))$, computed by a frozen vision encoder and an EMA-stabilized projector, so it lives in the same continuous space as the LLM's inputs. A generation branch consisting of independently initialized transformer layers reads hidden states at an intermediate layer (around $L/2$) and predicts that embedding sequence; only the shared lower layers and the projection receive generation gradients, while the upper understanding layers are supervised solely by the text objective. At inference the generation branch and its vision head are removed, so the understanding model is structurally identical to a standard baseline. The paper's evidence for the claim is a set of paired runs at two scales showing aggregate gains, representation diagnostics (better visual-information retention in deep layers, sharper attention on relevant regions, linear-probing gains on a region-level task but not on global classification), and ablations showing that the gains scale with the correlation between the generation task and the target capability.","pith_inferences":["An implication left implicit in the paper: the NEP target's information ceiling is set by the frozen vision encoder and the EMA-stabilized projector, so swapping in a higher-resolution or denser-feature encoder should amplify GAS's spatial gains; that is a direct testable prediction of the paper's own mechanism.","Because the method needs only projected embeddings as targets and discards the generation branch at inference, the same recipe could be applied to video, audio, or 3D data by treating temporal or spatial chunks as prediction targets; the paper lists video as future work, so the zero-overhead property is a natural extension, not an observed result.","The 'task correlation' principle implies an automatic data-selection heuristic: score a candidate generation sample by how strongly its target embedding overlaps the representations probed by the target benchmark, and prioritize those samples; the paper's Rewrite-versus-Easy experiment is evidence for the principle but stops short of a full curriculum.","The zero-inference-overhead claim should not be read as 'free training': the paper reports roughly 11.6% more GPU-hours, so GAS is a one-time training-budget tradeoff against a permanently cheaper serving model."],"forward_implications":["A GAS-trained model is deployed with exactly the same inference-time architecture as its baseline twin, so the understanding gains cost nothing at serving time despite roughly 11.6% more training GPU-hours.","Because the gains appear when GAS is applied from scratch, after large-scale pretraining, and after instruction fine-tuning, it can be inserted into existing training pipelines as a stage-agnostic augmentation.","The task mix determines the transfer profile: grounding and segmentation improve counting and spatial benchmarks, Visual-CoT improves reasoning benchmarks, and combining all five task categories beats the best single category on counting and spatial scores, indicating complementary supervision.","Within a fixed task and data volume, rewriting text-to-image prompts to carry compositional and world-knowledge structure makes the same samples transfer substantially better, showing that task correlation to the target capability, not sample count, drives the gain.","Representation diagnostics tie the gains to three mechanisms: deeper layers retain more visual information, attention stays concentrated on question-relevant image regions, and region-level linear probing improves while global classification stays flat; pure-text reasoning scores are not degraded and increase slightly in the reported settings."],"supporting_citations":[{"why":"Shows that autoregressive prediction of continuous embeddings is a strong visual learner; the paper builds NEP on this result.","marker":"[20]"},{"why":"Reports that autoregressing on LLM input embeddings can improve understanding at large scale; GAS differentiates itself by task-conditioned cross-image prediction and asymmetric decoupling.","marker":"[21]"},{"why":"Supplies the Mixture-of-Transformers architecture (shared lower trunk, parallel upper branches) that GAS adapts to isolate generation gradients.","marker":"[24]"},{"why":"A unified model that retains generation-side parameters at inference and shows plateauing understanding; motivates GAS's discardable branch and serves as a headline baseline.","marker":"[13]"},{"why":"Provides evidence that joint generation-and-understanding training degrades understanding, supporting GAS's decision to shield the upper understanding layers.","marker":"[22]"},{"why":"Documents modality conflict between generation and understanding gradients, further motivating the decoupled training route.","marker":"[23]"}],"fun_headline_variants":["Auxiliary image prediction boosts vision, costs nothing at inference","Train with image prediction, get sharper vision at no cost","Zero-overhead vision boost via auxiliary generation","Decoupled embedding prediction sharpens vision, zero inference cost"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the projected embedding of the target image—the thing the generation branch is trained to predict—still contains the fine-grained spatial detail (object boundaries, small-object structure, location) that the paper says this supervision injects; if a frozen vision encoder and a slowly-updated copy of the projector have already squeezed that detail out, the loss can only reinforce a coarse or noisy target.","fun_headline_variants_meta":{"raw":{"variants":["Auxiliary image prediction boosts vision, costs nothing at inference","Train with image prediction, get sharper vision at no cost","Zero-overhead vision boost via auxiliary generation","Decoupled embedding prediction sharpens vision, zero inference cost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000558,"raw_usage":{"total_tokens":2711,"prompt_tokens":1063,"completion_tokens":1648,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":679,"completion_tokens_details":{"reasoning_tokens":1583}},"tokens_in":679,"tokens_out":1648,"duration_ms":9036,"temperature":1.0,"reasoning_tokens":1583,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:12:18.875865+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the GAS training recipe and replace each target image's embedding with a spatially shuffled version of itself, keeping the global statistics intact; if counting and spatial scores still rise, the spatial content of the NEP target is not what carries the effect. Or probe the slowly-updated target projector with linear classifiers on small-object crops; if those target embeddings do not separate fine-grained spatial categories, the claimed injection of region-level detail is impossible.","supporting_citations":[{"cited_title":"Mixture-of-transformers: A sparse and scalable architecture for multi-modal foundation models","cited_arxiv_id":null,"evidence_quote":"Supplies the Mixture-of-Transformers architecture (shared lower trunk, parallel upper branches) that GAS adapts to isolate generation gradients."},{"cited_title":"UniFork: Exploring Modality Alignment for Unified Multimodal Understanding and Generation","cited_arxiv_id":"2506.17202","evidence_quote":"Provides evidence that joint generation-and-understanding training degrades understanding, supporting GAS's decision to shield the upper understanding layers."}],"review_version":1}