{"id":"13da96c5-b4c7-4896-b28c-1d95565f95d9","arxiv_id":"2607.26411","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Steering vectors from the understanding branch can control image generation, but vectors from the generation branch cannot control understanding, showing UMMs are architecturally unified but representationally asymmetric.","lead":"This paper probes whether the understanding and generation branches of unified multimodal models share one semantic space by learning steering vectors in one branch and injecting them in the other. Understanding-to-generation steering changes generated images as intended; generation-to-understanding steering does not, indicating the branches are not fully aligned.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gen→Und vectors may be read from noise-initialized VAE latents: App. A.3/Eq. 4 never fixes the denoising timestep, so the reported asymmetry could be an extraction-timing artifact rather than a semantic-space property.","rationale":"I read the paper in good faith. The U→G leg is well controlled: random-direction and unrelated-direction baselines both yield 0% SSR, the evaluator is human-validated (κ=0.894), and the qualitative examples are consistent. The load-bearing weakness is in the G→U leg. The reader's weakest assumption identified that the chosen contrastive protocol may not linearly encode semantics at the selected token positions. I agree, and I sharpen this to a specific technical gap: the paper never states the diffusion timestep at which VAE latent activations are harvested (A.3, Eq. 4, Fig. 8). If those activations come from the initial noise, the G→U failure is uninformative about the trained generation branch's semantic representations. The available ablations do not close this gap because C.2 and C.7 test extraction strategies and intervention positions using CAA at the same unspecified timestep, and they do not report ITI/RepE in the G→U setting. I do not think this invalidates the empirical U→G finding, but it materially weakens the claim that the reverse direction 'consistently fails' for reasons of semantic misalignment rather than protocol. The reader's CONDITIONAL verdict already captures this uncertainty; my concern adds a concrete check that could resolve it, so the verdict remains UNCHANGED pending that check.","tokens_in":24002,"tokens_out":7538,"duration_ms":72486,"concrete_test":"Re-run the Gen→Und pipeline for the red↔blue concept harvesting generation-branch VAE activations at three denoising timesteps: initial noise (t=0), mid-denoising (t≈25), and final latents (t≈49), using the same CAA protocol, and also train an ITI linear probe at each timestep. Apply each resulting vector to the understanding branch and measure ΔmLD/SSR. If any timestep yields positive ΔmLD or nonzero SSR, the reported asymmetry is an extraction-timing artifact; if all remain null across timesteps and methods, the claim is supported. The published code should also be inspected to verify which timestep the current experiments actually used.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central asymmetry claim rests on the premise that generation-branch steering vectors encode the generation branch's semantic content. Appendix A.3 (Eq. 4) extracts these vectors by averaging residual-stream activations over 'all VAE latent token positions' when feeding contrastive text prompts through the generation branch, but it never specifies the diffusion timestep at which these latents are read. Fig. 8 labels the inputs 'Noise VAE token,' suggesting the vectors may be computed from initial noise, before any semantic visual content has been denoised. If so, the Gen→Und failure says only that text-prompt conditioning at t=0 does not linearly encode object-level semantics; it says nothing about the trained generation branch's semantic space. The appendices that try to rule out extraction artifacts (C.2 object-aware/attention-weighted pooling, C.7 intervention positions) all use the same unspecified timestep and only report CAA-based vectors, so they do not resolve this. The PSP analysis in Sec. 5 is also self-referential: it projects generation vectors onto a subspace derived from the same contrastive activations. Thus the strongest claim—asymmetric semantic alignment—may be an artifact of when and how VAE latents are sampled.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper asks whether the understanding and generation branches of unified multimodal models (UMMs) share a transferable semantic space. It introduces cross-branch semantic steering: for 51 binary concept pairs in the new UMMSteer dataset, contrastive activations are collected from one branch (answer-token residual states for understanding; pooled VAE-latent states for generation), steering vectors are estimated with CAA/RepE/ITI, and the vectors are added to the other branch during inference. On BAGEL, understanding-to-generation (U→G) steering achieves high steering success rates on many concepts (Table 1, Fig. 1), while generation-to-understanding (G→U) steering leaves QA outputs unchanged (Appendix B.3, Table 6). The authors attribute the asymmetry to a representational mismatch: understanding-derived vectors are object-centric and transferable, while generation-derived vectors encode low-level appearance features. They also report U→G results on Janus-Pro, UniPic-1, and Show-o2, propose a PCA subspace projection metric, and demonstrate a hallucination-reduction application on relational prompts.","tokens_in":24274,"tokens_out":6665,"duration_ms":62980,"significance":"If the central asymmetry claim is correct, the paper makes a genuinely useful contribution: it provides a causal, intervention-based probe for semantic alignment in UMMs, introduces a reusable contrastive dataset, and shows that architectural unification does not automatically imply a shared semantic space. The paper also includes several good practices: a human-validated VLM judge (Cohen's κ = 0.894 on 350 images), random- and unrelated-direction baselines (Table 5), and multiple extraction/intervention ablations. However, the conclusion is currently stronger than the evidence. The generation-branch extraction protocol is underspecified in a way that could produce the observed G→U failure as an artifact, the reverse direction is tested on a single model, and no quantitative self-branch positive control for generation-derived vectors is reported. These issues are fixable with additional experiments, but they are load-bearing for the paper's main claim.","major_comments":[{"comment":"The generation-branch steering vectors are extracted from 'all VAE latent token positions' without specifying the denoising timestep at which those latents are read. The paper's own pipeline figures label the inputs as 'Noise VAE token' (Fig. 2, Fig. 8). If the contrastive activations are collected at or near t=0, the extracted vector contains essentially no learned visual-semantic content, so the G→U failure would say only that text-prompt conditioning at pure noise does not linearly encode object semantics. Appendix C.2 changes pooling (global mean, object-aware, attention-weighted) and Appendix C.7 changes intervention positions, but neither varies or even reports the timestep. This is a direct confound for the central asymmetry claim. The authors must specify the timestep and, ideally, extract vectors at multiple timesteps (early/mid/late) to show that the G→U result is not a timing","section":"Appendix A.3, Eq. (4), Fig. 8"},{"comment":"There is no quantitative positive control showing that generation-derived vectors can steer the generation branch itself. The only self-branch evidence is qualitative Fig. 5(a), which actually shows that the red→blue generation-derived vector fails to change the object color and instead alters background/illumination. That is evidence that the extraction protocol did not recover the target semantic direction, not evidence that the generation branch lacks transferable semantics. To support the asymmetry interpretation, the authors should demonstrate that some generation-branch extraction (e.g., at a particular timestep, token position, or pooling scheme) yields a vector that quantitatively steers generation in a self-branch setting, and then show that this same vector still fails to transfer to understanding.","section":"Sec. 5, Fig. 5(a), Appendix C.2"},{"comment":"The reverse (G→U) direction is evaluated only on BAGEL. The cross-architecture experiments in Sec. 6 and Table 4 cover only U→G steering. Thus the claim that the asymmetry is a general property of UMMs is not supported: on Janus-Pro, UniPic-1, and Show-o2, the paper does not test whether generation-derived vectors can or cannot steer understanding. At minimum, the G→U experiment should be run on the same additional models, or the conclusion should be explicitly limited to BAGEL.","section":"Sec. 4.2 vs. Sec. 6, Table 4"},{"comment":"The steering strength α is selected using the SSR metric on the red→blue concept, and Table 8 shows that CAA performance drops from 100% at α=0.1 to 10% at α=0.4. Because this same α is then used for all reported experiments, the headline numbers in Table 1 may be optimistic and concept-specific. The authors should either use a held-out validation split for α selection, report results across a range of α values for all concept categories, or justify why a single α chosen on one color concept is appropriate for all 51 concept pairs.","section":"Appendix C.6, Table 8"}],"minor_comments":[{"comment":"The test sample reads 'A red apple on the table. What color is this car?' — likely 'car' should be 'apple' or another object. This typo appears in the figure caption and could confuse readers.","section":"Appendix A.2, Fig. 7"},{"comment":"Several pairs (e.g., red↔blue, red↔green) appear in both the Color and Text families. If this is intentional, explain that the Text family includes color-word pairs that must be rendered as text; otherwise remove the duplicates.","section":"Table 2"},{"comment":"The caption says 'understanding ↔ generation steering' but the appendix shows only U→G examples. The directional arrow should be corrected.","section":"Fig. 17 caption"},{"comment":"The PSP metric fixes k=20 without sensitivity analysis. Since the 'semantic subspace' is built from 50 contrastive difference vectors, the interpretation of PSP as measuring alignment with the branch's semantic structure should be softened or accompanied by a k-ablation.","section":"Appendix C.4"}],"recommendation":"major_revision","confidential_remarks":"The timestep issue raised by the stress-test is real and should be the centerpiece of the revision. If the authors can show that generation vectors extracted at a meaningful denoising timestep (and validated in a self-branch positive control) still fail to transfer to understanding, the paper would make a strong contribution. I would not reject on the current evidence, but acceptance before resolving this artifact is premature."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the paper. The cross-branch steering setup is genuinely new — no one has tried transferring steering vectors between the understanding and generation branches of a UMM — and the U→G direction is convincingly demonstrated. The effects are large, the random/unrelated-direction baselines are clean (0% SSR), and the human-validated Qwen judge (κ=0.894) is a solid evaluation choice. The UMMSteer dataset covering 51 concept pairs is useful, and the out-of-template ConceptMix check is a nice touch.\n\nThe soft spot is the G→U half. The paper's central asymmetry claim is that generation-derived vectors cannot steer understanding. But the vectors are extracted by pooling VAE latent activations while feeding contrastive text prompts through the generation branch (Eq. 4), and no denoising timestep is ever specified. Fig. 8 labels the inputs 'Noise VAE token.' If those activations are read at t=0 before any denoising, the G→U failure only says that text conditioning at the initial noise step does not linearly encode object-level semantics — it says nothing about the trained generation branch's semantic space. The appendices that try to rule out extraction artifacts (C.2, C.7) use the same unspecified timestep and only report CAA vectors, so they do not resolve this. This is a load-bearing flaw, not a minor omission.\n\nThe PSP analysis in Sec. 5 is also partially self-referential: the subspace is built from the same contrastive differences used to define the vectors, so high alignment is partly by construction. That weakens the 'explanation' but doesn't by itself invalidate the transfer result.\n\nTwo lesser issues. The asymmetry is demonstrated mainly on one model (BAGEL) with one seed, and steering strength is chosen on the evaluation metric. No code or data are shipped. And the main-text claim that hybrid AR+diffusion architectures support unified semantics is explicitly walked back in Appendix B.4, which says the results show compatibility with the steering protocol, not that hybrid architectures inherently have more unified representations. The main text does not carry that caveat.\n\nBottom line: the U→G steering result is solid, novel, and worth building on. The G→U failure and the 'architectural unification does not guarantee semantic alignment' conclusion are not yet established. The paper deserves a serious referee — the question matters and the framework is useful — but the authors should be asked to specify and vary the extraction timestep, add repeated-seed intervals, release dataset/code, and align the architecture claims with the appendix. If G→U still fails at multiple timesteps, the asymmetry is real. If not, the paper becomes a much smaller U→G controllability result.","headline":"The U→G steering result is real and worth a referee; the G→U failure that anchors the 'asymmetric semantic space' claim is unproven because the generation-branch vectors may be extracted from initial noise at an unspecified timestep.","tokens_in":24745,"tokens_out":3227,"would_cite":true,"duration_ms":29195,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unified multimodal models do not think in one shared semantic space: understanding-derived steering transfers to generation, while generation-derived steering does not transfer back.","keywords":["unified multimodal models","cross-branch steering","semantic alignment","activation steering","representation probing","text-to-image generation","generation hallucination","contrastive steering dataset"],"falsifier":"Run the Gen→Und experiment again with generation vectors extracted by training a linear probe on VAE-latent activations to predict object-level attributes (or by decomposing latents into object-centric features with sparse autoencoders) at multiple layers, sweeping α densely. If any such protocol shifts held-out QA predictions above chance on color, counting, style, and appearance, the claimed asymmetry is extraction-dependent rather than representational, and the paper's conclusion would be falsified.","tokens_in":23817,"feed_emoji":"🧠","tokens_out":6132,"duration_ms":56447,"temperature":0.7,"pith_summary":"This paper asks whether unified multimodal models—systems that both understand images and generate them from text—actually think in one shared semantic space. To answer, it introduces cross-branch semantic steering: learn a direction for a concept such as 'red→pink' from one branch, inject it into the other during inference, and observe whether behavior shifts. The authors report that vectors learned from the understanding branch reliably control image generation (a red ribbon becomes pink, one object becomes three, inside becomes outside), while vectors learned from the generation branch barely move the understanding branch's answers. They trace the asymmetry to a representational mismatch: understanding-derived vectors capture object-centric semantics, whereas generation-derived vectors, under the extraction protocol used, encode low-level appearance statistics. If right, the result turns 'unified' from an architectural claim into a testable representational one, and offers steering as the test.","feed_headline":"Understanding steers generation, but not the reverse","feed_subtitle":"Cross-branch steering shows UMMs' understanding and generation branches do not share a symmetric semantic space.","key_machinery":"The load-bearing instrument is the steering vector: an additive direction in the residual stream estimated from contrastive pairs (e.g., 'blue car' vs. 'red car') by three standard estimators—mean difference (CAA), first principal component of difference vectors (RepE), and normal vector of a linear decision boundary (ITI). Extraction reads residual activations at answer-token positions for the understanding branch and at pooled VAE-latent positions for the generation branch. Transfer adds the vector, scaled by a strength α, to all tokens of the target branch at selected layers. The PCA Subspace Projection (PSP) metric—the fraction of a steering vector's energy lying in the top-20 PCA subspa","core_discovery":"Core discovery: an asymmetry in cross-branch semantic transfer. From 51 contrastive concept pairs, the authors extract steering vectors from answer-token activations in the understanding branch and inject them into VAE-latent hidden states during generation. This reliably changes attributes (red to pink, one to three, inside to outside) while preserving object identity, across three steering estimators. The reverse direction—vectors from generation-branch VAE latents, pooled by mean, object-aware, or attention weighting, applied to understanding text tokens—fails: QA accuracy stays at zero and logit differences stay near zero under dense strength sweeps. Subspace-projection analysis shows un","pith_inferences":["Beyond the paper: the asymmetry may be probe-dependent. The understanding branch is probed at answer tokens where semantics are explicitly decoded, while generation latents are probed at pooled positions; a nonlinear probe, or a probe at concept-specific VAE tokens, might find a shared space the current protocol cannot see.","Beyond the paper: cross-branch transferability could become a standard evaluation axis for UMMs—a model can top both understanding and generation benchmarks yet fail a steering test, and such failures may predict where it hallucinates.","Beyond the paper: a falsifiable design prediction follows—training generation latents with object-level semantic supervision (e.g., reconstruction or contrastive losses tied to attributes) should raise Gen→Und transfer and shrink the asymmetry.","Beyond the paper: the paper's own failure cases (wrong grounding, over-generalized attributes) suggest that stronger object grounding, not just a shared latent space, may be what enables transfer; this predicts that grounding-trained UMMs will show higher cross-branch steering success."],"forward_implications":["Text-only contrastive QA pairs can be turned into control signals for image synthesis, since understanding-derived vectors change generated attributes while preserving object identity.","Steering with understanding-derived vectors improves generation faithfulness on relational concepts (inside/outside, above/below), correcting a class of spatial hallucinations.","Generation-branch representations, as extracted in this work, cannot serve as semantic control for understanding tasks; the reverse transfer fails even under exhaustive strength and extraction ablations.","Strong performance on understanding and generation benchmarks does not imply aligned representations; cross-branch steering offers a complementary diagnostic.","Cross-branch transfer depends on architecture: hybrid AR+diffusion models support it, while two pure autoregressive models degrade or fail, suggesting shared continuous latent structure matters."],"fun_headline_variants":["One-way street: understanding guides generation in UMMs","Same architecture, different semantic spaces for UMMs","Understanding can steer UMM generation; reverse fails","Cross-branch steering exposes semantic asymmetry"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The central claim assumes that a concept's semantics are faithfully represented by a linear direction at the particular positions probed—answer-token activations for understanding and pooled VAE-latent activations for generation—so that failure to transfer means the branches genuinely lack shared structure rather than that the probe is looking in the wrong place.","fun_headline_variants_meta":{"raw":{"variants":["One-way street: understanding guides generation in UMMs","Same architecture, different semantic spaces for UMMs","Understanding can steer UMM generation; reverse fails","Cross-branch steering exposes semantic asymmetry"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000548,"raw_usage":{"total_tokens":2431,"prompt_tokens":697,"completion_tokens":1734,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":1683}},"tokens_in":441,"tokens_out":1734,"duration_ms":12391,"temperature":1.0,"reasoning_tokens":1683,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T16:31:22.001157+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the Gen→Und experiment again with generation vectors extracted by training a linear probe on VAE-latent activations to predict object-level attributes (or by decomposing latents into object-centric features with sparse autoencoders) at multiple layers, sweeping α densely. If any such protocol shifts held-out QA predictions above chance on color, counting, style, and appearance, the claimed asymmetry is extraction-dependent rather than representational, and the paper's conclusion would be falsified.","supporting_citations":[],"review_version":1}