The reviewed record of science sign in
Pith

arxiv: 2607.06445 · v1 · pith:YWUGTIEF · submitted 2026-07-07 · cs.CV · cs.AI

Analysis-by-Proxy: Localization Signals in VLMs Operating as Condition Encoders

Yoav Baron , Sara Dorfman , Roni Paiss , Daniel Cohen-Or , Or Patashnik This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-07-08 05:32 UTCglm-5.2pith:YWUGTIEFrecord.jsonopen to challenge →

classification cs.CV cs.AI
keywords vision-language modelsimage editingcondition encodinglocalizationinterpretabilityQ-Formerdiffusion transformerhidden states
0
0 comments X

The pith

VLM localization signals hide in middle layers, not the final one

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates why image editing pipelines that use Vision-Language Models (VLMs) as condition encoders frequently mislocalize edits in complex scenes, even though the same VLM can accurately identify the correct target object when allowed to generate text autoregressively. The authors hypothesize that the gap arises because the VLM is restricted to a single forward pass in the editing pipeline, preventing the spatial reasoning that normally emerges through sequential decoding from reaching the final-layer representations that pipelines extract for conditioning. To test this, they introduce Analysis-by-Proxy: they train a lightweight, interpretable Q-Former model on the VLM's intermediate hidden states to predict the bounding box of the edit target directly, bypassing autoregressive generation. They find that localization information is sparsely and unevenly distributed across the VLM's layers, peaking in intermediate layers (roughly 15-24 of 28) rather than the final layer, and that the specific layer carrying the strongest signal varies per input. At the token level, the signal concentrates on a few semantically meaningful nouns and adjectives. The authors then show that feeding the proxy's predicted bounding box as an explicit visual cue to the diffusion model measurably improves edit localization accuracy compared to the baseline pipeline and several alternatives.

Core claim

The central discovery is that when a VLM operates as a single-pass condition encoder, its spatial localization signal does not reliably propagate to the final-layer hidden states that editing pipelines conventionally extract. Instead, the signal remains decodable in intermediate representations whose optimal layer varies by input, and it is sparsely concentrated on specific semantically significant instruction tokens. A lightweight proxy trained to predict bounding boxes from these intermediate states can recover this hidden signal and, when the recovered bounding box is overlaid as an explicit spatial cue for the diffusion transformer, it corrects downstream localization failures.

What carries the argument

Analysis-by-Proxy: a Q-Former (a small transformer with learned cross-attention queries) trained on VLM hidden states from a single forward pass to predict the bounding box of the edit target. The proxy's attention patterns serve as a diagnostic instrument: its attention mass across VLM layers reveals which layer carries the strongest localization signal for a given input, and its attention mass across tokens within that layer reveals which specific instruction tokens anchor the spatial information.

If this is right

  • Editing pipelines that extract conditioning signals from fixed VLM layers (whether final, pooled, or hardcoded intermediate indices) are structurally suboptimal because the layer carrying the strongest localization signal shifts depending on the input.
  • Input-dependent layer selection for conditioning signals could become a standard design principle, replacing the current practice of extracting from a predefined layer configuration.
  • The Analysis-by-Proxy framework could be applied to other tasks where a generative model is repurposed as a non-generative encoder, to diagnose which internal representations carry task-relevant information and whether they survive to the extracted layer.
  • The finding that spatial signals are sparsely concentrated on specific semantic tokens suggests that token-level pruning or dynamic selection during conditioning could improve both efficiency and accuracy.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the proxy's layer and token preferences are artifacts of the Q-Former's own architecture rather than properties of the VLM, then the specific layers and tokens identified as 'dominant' may not generalize to other proxy designs. A control experiment with a structurally different proxy (e.g., an MLP or a different cross-attention architecture) would test whether the observed sparsity patterns are
  • The 31.5% localization gap between the VLM's autoregressive predictions (89%) and the editing pipeline (57.5%) is attributed to single-pass information loss, but an alternative explanation is that the DiT itself fails to decode spatial cues even when they are present in the conditioning signal. The paper investigates the latter possibility but does not fully rule out the former as a contributing f
  • If intermediate-layer localization signals are genuinely input-dependent, then a conditioning architecture that learns to dynamically route representations across layers based on the input could close the gap without requiring an explicit bounding-box overlay, potentially offering a more elegant solution than the two-stage pipeline proposed.
  • The finding that final-layer representations over-abstract spatial cues raises the question of whether this is an inherent property of transformer depth or a consequence of the autoregressive pre-training objective, which could be tested by comparing VLMs trained with different objectives.

Load-bearing premise

The proxy (Q-Former) trained to predict bounding boxes from VLM hidden states is assumed to faithfully reveal where localization information actually resides in the VLM. If the Q-Former's own architectural biases cause it to prefer certain layers or tokens regardless of the VLM's actual information flow, then the observed sparsity patterns could be artifacts of the proxy rather than properties of the VLM.

What would settle it

Train a structurally different proxy (e.g., a simple linear probe or an MLP) on the same VLM hidden states for the same bounding-box prediction task. If the alternative proxy identifies a different set of dominant layers or tokens, or if it does not exhibit the same sparse, input-dependent layer-routing pattern, then the observed patterns are likely artifacts of the Q-Former architecture rather than genuine properties of the VLM's internal representations.

Figures

Figures reproduced from arXiv: 2607.06445 by Daniel Cohen-Or, Or Patashnik, Roni Paiss, Sara Dorfman, Yoav Baron.

Figure 1
Figure 1. Figure 1: The intriguing discrepancy: standalone VLM localization succeeds, while VLM-conditioned editing mislocalizes the target. VLMs rely on the model autoregressively generating text (Kaduri et al., 2024; Cohen et al., 2026; Nikankin et al., 2025). Consequently, despite the growing adoption of VLMs as condition encoders, their internal behavior in this restricted operating mode remains under-explored. Our anal￾y… view at source ↗
Figure 2
Figure 2. Figure 2: Failure cases across diverse editing tasks. For each pair: Left: Inputs overlaid with accurately extracted VLM bounding boxes. Right: Resulting failed edits. From left to right, failure modes include: localization errors (modifying the wrong object in a sequence), leakage (applying an intended attribute to multiple similar objects), hallucination (generating a new object rather than altering the target), a… view at source ↗
Figure 3
Figure 3. Figure 3: Proxy architecture (left) and the proxy training scheme (right). 2. Architectural Simplicity: Compared to the DiT, the Q-Former operates with substantially simpler mechan￾ics. It requires only a single forward pass, rather than multiple denoising steps, consists of fewer and smaller components, and is significantly more lightweight to train. 3. Signal Clarity: The Q-Former is trained on a single lo￾calizat… view at source ↗
Figure 4
Figure 4. Figure 4: Mean localization error of the Q-Former proxy (measured by bounding box center distance) across different VLM layer configurations. The baseline model trained on the standard final-layer representations, H(27) (red), exhibits slow convergence. Similarly, proxies trained on early and very late layers, such as l ∈ {0 . . . 13, 25 . . . 26} (blue), demonstrate poor performance and struggle to converge. In con… view at source ↗
Figure 5
Figure 5. Figure 5: Decomposing the Proxy. (A) We identify “decisive” attention heads in the Q-Former proxy, where each query assigns substantial weight to a single layer. While these layers are input-dependent, these heads maintain a sparse attention pattern (|r1 −r2| > T). (B) Within a decisive head and the dynamically identified VLM layer, we pinpoint the dominant hidden states anchoring the spatial signal. Decoding these … view at source ↗
Figure 6
Figure 6. Figure 6: Sparsity analysis of the Q-Former proxy’s attention mechanisms, demonstrating highly localized attention patterns at both the layer and token levels. level attention distribution exclusively within the identified dominant layer l ∗ . Applying the same magnitude-based ranking to the attention scores a (l ∗ ,i) h,q , for every query q to VLM token i, in heads h ∈ H, reveals an equally steep drop-off immediat… view at source ↗
Figure 7
Figure 7. Figure 7: Analysis of VLM internal representations across multiple editing instructions. artifact from the final generated output. Evaluation We evaluate our approach with Qwen-Image￾Edit against several established image editing pipelines and alternative conditioning strategies. Specifically, we compare our performance to state-of-the-art models including FLUX￾Kontext (Labs et al., 2025), FLUX.2 (Labs, 2025), and F… view at source ↗
Figure 8
Figure 8. Figure 8: Qualitative comparison of image editing results. While existing methods frequently modify incorrect objects or apply changes broadly across the scene, our method successfully localizes the edit to the intended target. This demonstrates that providing explicit spatial conditioning enables more precise and reliable edit localization. Additional qualitative comparisons are provided in [PITH_FULL_IMAGE:figure… view at source ↗
Figure 9
Figure 9. Figure 9: The horizontal axis shows LPIPS measured outside the ground-truth bounding box (LPIPS background), capturing struc￾tural changes to the surrounding scene. The vertical axis shows the mean VQA score (Gemini 2.5 Pro) evaluating subject modification, background cleanliness, and overall instruction adherence. Our method achieves the highest mean VQA score among all meth￾ods while maintaining low background dis… view at source ↗
Figure 10
Figure 10. Figure 10: Green: the explicit bounding box prediction generated by the trained Q-Former proxy, which isolates the target region to guide the downstream localized edit. The bottom row shows the corresponding failures of the original Qwen-Image-Edit pipeline on the same inputs. C.2. Additional Qualitative Results We provide further qualitative evidence supporting the claims in the main paper [PITH_FULL_IMAGE:figures… view at source ↗
Figure 11
Figure 11. Figure 11: Additional qualitative comparisons of image editing results. As in the main paper, our method successfully localizes the edit to the intended target (the lowest quartz cluster, the leftmost server rack), while existing methods frequently modify incorrect objects or apply changes broadly across the scene. the positive side, our Analysis-by-Proxy framework contributes to the growing field of VLM interpretab… view at source ↗
read the original abstract

Vision-Language Models (VLMs) are increasingly utilized as the conditioning backbone for diffusion-based image editing due to their remarkable multimodal reasoning capabilities. While standalone VLMs demonstrate strong localization capabilities, editing pipelines frequently struggle to maintain this accuracy, particularly in complex, multi-entity scenes. In this work, we investigate this performance gap, hypothesizing that it stems from treating the VLM as a condition encoder. In this role, the model is restricted to a single forward pass, preventing the autoregressive generation process for which it was optimized, thereby failing to fully expose its capabilities. To investigate whether this spatial understanding persists when the VLM is used as a condition encoder, we introduce Analysis-by-Proxy. In this framework, we train a lightweight, interpretable proxy model on the VLM's intermediate representations using an auxiliary localization task. By analyzing the VLM through this proxy, we uncover the specific VLM representations that encode localization information. Our findings expose a fundamental mismatch between how spatial knowledge is represented within a VLM condition encoder and how it is extracted by current editing pipelines. We reveal that under single-pass constraints, the localization signal does not reliably propagate to the predefined layer configurations commonly used for conditioning. Instead, this crucial signal remains hidden within intermediate representations, at locations that vary depending on the input prompt. Using our introduced Analysis-by-Proxy framework, we reveal the fundamental failures of existing condition extraction strategies in editing pipelines, opening the door to more principled design of conditioning architectures.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 6 minor

Summary. The paper introduces 'Analysis-by-Proxy,' a framework to investigate whether localization signals in Vision-Language Models (VLMs) persist when the models are restricted to a single forward pass as condition encoders for diffusion-based image editing. Using a Q-Former as a proxy, the authors train on an auxiliary bounding box prediction task using the hidden states of Qwen2.5-VL-7B. They find that localization signals are more decodable from intermediate layers than from the final layer, and that the optimal layer varies by input. The paper validates these findings by showing that extracting this signal via the proxy and feeding it as an explicit bounding box to a DiT improves edit localization in complex scenes.

Significance. The paper addresses a timely and well-motivated problem: the discrepancy between a VLM's strong localization capabilities in autoregressive settings and its poor performance when used as a single-pass condition encoder. The Analysis-by-Proxy framework offers a structured way to probe internal representations without autoregressive sampling. The downstream validation (Table 1, Figure 9) provides a concrete, falsifiable demonstration that recovering intermediate signals improves edit localization. The authors also commit to releasing code, model weights, and datasets, which supports reproducibility.

major comments (4)
  1. §4.2–§4.4: The central analytical claim—that localization signals peak in input-dependent intermediate VLM layers rather than final layers—rests entirely on the Q-Former proxy's convergence behavior (Figure 4) and attention patterns (Figures 5–6). The Q-Former is a 6-layer cross-attention transformer with learned queries, which is substantially more expressive than the linear or shallow probes that are standard in the probing literature the paper cites (Belinkov, 2022). The concern is concrete: the Q-Former's cross-attention mechanism may find it easier to decode bounding-box coordinates from intermediate-layer representations due to geometric alignment with its own key-value projection space, rather than because those layers contain more localization information in an architecture-independent sense. The paper justifies the Q-Former choice (§4.2, advantages 1–3) but never addresses this.
  2. §4.6, Table 1: The downstream validation cannot cleanly isolate the proxy's contribution. The 'Ours' method adds both the proxy-predicted bounding box AND a LoRA fine-tuning module (§B.2) to the baseline Qwen-Image-Edit. The baseline does not have this LoRA. It is unclear how much of the improvement in VQA scores and LPIPS is due to the explicit spatial cue versus the additional LoRA-adapted capacity. A control where the LoRA is trained with ground-truth bounding boxes (rather than proxy-predicted ones) would help disentangle these contributions and validate the proxy's utility.
  3. §A.1: The ground-truth bounding boxes used for proxy training are generated by Qwen2.5-VL-7B-Instruct, the same model whose hidden states are being analyzed. This creates a partial circularity: the proxy is trained to recover what the VLM can produce autoregressively, and the paper then shows the proxy can recover it from single-pass hidden states. While this does not invalidate the findings, it limits the interpretive scope. The paper should explicitly discuss whether the proxy is discovering general localization signals or merely learning to shortcut the VLM's own autoregressive decoding path.
  4. §4.1: The evaluation set consists of only 200 samples, curated by the authors. The paper states that existing datasets were 'overly noisy' (§A.2), but the small size and synthetic nature of the evaluation set (generated via FIBO and Gemini) raise concerns about generalizability. The 31.5% localization gap (89.0% vs 57.5%) and the downstream improvements in Table 1 are the paper's key empirical pillars; they would be more convincing on a larger, more diverse, or non-synthetic benchmark.
minor comments (6)
  1. Figure 4: The y-axis label 'Pred-GT Centers Diff' is unclear. It would benefit from explicit units (e.g., normalized coordinates or pixels) and a clearer description in the caption.
  2. §4.4: The threshold T for identifying 'decisive' attention heads (|r1 - r2| > T) is mentioned in Figure 5 but its value is not specified in the main text or the appendix. This should be reported for reproducibility.
  3. Table 1: The 'Norm-AVG' method has the best LPIPS and L2 scores but lower VQA scores. The paper should discuss this trade-off more explicitly in the text, as it suggests the norm-averaging technique preserves background structure but fails to localize edits.
  4. §B.1: The loss function is written as L = λ_GIoU * L_GIoU + λ_L1 * L_L1 in the main text (§4.3) but as L_total = (1-λ)L_L1 + λL_GIoU in the appendix (§B.1). These should be reconciled for consistency.
  5. Figure 3b: The caption contains a typo: '{H^(l)}_{l⊆L}' should probably be '{H^(l)}_{l∈L}'.
  6. §4.6: The paper states that the LoRA is trained on the 'curated triplet dataset' (§A.2) while the Q-Former is trained on OmniEdit-Filtered-1.2M (§B.1). The difference in training data between the proxy and the LoRA-augmented DiT should be clarified to ensure the comparison is fair.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and constructive reading of our manuscript. The four major comments raise legitimate concerns about (1) the expressiveness of the Q-Former proxy relative to standard linear probes, (2) the confounding effect of LoRA fine-tuning in the downstream validation, (3) the partial circularity introduced by using the same VLM for ground-truth generation and hidden-state analysis, and (4) the size and synthetic nature of the evaluation set. We address each below and describe concrete revisions we will make.

read point-by-point responses

  1. Referee: §4.2–§4.4: The Q-Former is substantially more expressive than linear/shallow probes standard in the probing literature. Its cross-attention may decode bounding-box coordinates from intermediate layers due to geometric alignment with its own key-value projection space, rather than because those layers contain more localization information in an architecture-independent sense. The paper never addresses this.

    Authors: The referee raises a valid concern. We agree that the Q-Former's expressiveness is a double-edged sword: it enables clean, interpretable cross-attention analysis (our Figures 5–7), but it also means the proxy could in principle exploit architecture-specific geometric affinities rather than reflecting a layer's intrinsic decodability in an architecture-independent sense. We will address this in two ways in the revision. First, we will add an experiment using a linear probe (a single linear layer mapping pooled hidden states to bounding-box coordinates) trained on the same task for each VLM layer. If the relative layer-wise pattern persists under linear probing—intermediate layers converging faster than the final layer—this would demonstrate that the finding is not an artifact of the Q-Former's cross-attention geometry. Second, we will add an explicit discussion in §4.2 acknowledging the expressiveness concern, noting the trade-off between interpretability and probe capacity, and explaining why the Q-Former's attention patterns provide additional qualitative evidence (the semantic correspondence of dominant tokens in Figure 7b, the VLM internal attention grounding in Figure 7a) that supports the layer-wise finding beyond what convergence curves alone show. We note that the VLM-internal attention analysis in §4.5 is independent of the Q-Former's projection geometry: it examines the VLM's own attention maps at the proxy-identified layer, and the spatial grounding observed there is a property of the VLM, not the proxy. revision: partial

  2. Referee: §4.6, Table 1: The downstream validation cannot cleanly isolate the proxy's contribution. 'Ours' adds both the proxy-predicted bounding box AND a LoRA fine-tuning module, while the baseline does not have this LoRA. A control where the LoRA is trained with ground-truth bounding boxes would help disentangle these contributions.

    Authors: This is a fair and important point. The current experimental design does not cleanly separate the contribution of the proxy-predicted bounding box from the additional capacity introduced by the LoRA adapter. We will add the requested control experiment: a variant where the LoRA is trained with ground-truth bounding boxes overlaid on the source images (rather than proxy-predicted boxes). This will establish an upper bound for the spatial-cue contribution, isolating it from the proxy's prediction quality. Additionally, we will include a second control: the LoRA trained with no bounding box overlay at all (LoRA-only, no spatial cue), which will measure the contribution of the additional LoRA capacity itself. Together, these two controls will allow us to decompose the improvement into (a) the effect of additional fine-tuning capacity and (b) the effect of the explicit spatial cue. We expect the ground-truth-box variant to perform at least as well as our method, and the LoRA-only variant to perform closer to the baseline, which would confirm that the spatial cue—not the LoRA capacity—is the primary driver of the improvement. We will update Table 1 and the surrounding discussion accordingly. revision: yes

  3. Referee: §A.1: The ground-truth bounding boxes used for proxy training are generated by Qwen2.5-VL-7B-Instruct, the same model whose hidden states are being analyzed. This creates a partial circularity: the proxy is trained to recover what the VLM can produce autoregressively, and the paper then shows the proxy can recover it from single-pass hidden states. The paper should discuss whether the proxy is discovering general localization signals or merely learning to shortcut the VLM's own autoregressive decoding path.

    Authors: We agree that this circularity should be explicitly discussed, and we will add a paragraph in §A.1 (and a note in §4.3) addressing it. The key distinction we will draw is between the information available in a single forward pass and the information that emerges through autoregressive generation. The VLM's autoregressive decoding path involves iterative token-by-token generation, where each step conditions on previously generated tokens. The hidden states produced during a single forward pass—before any autoregressive generation—do not have access to this iterative refinement. Therefore, if the proxy successfully recovers bounding boxes from single-pass hidden states, it is not merely shortcutting the autoregressive path (which produces different, sequentially-conditioned hidden states), but rather accessing spatial information that is already present in the model's intermediate representations before generation begins. That said, we acknowledge that the ground-truth labels reflect what the VLM 'knows' in its autoregressive mode, and the proxy is trained to recover this specific output. To partially address this, we note that the Q-Former is trained on a large-scale external dataset (OmniEdit-Filtered, 128K samples, §B.1) with bounding boxes generated by the same VLM, so the proxy learns a general mapping from hidden states to spatial coordinates rather than memorizing a fixed set of examples. We will also note that using an independent model for ground-truth generation (e.g., a different VLM or human annotators) would further strengthen the claim and is a natural direction for future work, though the choice of Qwen2.5-VL for ground-truth generation was motivated by the need for consistency with the editing pipeline's backbone. revision: yes

  4. Referee: §4.1: The evaluation set consists of only 200 samples, curated by the authors. The small size and synthetic nature raise concerns about generalizability. The 31.5% localization gap and downstream improvements would be more convincing on a larger, more diverse, or non-synthetic benchmark.

    Authors: The referee's concern about evaluation set size and synthetic provenance is reasonable. We will address this in two ways. First, we will expand the evaluation set. We are currently constructing additional evaluation samples using real (non-synthetic) source images sourced from existing multi-object benchmarks, paired with spatially-grounded edit instructions. We aim to at least double the evaluation set to 400+ samples and include a non-synthetic subset. We will report results on both the original synthetic set and the expanded set to show consistency. Second, we will add a more thorough discussion in §A.2 explaining why existing datasets were insufficient—specifically, the rarity of scenes containing multiple semantically similar objects distinguishable only by spatial position, which is the precise failure mode our work targets. We acknowledge that the synthetic nature of the current set is a limitation, and we will state this explicitly. Regarding the 31.5% localization gap (§4.1): this gap is measured between the VLM's autoregressive bounding-box accuracy and the end-to-end pipeline's localization accuracy, both evaluated on the same 200 samples. While a larger set would increase confidence in the exact magnitude, the gap is large enough (31.5 percentage points) that it is unlikely to be an artifact of sample size alone. We will report confidence intervals or standard errors for this measurement in the revision. revision: yes

Circularity Check

0 steps flagged

No significant circularity: the proxy is trained on VLM-generated labels and used to probe VLM hidden states, but the central finding (intermediate layers outperform final layers) is not forced by this construction.

full rationale

The paper trains a Q-Former proxy on bounding-box labels generated by Qwen2.5-VL-7B-Instruct (the same model whose hidden states are probed). This creates a partial dependency: the proxy learns to recover what the VLM can produce autoregressively, and then the paper shows the proxy can recover it from single-pass hidden states. However, this is not circular in the sense the framework targets. The proxy's training labels (autoregressive bounding-box outputs) are distinct from the analytical claim (that localization signals peak in intermediate layers, not the final layer). The labels do not encode which layer to attend to; the layer-preference pattern emerges from training dynamics on the hidden states. The downstream validation (§4.6) compares against external baselines (FLUX.2, FIBO, Kontext) and ablations (Full Description, Norm-AVG), providing independent benchmarks. The concern that the Q-Former's architecture might bias layer selection is a validity threat (correctness risk), not a circularity: the paper does not define its conclusion in terms of its inputs. No equation reduces to its inputs by construction, and no self-citation chain forces the result. Score 1 reflects the mild dependency between label source and probe target, which is acknowledged but does not undermine the independent content of the findings.

Axiom & Free-Parameter Ledger

6 free parameters · 4 axioms · 2 invented entities

The axiom ledger reveals that the paper's central analytical claims rest on the assumption that the Q-Former proxy faithfully reflects VLM internal structure (unvalidated), that VLM-generated bounding boxes are accurate ground truth (partially validated with human eval on 200 samples), and that Gemini VQA is a reliable judge (unvalidated). The free parameters are mostly standard choices, but the layer subset L={15,...,24} is selected post-hoc from experimental results.

free parameters (6)
  • λ (GIoU loss weight) = 0.4
    Chosen to balance L1 and GIoU loss components; stated in §B.1 as a fixed weighting factor after warmup.
  • Q-Former number of learnable queries N = Not explicitly stated
    Architecture hyperparameter; figure shows 4 queries but text does not confirm the exact number used in experiments.
  • Q-Former number of transformer layers = 6
    Shown in Figure 3a as 6 layers; not justified or swept.
  • LoRA rank r = 16
    Standard choice; scaling factor α=16, stated in §B.2.
  • Layer subset L for specialized proxy = {15,...,24}
    Middle layers selected based on convergence observations from per-layer experiments; chosen post-hoc after observing Fig. 4 results.
  • Attention sparsity threshold T = Not explicitly stated
    Used in Figure 5 to define 'decisive' heads (|r1-r2| > T) but the value is not specified in the main text.
axioms (4)
  • domain assumption The Q-Former proxy's attention patterns faithfully reflect where localization information resides in the VLM, rather than reflecting the proxy's own architectural inductive biases.
    Invoked implicitly throughout §4.3-§4.5 when proxy attention patterns are interpreted as VLM internal structure. No control experiment validates this.
  • domain assumption Bounding boxes predicted by Qwen2.5-VL-7B-Instruct via autoregressive generation are accurate ground truth for localization.
    Used in §A.1 for ground truth generation and in §4.1 for the 89% accuracy baseline. Human evaluation is used but on only 200 samples.
  • domain assumption Gemini 2.5 Pro VQA scores are a reliable measure of edit quality for the downstream evaluation.
    Used in §4.6 for the main quantitative comparison (Table 1). No human validation of the VQA judge is reported.
  • domain assumption The 200-sample curated evaluation set is representative of real-world multi-entity editing scenarios.
    The dataset is synthetically generated (§A.2) with a specific anomaly-among-uniform-objects structure that may not generalize to natural images.
invented entities (2)
  • Analysis-by-Proxy framework independent evidence
    purpose: A diagnostic framework using a lightweight proxy model to probe VLM hidden states for task-relevant information in non-generative settings.
    The framework is validated through downstream task improvement (Table 1, Figure 8) and produces falsifiable predictions about which layers contain localization signals.
  • 'Decisive' attention heads independent evidence
    purpose: Q-Former heads that route attention to a single dominant VLM layer per input, enabling input-dependent layer selection.
    These are identified empirically and their selectivity is quantified (Fig. 6a, mean drop of 0.6). The concept is falsifiable: if no heads showed this pattern, the claim would fail.

pith-pipeline@v1.1.0-glm · 21084 in / 3375 out tokens · 274493 ms · 2026-07-08T05:32:25.726875+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages · 21 internal anchors

  1. [1]

    Bai, S. et al. Qwen2.5-vl technical report.arXiv preprint arXiv:2502.13923, 2025a. Bai, S. et al. Qwen3-vl technical report.arXiv preprint arXiv:2511.21631, 2025b. Belinkov, Y . Probing classifiers: Promises, shortcomings, and advances.Computational Linguistics, 48(1):207–250,

    work page internal anchor Pith review Pith/arXiv arXiv

  2. [2]

    Dar, G., Geva, M., Gupta, A., and Berant, J

    URL https: //arxiv.org/abs/2412.14133. Dar, G., Geva, M., Gupta, A., and Berant, J. Analyzing transformers in embedding space,

    work page arXiv

  3. [3]

    URL https: //arxiv.org/abs/2209.02535. Esser, P., Kulal, S., Blattmann, A., Entezari, R., Müller, J., Saini, H., Levi, Y ., Lorenz, D., Sauer, A., Boesel, F., Podell, D., Dockhorn, T., English, Z., Lacey, K., Goodwin, A., Marek, Y ., and Rombach, R. Scaling recti- fied flow transformers for high-resolution image synthe- sis,

    work page internal anchor Pith review Pith/arXiv arXiv

  4. [4]

    URL https://arxiv.org/abs/2403. 03206. Gemini Team, Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.arXiv preprint arXiv:2507.06261,

    work page internal anchor Pith review Pith/arXiv arXiv

  5. [5]

    Transformer Feed-Forward Layers Are Key-Value Memories

    URL https://arxiv.org/abs/2012.14913. Geva, M., Caciularu, A., Wang, K. R., and Goldberg, Y . Transformer feed-forward layers build predictions by pro- moting concepts in the vocabulary space,

    work page internal anchor Pith review Pith/arXiv arXiv 2012

  6. [6]

    Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space

    URL https://arxiv.org/abs/2203.14680. Gutflaish, E., Kachlon, E., Zisman, H., Hacham, T., Sarid, N., Visheratin, A., Huberman, S., Davidi, G., Bukchin, G., Goldberg, K., and Mokady, R. Generating an image from 1,000 words: Enhancing text-to-image with structured captions,

    work page internal anchor Pith review Pith/arXiv arXiv

  7. [7]

    URL https://arxiv.org/abs/ 2511.06876. HaCohen, Y ., Brazowski, B., Chiprut, N., Bitterman, Y ., Kvochko, A., Berkowitz, A., Shalem, D., Lifschitz, D., Moshe, D., Porat, E., Richardson, E., Shiran, G., Chachy, I., Chetboun, J., Finkelson, M., Kupchick, M., Zabari, N., Guetta, N., Kotler, N., Bibi, O., Gordon, O., Panet, P., Benita, R., Armon, S., Kulikov,...

    work page arXiv

  8. [8]

    LTX-2: Efficient Joint Audio-Visual Foundation Model

    URL https: //arxiv.org/abs/2601.03233. Hertz, A., Mokady, R., Tenenbaum, J., Aberman, K., Pritch, Y ., and Cohen-Or, D. Prompt-to-prompt im- age editing with cross attention control,

    work page internal anchor Pith review Pith/arXiv arXiv

  9. [9]

    URL https://arxiv.org/abs/2208.01626. Hu, E. J., Shen, Y ., Wallis, P., Allen-Zhu, Z., Li, Y ., Wang, S., Wang, L., and Chen, W. Lora: Low-rank adaptation of large language models,

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    LoRA: Low-Rank Adaptation of Large Language Models

    URL https://arxiv. org/abs/2106.09685. Jiang, N., Kachinthaya, A., Petryk, S., and Gandelsman, Y . Interpreting and editing vision-language representa- tions to mitigate hallucinations,

    work page internal anchor Pith review Pith/arXiv arXiv

  11. [11]

    Interpreting and Editing Vision-Language Representations to Mitigate Hallucinations

    URL https: //arxiv.org/abs/2410.02762. Kaduri, O., Bagon, S., and Dekel, T. What’s in the image? a deep-dive into the vision of vision language models,

    work page internal anchor Pith review Pith/arXiv arXiv

  12. [12]

    URLhttps://arxiv.org/abs/2411.17491. Labs, B. F. FLUX.2: Frontier Visual Intelligence. https: //bfl.ai/blog/flux-2,

    work page internal anchor Pith review Pith/arXiv arXiv

  13. [13]

    FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Space

    URLhttps://arxiv.org/abs/2506.15742. Li, J., Li, D., Savarese, S., and Hoi, S. Blip-2: Boot- strapping language-image pre-training with frozen im- age encoders and large language models,

    work page internal anchor Pith review Pith/arXiv arXiv

  14. [14]

    BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

    URL https://arxiv.org/abs/2301.12597. Li, Z., Wu, X., Du, H., Liu, F., Nghiem, H., and Shi, G. A survey of state of the art large vision language models: Alignment, benchmark, evaluations and challenges,

    work page internal anchor Pith review Pith/arXiv arXiv

  15. [15]

    A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges

    URLhttps://arxiv.org/abs/2501.02189. Lin, Z., Pathak, D., Li, B., Li, J., Xia, X., Neubig, G., Zhang, P., and Ramanan, D. Evaluating text-to-visual generation with image-to-text generation,

    work page internal anchor Pith review Pith/arXiv arXiv

  16. [16]

    Evaluating Text-to-Visual Generation with Image-to-Text Generation

    URL https://arxiv.org/abs/2404.01291. Liu, B., Akhgari, E., Visheratin, A., Kamko, A., Xu, L., Shrirao, S., Lambert, C., Souza, J., Doshi, S., and Li, D. Playground v3: Improving text-to-image alignment 11 Analysis-by-Proxy: Localization Signals in VLM Condition Encoders with deep-fusion large language models,

    work page internal anchor Pith review Pith/arXiv arXiv

  17. [17]

    Playground v3: Improving Text-to-Image Alignment with Deep-Fusion Large Language Models

    URL https://arxiv.org/abs/2409.10695. Liu, H., Li, C., Li, Y ., and Lee, Y . J. Improved base- lines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023a. Liu, H., Li, C., Wu, Q., and Lee, Y . J. Visual instruction tuning. InNeurIPS, 2023b. Liu, Z., Chen, Z., Liu, H., Luo, C., Tang, X., Wang, S., Zeng, J., Dai, Z., Shi, Z., Wei, T., Dumou...

    work page internal anchor Pith review Pith/arXiv arXiv

  18. [18]

    Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E

    URLhttps://arxiv.org/abs/2510.17771. Merullo, J., Castricato, L., Eickhoff, C., and Pavlick, E. Linearly mapping from image to text space,

    work page arXiv

  19. [19]

    Linearly Mapping from Image to Text Space

    URL https://arxiv.org/abs/2209.15162. Neo, C., Ong, L., Torr, P., Geva, M., Krueger, D., and Barez, F. Towards interpreting visual information processing in vision-language models,

    work page internal anchor Pith review Pith/arXiv arXiv

  20. [20]

    Towards Interpreting Visual Information Processing in Vision-Language Models

    URL https://arxiv. org/abs/2410.07149. Nikankin, Y ., Arad, D., Gandelsman, Y ., and Belinkov, Y . Same task, different circuits: Disentangling modality- specific mechanisms in vlms,

    work page internal anchor Pith review Pith/arXiv arXiv

  21. [21]

    nostalgebraist

    URL https:// arxiv.org/abs/2506.09047. nostalgebraist. Interpreting GPT: the logit lens. https://www.lesswrong. com/posts/AcKRB8wDpdaN6v6ru/ interpreting-gpt-the-logit-lens , Au- gust

    work page arXiv

  22. [22]

    Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S

    Accessed: 2025-02-22. Rezatofighi, H., Tsoi, N., Gwak, J., Sadeghian, A., Reid, I., and Savarese, S. Generalized intersection over union: A metric and a loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 658–666,

    work page 2025

  23. [23]

    URL https://arxiv.org/ abs/2112.10752. Team, M. A. Pixtral 12b.arXiv preprint arXiv:2410.07073,

    work page internal anchor Pith review Pith/arXiv arXiv

  24. [24]

    URL https:// arxiv.org/abs/2106.13884. Wang, A. Z., Ge, S., Karras, T., Liu, M.-Y ., and Balaji, Y . A comprehensive study of decoder-only llms for text-to- image generation. InProceedings of the Computer Vision and Pattern Recognition Conference, pp. 28575–28585, 2025a. Wang, W., Gao, Z., Gu, L., Pu, H., Cui, L., Wei, X., Liu, Z., Jing, L., Ye, S., Shao,...

    work page internal anchor Pith review Pith/arXiv arXiv

  25. [25]

    OmniEdit: Building Image Editing Generalist Models Through Specialist Supervision

    URL https:// arxiv.org/abs/2411.07199. Wu, C., Li, J., Zhou, J., Lin, J., Gao, K., Yan, K., Yin, S.-m., Bai, S., Xu, X., Chen, Y ., et al. Qwen-image technical report.arXiv preprint arXiv:2508.02324,

    work page internal anchor Pith review Pith/arXiv arXiv

  26. [26]

    URLhttps://arxiv.org/ abs/1801.03924. 12 Analysis-by-Proxy: Localization Signals in VLM Condition Encoders In this supplementary material, we provide additional details on the datasets (§A), implementation (§B), and further experimental results (§C). A. Datasets Details A.1. Ground Truth Bounding Box Extraction The experiment in Section 4.1 of the main pa...

    work page internal anchor Pith review Pith/arXiv arXiv

  27. [27]

    We choose FIBO due to its strong adherence to spatial layouts

    with its standard runtime configuration. We choose FIBO due to its strong adherence to spatial layouts. To generate the corresponding paired images, we use Qwen-Image-Edit-2509, guided by the edit prompts, which instruct the model to modify the anomalous object so that it matches the rest of the scene. This editing step, which harmonizes a single outlier ...

    work page 2025

  28. [28]

    To stabilize the initial training phase, we apply a warmup period of 100 steps during which the model is optimized using only the L1 loss

    loss. To stabilize the initial training phase, we apply a warmup period of 100 steps during which the model is optimized using only the L1 loss. After this warmup, we introduce the GIoU loss with a fixed weighting factor λ= 0.4 , resulting in the following combined loss function: Ltotal = (1−λ)L L1 +λL GIoU The model is trained for a single epoch using th...

    work page 2000

  29. [29]

    Recolour the bottom right clownfish to be black and white

    to consume the bounding- box overlay produced by the Q-Former at inference time. 14 Analysis-by-Proxy: Localization Signals in VLM Condition Encoders Training data.The LoRA is trained on the curated triplet dataset described in Section A.2 (source image, edit instruction, edited image), paired with the human-verified ground-truth bounding boxes obtained a...

    work page arXiv 2050

  30. [30]

    All automated Vision Question Answering evaluations and prompt generations utilizing the Gemini 2.5 Pro API (Gemini Team, Google,

    , FIBO-Edit (Gutflaish et al., 2025), and the LPIPS metric (Zhang et al., 2018). All automated Vision Question Answering evaluations and prompt generations utilizing the Gemini 2.5 Pro API (Gemini Team, Google,

    work page 2025