Pith. sign in

REVIEW 3 major objections 5 minor 37 references

The paper claims that cross-modal fusion in VLMs is carried by local low-rank neighborhoods, not global linear directions, and that these neighborhoods can be steered and used for retrieval.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Local low-rank Gaussian neighborhoods in VLM residual streams reveal model-specific fusion trajectories and serve as causal steering and retrieval units.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection Solid empirical extension of local-geometric MFA to VLMs with interesting steering/retrieval results, but the headline fusion-trajectory claim rests on unweighted component counts and only three layers per model. the 3 major comments →

arxiv 2608.00561 v1 pith:7CYGY5AD submitted 2026-08-01 cs.AI cs.CLcs.CV

Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations

Shalom Kachko , Raz Lapid , Margarita Vald , Almog Dubin , Moshe Sipper This is my paper
classification cs.AI cs.CLcs.CV
keywords vision-language modelsmixture of factor analyzerslocal geometryinterpretabilityactivation steeringcross-modal fusionresidual streamretrieval
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LENS, a decomposition of vision-language model activations into local low-rank Gaussian neighborhoods using a Mixture of Factor Analyzers. It claims that cross-modal fusion happens in these neighborhoods, which global linear directions miss because the representation space is high-dimensional globally but low-dimensional locally. On two open VLMs, the method yields two distinct depth-dependent fusion signatures: LLaVA mixes visual and text neighborhoods progressively toward later layers, while Qwen3-VL mixes early, partially re-segregates, and recombines near the output. The neighborhoods are also functional: interpolating activations toward their centroids steers generation within and across modalities, outperforming difference-in-means and a vision-language sparse autoencoder in most early- and middle-layer conditions, and their coefficient space lifts cross-modal retrieval from near-random to 48.6% R@1 in the deepest evaluated Qwen3-VL setting. A reader should care because this reframes VLM interpretability from finding global directions to locating local neighborhoods, and it gives a concrete handle for controlling what these models say.

Core claim

LENS's central discovery is that the fusion of visual and textual information in a VLM can be read off the local geometry of its residual stream. Training an MFA on token-level activations produces K Gaussian neighborhoods, and a neighborhood is 'mixed' when hard-assigned activations contain substantial shares of both image patches and text tokens. Counting mixed neighborhoods by depth reveals architecture-dependent trajectories: LLaVA-1.5 moves from 0 to 55 to 472 mixed components across layers 8, 16, and 29, a progressive late-fusion pattern, while Qwen3-VL moves from 464 to 61 to 128 across layers 9, 18, and 27, an early-fusion, partial-segregation, late-recombination pattern. The same ne

What carries the argument

The load-bearing object is the Mixture of Factor Analyzers (MFA): a probabilistic model in which each activation x is generated by one of K Gaussian neighborhoods with centroid μ_k, low-rank loading matrix W_k, local latent z_k, and diagonal noise Ψ. Its components partition activation space through responsibilities R_k(x) and locate samples through local coordinates ẑ_k(x). LENS uses this machinery to define 'mixed' components, to label neighborhoods by feeding top-activating image patches and text contexts to a judge VLM, to steer generations by interpolating activations toward μ_k, and to build a shared cross-modal coefficient space b(x) for retrieval.

Load-bearing premise

The depth-trajectory comparison rests on activations from only three layers per model; if those layers are not representative of each model's depth behavior, the claimed architectural signatures could be artifacts of layer choice.

What would settle it

Fit the local-Gaussian decomposition at every layer (not just three) for LLaVA-1.5 and Qwen3-VL, and plot the share of mixed vision-text neighborhoods by depth. If LLaVA's share does not rise monotonically or Qwen3-VL's early peak disappears, the claimed fusion trajectories are falsified.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Cross-modal fusion is not a single global separation but a depth-dependent local phenomenon: mixed neighborhoods can accumulate monotonically (LLaVA) or non-monotonically (Qwen3-VL), so architecture design shapes fusion geometry.
  • Local neighborhoods are causal handles: interpolating activations toward a centroid redirects generation both within a modality and across modalities, so a concept localized in one modality can be elicited by intervening on the other.
  • MFA coordinate space is a training-free shared representation: cosine similarity in responsibility or coefficient space retrieves cross-modal correspondences that raw activations miss, including Image-Written retrieval rising from 14.9% to 48.6% R@1.
  • Steering effectiveness changes with depth: early layers favor vision-dominated components, middle layers reverse the pattern, and late-layer steering is brittle—so interventions should be layer-aware.
  • Steering quality can be assessed automatically: human ratings correlate with the VLM judge scores (Spearman 0.727), supporting scaled-up evaluation of this kind of intervention.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the geometry claim generalizes, dense layer sweeps on other VLM architectures would likely reveal transition layers where mixed-neighborhood share jumps, making those layers the natural targets for concept editing or jailbreak mitigation.
  • The retrieval result suggests a lightweight, contrastive-free alignment method: any two modalities can be compared in MFA coefficient space, which could extend to zero-shot classification or image-text matching in models without explicit alignment training.
  • Late-layer steering brittleness implies that output-side geometry is less causally malleable; a testable extension is to combine MFA steering at early layers with vocabulary-side constraints at the final projection.
  • The mixed-neighborhood count is a measurable architectural signature; one could use it as a diagnostic to compare fusion across model families, check whether fine-tuning shifts fusion depth, or predict where cross-modal interference will occur.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces LENS, an MFA-based decomposition of token-level residual-stream activations in two VLMs (LLaVA-1.5-7B and Qwen3-VL-8B). It claims that local Gaussian neighborhoods with low-rank structure reveal architecture-dependent cross-modal fusion trajectories (progressive late mixing in LLaVA; early non-monotonic mixing in Qwen3-VL), that steering activations toward MFA centroids outperforms DiffMeans and VL-SAE in most early- and middle-layer conditions, that a 80-output human evaluation supports the automated steering scores, and that the MFA coefficient space improves cross-modal retrieval (e.g., Qwen3-VL Image-Written R@1 from 14.9% to 48.6% at layer 27). The paper includes ablations over K, q, and τ, plus a limitation section acknowledging small human evaluation and brittle late-layer steering.

Significance. If the claims hold, LENS offers a genuinely different unit of analysis from global linear directions and SAE features, with potential value for interpreting and controlling multimodal representations. The paper's strengths include: a well-specified MFA formulation; automated labeling with an explicit protocol; ablations over principal hyperparameters; bootstrap confidence intervals in retrieval; and an independent, if small, human evaluation. The comparison against DiffMeans and VL-SAE is a useful benchmark. However, the central qualitative claim about architecture-dependent fusion trajectories is based on unweighted component counts at only three layers per model, and the steering comparison relies on judge-based component selection and best-alpha oracle reporting. These issues are load-bearing for the headline conclusions, so the current evidence is not yet sufficient for acceptance.

major comments (3)
  1. [Section 4.2, Fig. 3] The architecture-dependent fusion trajectory is measured solely by the number/share of MFA components whose hard-assigned activations are mixed. This is an unweighted component count: with K=8,192 (Section 4.1), a component containing a handful of boundary activations contributes as much as a component covering millions of tokens. The Qwen3-VL non-monotonic pattern (464 → 61 → 128 across layers 9/18/27) could be driven by changes in component-size distribution or by tiny mixed components at layer 9 rather than by genuine early fusion followed by re-segregation. No responsibility-weighted or activation-mass-weighted mixed share is reported. Additionally, only three layers per model are analyzed; the dense layer sweep in Appendix B.1 covers LLaVA only and sweeps K/q, not layer density. Please add mass-weighted mixed shares, component-size distributions, and a denser layer sweep (or justify
  2. [Section 4.3, Appendix A.3] The central steering comparison depends on component selection by an LLM judge and on best-α oracle reporting. The 100 MFA components are selected using MFA labels and MFA-derived modality distributions (text_ratio/image_ratio ≥ 0.8). For VL-SAE, the paper says a 'structurally identical' prompt was used, but it is not specified how text/vision dominance is computed for VL-SAE features or whether the selection is equally favorable. The reported scores are each component's best over a swept α, which is an oracle metric that can overstate practical steering and may favor methods with wider/less constrained ranges. The 80-output human evaluation (Appendix A.4) is too small to validate per-condition MFA-vs-baseline differences, despite the moderate overall correlation (ρ=0.727). Please report results at fixed α, describe the VL-SAE selection criteria precisely, and provide a larger human eval
  3. [Abstract, Section 4.3 vs Appendix B.3] The abstract and Section 5 state that MFA steering 'outperforms DiffMeans and VL-SAE in most evaluated conditions,' but Appendix B.3 (Fig. 9) shows that at late layers DiffMeans often outperforms MFA in text-targeted scopes, and vision-only steering largely fails for both models. The paper acknowledges late-layer brittleness only in the Limitations paragraph. The claim should be qualified to early- and middle-layer, vision-targeted conditions, or the abstract and summary should be revised to avoid overclaiming.
minor comments (5)
  1. [Appendix A.1] The text mentions an 'anonymized repository and supplementary code archive' but provides no URL or link. For reproducibility, please include the actual repository/data link or state clearly that code will be released upon acceptance.
  2. [Fig. 3] The UMAP centroid plots are visually appealing but not quantitative. Consider adding a quantitative measure of inter-centroid separation or a complementary mass-weighted visualization to support the trajectory claim.
  3. [Appendix B.3] Typo: 'rendereing' should be 'rendering'.
  4. [Section 4.3 / Fig. 4] Prompting is a non-interventional reference but is displayed alongside intervention methods in boxplots. This is fine if clearly labeled, but the caption should state that prompting is not directly comparable as an intervention.
  5. [Appendix A.4] The human evaluation instructions label the first rating as 'Correlation,' but the text and scoring define it as concept alignment. Please use consistent terminology.

Circularity Check

0 steps flagged

No significant circularity: the reported decompositions, steering, and retrieval results are empirical and independently evaluated.

full rationale

The paper's central claims are empirical measurements and held-out evaluations, not formal derivations that reduce to their inputs. The MFA is fit to residual-stream activations, and the mixed-component counts in Section 4.2 are descriptive summaries of that fitted decomposition; they are not predictions obtained from the same fitted quantity. The steering results in Section 4.3 are evaluated on generated outputs using both a VLM judge and an independent human evaluation on 80 outputs (Appendix A.4), so the claimed advantage over DiffMeans and VL-SAE is not forced by construction. The retrieval experiments in Section 4.4 use ImageNet, held out from the CC3M activations used to train the MFA, and compare raw activations, responsibilities, and coefficients; no contrastive or task-specific objective is fit to the retrieval metric. Hyperparameter ablations in Appendices B.1 and B.2 test robustness to K, q, and tau. There are no load-bearing self-citations: the cited MFA initialization (Shafran et al.), baselines (VL-SAE, DiffMeans), and architectural prior work are by other authors, and the paper invokes no uniqueness theorem from its own authors. The limitations noted in the paper—three sampled layers per model, mass-unweighted component counts, and model-based labels/scores—are correctness and robustness concerns rather than circularity, because none of the central claims reduces by equation or by definition to its own inputs.

Axiom & Free-Parameter Ledger

4 free parameters · 7 axioms · 0 invented entities

The central claims rest on the MFA generative assumption, the judge-based labeling/scoring/selection loop, and the representativeness of three sampled layers. These are modeling and evaluation choices rather than derivation steps; the human eval and retrieval results provide partial independent support.

free parameters (4)
  • K = 8192 (ablated: 1024, 4096, 16384)
    Number of MFA Gaussian components at each layer; chosen by hand, not fit, but central trajectory and steering are shown to be stable across it in Appendix B.1.
  • q = 10 (ablated: 5, 20; additional late-layer sweep to 35)
    Local latent rank per MFA component; chosen by hand. Steering at late layers is sensitive to q (Appendix B.3).
  • tau = 0.80 (ablated: 0.70 to 0.90)
    Modality-purity threshold defines mixed vs pure components; trajectory shape is stable across sweep, but absolute mixed counts scale with tau.
  • hmax = 10
    Per-pair sampling budget per modality; controls modality balance in activation corpus and could affect mixed-component estimates. Not ablated.
axioms (7)
  • domain assumption Residual-stream activations at a given layer are well approximated by a mixture of low-rank Gaussian neighborhoods (Eq. 1-2).
    Section 3.1; the MFA generative model is assumed true for VLM hidden states; no goodness-of-fit test is reported.
  • domain assumption Clustering in a 256-d random orthonormal projection preserves the local cluster structure needed for centroid initialization.
    Section 3.2, Phase 2; if projection collapses local structure, fitted centroids could be mis-initialized.
  • domain assumption Maximum-responsibility hard assignment plus purity threshold tau=0.8 yields a meaningful partition into vision/text/mixed components.
    Section 3.2 Stage 3 and Section 4.2; the fusion trajectory counts depend on this classification rule.
  • domain assumption The judge VLMs (Qwen2.5-VL-7B for labels and Qwen2.5-32B for steering scores) provide valid semantic descriptions and alignment/fluency scores.
    Section 4.3 and Appendix A.3; human correlation (rho=0.727) supports but is limited to 80 outputs.
  • ad hoc to paper The LLM-judge protocol for selecting 50 text-dominant and 50 vision-dominant components does not unfairly favor MFA over baselines.
    Appendix A.3; the selection criteria ('concrete', 'testable') could bias toward concepts a VLM judge can recognize and score, though a structurally identical prompt is used for VL-SAE.
  • domain assumption The three layers per model (8/16/29 for LLaVA, 9/18/27 for Qwen3) are representative of each model's depth-dependent fusion behavior.
    Section 4.1; trajectory claims are based on only three layers per model; no denser layer sweep is reported.
  • domain assumption The VLM residual stream is a shared space in which image-patch and text-token activations can be compared geometrically.
    Section 1; this is the premise of the modality-gap literature and of LENS.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations." pith.science (2026). https://pith.science/paper/7CYGY5AD

@misc{pith2026260800561,
  author       = {Pith},
  title        = {Pith review of: Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7CYGY5AD}},
  note         = {Machine review of arXiv:2608.00561}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Vision-language models (VLMs) process image patches and text tokens in a shared residual stream, but the local geometry through which the two modalities interact remains poorly understood. Most interpretability methods identify global linear directions, which may miss representations that are globally high-dimensional but locally low-dimensional. We introduce LENS (Local Explanation of Neighborhood Subspaces), a method that decomposes VLM activations into local low-rank Gaussian neighborhoods using a Mixture of Factor Analyzers. Applied to LLaVA-1.5-7B and Qwen3-VL-8B, LENS reveals distinct depth-dependent fusion trajectories consistent with each model's fusion mechanism: LLaVA progressively mixes modalities at later layers, whereas Qwen3-VL mixes them early, partially re-segregates them, and recombines them near the output. An automated multimodal labeling pipeline assigns concise semantic descriptions to these neighborhoods. Interpolating activations toward neighborhood centroids causally redirects generation within and across modalities and outperforms difference-in-means and VL-SAE in most evaluated conditions; in one LLaVA vision-to-vision setting, MFA achieves 5.7 times the VL-SAE score. Human evaluation finds MFA steering competitive with prompting and substantially stronger than the other intervention baselines. Finally, the MFA coefficient space improves Qwen3-VL image-to-rendered-text retrieval at the deepest evaluated layer from 14.9% to 48.6% R@1. Ablations show that the reported fusion trajectories are stable across component counts, local ranks, and modality-purity thresholds. These results support local geometric neighborhoods as useful interpretable and causal units for analyzing cross-modal representations in the evaluated VLMs.

Figures

Figures reproduced from arXiv: 2608.00561 by Almog Dubin, Margarita Vald, Moshe Sipper, Raz Lapid, Shalom Kachko.

Figure 1
Figure 1. Figure 1: Activation extraction and labeling pipeline. (1) Image-caption pairs are processed by a VLM to extract and store [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Mixed components bind specific visual and textual content. Two representative mixed components (LLaVA-1.5 layer 29 and Qwen3-VL-8B layer 27), each shown with its top max-activating image patches (left), text-token contexts with the responsible token highlighted (middle), label (right). Both activate on visual patches and text tokens denoting the same concept. Layer 8 Text (2091) Vision (5783) Layer 16 Text… view at source ↗
Figure 3
Figure 3. Figure 3: Cross-modal fusion regimes across architec￾tures. Top (LLaVA-1.5-7B): Increasing late-layer mix￾ing. Mixed centroids accumulate monotonically with depth (0 → 55 → 472 across layers 8, 16, 29).Bottom (Qwen3-VL￾8B): Early, non-monotonic mixing. Mixed centroids follow a non-monotonic trajectory (464 → 61 → 128 across layers 9, 18, 27). (iii) Relative steering performance shifts with depth. At the early layers… view at source ↗
Figure 5
Figure 5. Figure 5: Human evaluation of steering outputs. Harmonic￾mean scores combining concept alignment and fluency, shown overall and separately for each model. Boxes show the in￾terquartile range of the harmonic-mean values, horizontal lines the median, and whiskers extending to 1.5× the interquar￾tile range. MFA has the highest overall median, although prompting slightly leads on LLaVA-1.5 and MFA leads on Qwen3-VL-8B. … view at source ↗
Figure 4
Figure 4. Figure 4: MFA centroids support within- and cross-modal control. Distributions show each component’s best-α har￾monic score across six intervention-scope-to-target-modality conditions at early and middle layers. Boxes show the in￾terquartile range, horizontal lines indicate the median, and whiskers extending to 1.5× the interquartile range. MFA gen￾erally outperforms the computational baselines and transfers concept… view at source ↗
Figure 6
Figure 6. Figure 6: MFA coordinates expose cross-modal alignment. Bidirectional R@1 across layers for LLaVA-1.5 (top) and Qwen3-VL (bottom). We compare raw residual-stream activa￾tions, MFA responsibilities r(x), full MFA coefficients b(x), and standard and aligned VL-SAE representations. Error bars show 95% bootstrap confidence intervals. prompting varies by architecture. Together, these findings establish local geometric ne… view at source ↗
Figure 8
Figure 8. Figure 8: Sensitivity of cross-modal fusion regimes to the modality-purity threshold τ . Mixed-centroid share across layers for (a) LLaVA-1.5 and (b) Qwen3-VL-8B. Although the magnitude changes with τ , the architecture-specific tra￾jectories remain stable; the main-text setting τ = 0.80 is highlighted. B.3 Late-Layer Steering and Rank Sensitivity Following the early and middle layer evaluations (Section 4.3), [PIT… view at source ↗
Figure 9
Figure 9. Figure 9: Late-layer steering performance. Distributions show each component’s best-α harmonic score across six intervention-scope-to-target-modality conditions at late-layer: (a) LLaVA-1.5 Layer 29 across local ranks and (b) Qwen3-VL￾8B Layer 27. Boxes show the interquartile range, horizontal lines indicate the median, and whiskers extending to 1.5x the interquartile range. Steering is generally weaker and less sta… view at source ↗
Figure 11
Figure 11. Figure 11: Representative steering outputs for LLaVA￾1.5-7B. Each intervention condition includes one example from MFA, DiffMeans, and VL-SAE. Rows report the layer, intervention type, steering strength α, harmonic score, target concept, and output [PITH_FULL_IMAGE:figures/full_fig_p013_11.png] view at source ↗
Figure 10
Figure 10. Figure 10: Additional mixed components. Four components from multiple layers, shown with their highest-responsibility image patches, text contexts with the responsible token highlighted, and automated labels. C.2 Representative Steering Outputs [PITH_FULL_IMAGE:figures/full_fig_p013_10.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

37 extracted references · 22 canonical work pages · 2 internal anchors

  1. [1]

    Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

    2022

  2. [2]

    Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Cheng, Z.; Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025. Qwen2.5-VL Technical Report. arXiv:2502.13923

    Pith/arXiv arXiv 2025

  3. [3]

    Bills, S.; Cammarata, N.; Mossing, D.; Tillman, H.; Gao, L.; Goh, G.; Sutskever, I.; Leike, J.; Wu, J.; and Saunders, W. 2023. Language models can explain neurons in language models. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html

    work page 2023

  4. [4]

    E.; Hume, T.; Carter, S.; Henighan, T.; and Olah, C

    Bricken, T.; Templeton, A.; Batson, J.; Chen, B.; Jermyn, A.; Conerly, T.; Turner, N.; Anil, C.; Denison, C.; Askell, A.; Lasenby, R.; Wu, Y.; Kravec, S.; Schiefer, N.; Maxwell, T.; Joseph, N.; Hatfield-Dodds, Z.; Tamkin, A.; Nguyen, K.; McLean, B.; Burke, J. E.; Hume, T.; Carter, S.; Henighan, T.; and Olah, C. 2023. Towards Monosemanticity: Decomposing L...

    2023

  5. [5]

    Chanin, D.; Wilken-Smith, J.; Dulka, T.; Bhatnagar, H.; Golechha, S.; and Bloom, J. 2024. A is for absorption: Studying feature splitting and absorption in sparse autoencoders. arXiv preprint arXiv:2409.14507

    arXiv 2024

  6. [6]

    R.; Ewart, A.; and Sharkey, L

    Cunningham, H.; Huben, R.; Smith, L. R.; Ewart, A.; and Sharkey, L. 2024. Sparse Autoencoders Find Highly Interpretable Features in Language Models. In The Twelfth International Conference on Learning Representations

    work page 2024

  7. [7]

    Elhage, N.; Hume, T.; Olsson, C.; Schiefer, N.; Henighan, T.; Kravec, S.; Hatfield-Dodds, Z.; Lasenby, R.; Drain, D.; Chen, C.; et al. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652

    Pith/arXiv arXiv 2022

  8. [8]

    Fry, H. 2024. Towards Multimodal Interpretability: Learning Sparse Interpretable Features in Vision Transformers. LESSWRONG

    work page 2024

  9. [9]

    Ghahramani, Z.; and Hinton, G. E. 1996. The EM algorithm for mixtures of factor analyzers. Technical Report CRG-TR-96-1, University of Toronto

    work page 1996

  10. [10]

    Hu, Y.; Yuan, J.; Wen, C.; Lu, X.; Liu, Y.; and Li, X. 2025. Rsgpt: A remote sensing vision language model and benchmark. ISPRS Journal of Photogrammetry and Remote Sensing, 224: 272--286

    work page 2025

  11. [11]

    P.; and Ba, J

    Kingma, D. P.; and Ba, J. 2017. Adam: A Method for Stochastic Optimization. arXiv:1412.6980

    Pith/arXiv arXiv 2017

  12. [12]

    Lee, A.; Weber, M.; Vi \'e gas, F.; and Wattenberg, M. 2025. Shared Global and Local Geometry of Language Model Embeddings. In Second Conference on Language Modeling

    work page 2025

  13. [13]

    Li, J.; Li, D.; Savarese, S.; and Hoi, S. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, 19730--19742. PMLR

    work page 2023

  14. [14]

    Liang, W.; Zhang, Y.; Kwon, Y.; Yeung, S.; and Zou, J. 2022. Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. In Oh, A. H.; Agarwal, A.; Belgrave, D.; and Cho, K., eds., Advances in Neural Information Processing Systems

    work page 2022

  15. [15]

    A.; Wang, Z.; Zhou, Y.; Balasubramanian, S.; Zarei, A.; Rezaei, K.; et al

    Lin, Z.; Basu, S.; Beigi, M.; Manjunatha, V.; Rossi, R. A.; Wang, Z.; Zhou, Y.; Balasubramanian, S.; Zarei, A.; Rezaei, K.; et al. 2025. A survey on mechanistic interpretability for multi-modal foundation models. arXiv preprint arXiv:2502.17516

    Pith/arXiv arXiv 2025

  16. [16]

    Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. Advances in neural information processing systems, 36: 34892--34916

    work page 2023

  17. [17]

    McInnes, L.; Healy, J.; and Melville, J. 2020. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv:1802.03426

    Pith/arXiv arXiv 2020

  18. [18]

    Meng, L.; Yang, J.; Tian, R.; Dai, X.; Wu, Z.; Gao, J.; and Jiang, Y.-G. 2024. DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMM s. In The Thirty-eighth Annual Conference on Neural Information Processing Systems

    work page 2024

  19. [19]

    Pach, M.; Karthik, S.; Bouniot, Q.; Belongie, S.; and Akata, Z. 2025. Sparse autoencoders learn monosemantic features in vision-language models. arXiv preprint arXiv:2504.02821

    arXiv 2025

  20. [20]

    Qwen; :; Yang, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Li, C.; Liu, D.; Huang, F.; Wei, H.; Lin, H.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Lin, J.; Dang, K.; Lu, K.; Bao, K.; Yang, K.; Yu, L.; Li, M.; Xue, M.; Zhang, P.; Zhu, Q.; Men, R.; Lin, R.; Li, T.; Tang, T.; Xia, T.; Ren, X.; Ren, X.; Fan, Y.; Su, Y.; Zhang, Y.; Wa...

    Pith/arXiv arXiv 2025

  21. [21]

    W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

    Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PmLR

    2021

  22. [22]

    Rimsky, N.; Gabrieli, N.; Schulz, J.; Tong, M.; Hubinger, E.; and Turner, A. 2024. Steering Llama 2 via Contrastive Activation Addition. In Ku, L.-W.; Martins, A.; and Srikumar, V., eds., Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 15504--15522. Bangkok, Thailand: Association for Computa...

    work page 2024

  23. [23]

    C.; and Fei-Fei, L

    Russakovsky, O.; Deng, J.; Su, H.; Krause, J.; Satheesh, S.; Ma, S.; Huang, Z.; Karpathy, A.; Khosla, A.; Bernstein, M.; Berg, A. C.; and Fei-Fei, L. 2015. ImageNet Large Scale Visual Recognition Challenge . International Journal of Computer Vision (IJCV), 115(3): 211--252

    work page 2015

  24. [24]

    Saglam, B.; Kassianik, P.; Nelson, B.; Weerawardhena, S.; Singer, Y.; and Karbasi, A. 2025. Large Language Models Encode Semantics and Alignment in Linearly Separable Representations. In Inui, K.; Sakti, S.; Wang, H.; Wong, D. F.; Bhattacharyya, P.; Banerjee, B.; Ekbal, A.; Chakraborty, T.; and Singh, D. P., eds., Proceedings of the 14th International Joi...

    work page 2025

  25. [25]

    Shafran, O.; Ronen, S.; Fahn, O.; Ravfogel, S.; Geiger, A.; and Geva, M. 2026. From Directions to Regions: Decomposing Activations in Language Models via Local Geometry. arXiv preprint arXiv:2602.02464

    arXiv 2026

  26. [26]

    Sharma, P.; Ding, N.; Goodman, S.; and Soricut, R. 2018. Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning. In Gurevych, I.; and Miyao, Y., eds., Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2556--2565. Melbourne, Australia: Association for ...

    work page 2018

  27. [27]

    Shen, S.; Sun, J.; Huang, Q.; and Wang, S. 2026. VL - SAE : Interpreting and Enhancing Vision-Language Alignment with a Unified Concept Set. In The Thirty-ninth Annual Conference on Neural Information Processing Systems

    work page 2026

  28. [28]

    L.; McDougall, C.; MacDiarmid, M.; Freeman, C

    Templeton, A.; Conerly, T.; Marcus, J.; Lindsey, J.; Bricken, T.; Chen, B.; Pearce, A.; Citro, C.; Ameisen, E.; Jones, A.; Cunningham, H.; Turner, N. L.; McDougall, C.; MacDiarmid, M.; Freeman, C. D.; Sumers, T. R.; Rees, E.; Batson, J.; Jermyn, A.; Carter, S.; Olah, C.; and Henighan, T. 2024. Scaling Monosemanticity: Extracting Interpretable Features fro...

    2024

  29. [29]

    Thasarathan, H.; Forsyth, J.; Fel, T.; Kowal, M.; and Derpanis, K. G. 2025. Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment. In Forty-second International Conference on Machine Learning

    work page 2025

  30. [30]

    van Sprang, A.; Samson, L.; Lucic, A.; Acar, E.; Ghebreab, S.; and Asano, Y. M. 2026. Same Content, Different Answers: Cross-Modal Inconsistency in MLLMs. arXiv:2512.08923

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  31. [31]

    D.; and Potts, C

    Wu, Z.; Arora, A.; Geiger, A.; Wang, Z.; Huang, J.; Jurafsky, D.; Manning, C. D.; and Potts, C. 2025. AxBench: Steering LLM s? Even Simple Baselines Outperform Sparse Autoencoders. In Forty-second International Conference on Machine Learning

    work page 2025

  32. [32]

    Xiang, J.; Wang, X.; Zhang, X.; Xi, Y.; Eweje, F.; Chen, Y.; Li, Y.; Bergstrom, C.; Gopaulchan, M.; Kim, T.; et al. 2025. A vision--language foundation model for precision oncology. Nature, 638(8051): 769--778

    work page 2025

  33. [33]

    Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; Zheng, C.; Liu, D.; Zhou, F.; Huang, F.; Hu, F.; Ge, H.; Wei, H.; Lin, H.; Tang, J.; Yang, J.; Tu, J.; Zhang, J.; Yang, J.; Yang, J.; Zhou, J.; Zhou, J.; Lin, J.; Dang, K.; Bao, K.; Yang, K.; Yu, L.; Deng, L.; Li, M.; Xue, M.; Li, M.; Zhang, P.; Wang, P.; Zhu, Q...

    Pith/arXiv arXiv 2025

  34. [34]

    Yang, T.; Chang, L.; Yan, J.; Li, J.; Wang, Z.; and Zhang, K. 2025 b . A Survey on Foundation-Model-Based Industrial Defect Detection. arXiv:2502.19106

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  35. [35]

    Zhang, K.; Shen, Y.; Li, B.; and Liu, Z. 2025 a . Large multi-modal models can interpret features in large multi-modal models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 3650--3661

    work page 2025

  36. [36]

    Zhang, Z.; Yadav, S.; Han, F.; and Shutova, E. 2025 b . Cross-modal information flow in multimodal large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 19781--19791

    work page 2025

  37. [37]

    Zhang, Z.; Zhang, A.; Li, M.; Zhao, H.; Karypis, G.; and Smola, A. 2023. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXiv:2302.00923

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.