Pith. sign in

REVIEW 3 major objections 2 minor 15 references

The commitment to one interpretation of a bistable image occurs after the vision tower in vision-language models.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-27 20:14 UTC pith:AQ3OWJ7A

load-bearing objection The paper finds simultaneous vision-tower activation on most bistable stimuli but steering only flips default-dominant regimes, pointing to a downstream bottleneck, though SAE fidelity and regime partitioning need checking. the 3 major comments →

arxiv 2606.08031 v1 pith:AQ3OWJ7A submitted 2026-06-06 cs.CV

Vision-Language Asymmetry in Bistable Image Captioning

Arohan Agate This is my paper
classification cs.CV
keywords bistable imagesvision-language modelssparse autoencodersimage captioningduck-rabbitCLIPLLaVAseeing-as
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tests how vision-language models handle ambiguous images like the duck-rabbit illusion by generating many captions under different prompts. It finds that the vision encoder often represents both possible interpretations at once for most stimuli. However, interventions that steer the vision features can change the caption for some types of stimuli but not for others where both interpretations are balanced. This suggests the point where the model commits to one view is later, in the language processing part. A reader would care because it gives a concrete way to study the difference between detecting features and deciding on a meaning.

Core claim

Across 69 bistable stimuli, 72% show simultaneous activation of both per-aspect feature pools at the vision tower. Causal steering at CLIP layer 22 flips captions on default-dominant stimuli but cannot flip captions on force-balanced young/old at any tested coefficient. The dominance bottleneck lives downstream of the vision tower; the gap between vision-side representation and language-side commitment is an empirical handle on the seeing/seeing-as distinction.

What carries the argument

TopK sparse autoencoder on the CLIP layer consumed by LLaVA-1.6-7B that isolates per-aspect feature pools, used for both representation analysis and causal steering at layer 22.

Load-bearing premise

The TopK SAE faithfully isolates the per-aspect feature pools without introducing selection artifacts, and the behavioral baseline correctly partitions stimuli into default-dominant, force-dominant, and force-balanced regimes.

What would settle it

Finding a coefficient where steering the vision features flips the caption for force-balanced stimuli like young/old would falsify the claim that the bottleneck is downstream.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • 72% of bistable stimuli show simultaneous activation of both aspect pools at the vision tower.
  • Causal steering flips captions on default-dominant stimuli but not on force-balanced ones.
  • The three regimes of stimuli are identified from 3,320 generations under neutral and forced-choice prompts.
  • Steering at vision layer cannot override the dominance in balanced cases despite superposition.
  • Rank-based statistics on SAE outputs require tie-correction to avoid bias.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The location of the bottleneck could be mapped in other multimodal architectures to check generality.
  • Similar probing might distinguish perceptual detection from interpretive commitment in other AI tasks.
  • This could inform designs that allow models to express ambiguity rather than committing early.
  • The seeing/seeing-as distinction might be studied by varying the language model component while fixing vision.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 2 minor

Summary. The paper examines where vision-language models make aspectual commitments on bistable images (e.g., duck-rabbit). A 3,320-generation behavioral baseline over 83 stimuli identifies three regimes (default-dominant, force-dominant, force-balanced) under neutral vs. forced-choice prompts. A TopK SAE (validation EV 0.93) is trained on the CLIP layer consumed by LLaVA-1.6-7B; across 69 stimuli with both per-aspect pools available, 72% (50/69) exhibit simultaneous activation at the vision tower (including 12/12 default-dominant duck/rabbit cases). Causal steering at layer 22 flips captions on default-dominant stimuli (33% flip rate) but not on force-balanced young/old stimuli at any tested coefficient. The authors conclude that the dominance bottleneck lies downstream of the vision tower and that the vision-language gap provides an empirical handle on the seeing/seeing-as distinction; they also note that rank-based TopK statistics require tie correction.

Significance. If the SAE faithfully isolates the per-aspect pools and the regime partitioning is robust, the work supplies a concrete, falsifiable behavioral and causal probe into multimodal aspectual commitment that is not reducible to prior fitted parameters. The combination of large-scale generation baselines, TopK SAE decomposition, and targeted steering experiments is a methodological strength that could be extended to other VLMs and ambiguity types.

major comments (3)
  1. [Abstract / Methods] Abstract and methods (regime partitioning): the reported counts (72% of 69 stimuli, 12/12 duck/rabbit, 7/8 young/old) and the differential steering result rest on the partitioning of stimuli into default-dominant/force-balanced regimes via neutral vs. forced-choice prompts. Without the exact prompt templates, data-exclusion rules, or error bars on the 3,320-generation baseline, it is impossible to verify that post-hoc choices do not affect the regime labels that underwrite the central claim.
  2. [Abstract / SAE section] SAE training and feature-pool isolation (abstract): the claim that 72% of stimuli show simultaneous activation of both per-aspect pools at the vision tower, and that steering at layer 22 cannot flip force-balanced cases, depends on the TopK SAE having isolated the two pools without selection or ranking artifacts. The validation EV=0.93 is given, but the sparsity level, training corpus, exact layer index, and raw activation vectors are not supplied; these are listed as free parameters and directly affect whether the 50/69 simultaneous-activation statistic is artifact-free.
  3. [Abstract / Steering experiments] Causal steering results (abstract): the 33% rabbit-flip rate on default-dominant stimuli versus zero flips on force-balanced young/old is presented as evidence that the bottleneck is downstream. This differential effect is load-bearing for the conclusion, yet the paper provides no quantitative comparison of steering coefficients across regimes or controls for fluency-guard interactions that could produce the observed asymmetry.
minor comments (2)
  1. [Abstract] The methodological note on tie-corrected ranking is useful but should be expanded with a short worked example showing how uncorrected ranking would bias the reported percentages.
  2. [Figures/Tables] Figure captions and table legends should explicitly state the number of stimuli per regime and the exact steering coefficients tested so that the 72% and 33% figures can be reproduced from the reported numbers alone.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback emphasizing reproducibility. We have revised the manuscript to supply the requested details on prompts, SAE hyperparameters, and steering controls, which strengthen the transparency of our regime partitioning, feature isolation, and causal results without altering the core findings.

read point-by-point responses

  1. Referee: [Abstract / Methods] Abstract and methods (regime partitioning): the reported counts (72% of 69 stimuli, 12/12 duck/rabbit, 7/8 young/old) and the differential steering result rest on the partitioning of stimuli into default-dominant/force-balanced regimes via neutral vs. forced-choice prompts. Without the exact prompt templates, data-exclusion rules, or error bars on the 3,320-generation baseline, it is impossible to verify that post-hoc choices do not affect the regime labels that underwrite the central claim.

    Authors: We agree that the exact templates, exclusion rules, and error bars are required for independent verification. The revised manuscript adds the full neutral and forced-choice prompt templates to Appendix A, specifies the exclusion criteria (generations under 8 tokens or containing explicit refusals) in Section 2.1, and reports bootstrap 95% confidence intervals on all regime proportions derived from the 3,320 generations. These intervals confirm that the 72% simultaneous-activation and 12/12 duck-rabbit statistics remain stable under resampling. revision: yes

  2. Referee: [Abstract / SAE section] SAE training and feature-pool isolation (abstract): the claim that 72% of stimuli show simultaneous activation of both per-aspect pools at the vision tower, and that steering at layer 22 cannot flip force-balanced cases, depends on the TopK SAE having isolated the two pools without selection or ranking artifacts. The validation EV=0.93 is given, but the sparsity level, training corpus, exact layer index, and raw activation vectors are not supplied; these are listed as free parameters and directly affect whether the 50/69 simultaneous-activation statistic is artifact-free.

    Authors: The revised Section 3.1 now states the sparsity (k=64), training corpus (balanced LAION-400M + COCO subset), and exact layer (CLIP ViT-L/14 final output as consumed by LLaVA-1.6-7B). Feature-pool isolation used activation thresholds on unambiguous held-out images, and tie-corrected ranking is applied as already noted in the text. Raw vectors exceed practical appendix size; we have released the trained SAE weights, feature indices for the 69 stimuli, and reproduction code in the supplementary repository so that the 50/69 count can be recomputed directly. revision: partial

  3. Referee: [Abstract / Steering experiments] Causal steering results (abstract): the 33% rabbit-flip rate on default-dominant stimuli versus zero flips on force-balanced young/old is presented as evidence that the bottleneck is downstream. This differential effect is load-bearing for the conclusion, yet the paper provides no quantitative comparison of steering coefficients across regimes or controls for fluency-guard interactions that could produce the observed asymmetry.

    Authors: A new Figure 5 in the revision plots flip rate versus steering coefficient (-2.0 to +2.0) for both regimes, with per-stimulus means and standard errors. The asymmetry is preserved across the coefficient range. Appendix C reports an ablation removing the fluency guard entirely; default-dominant flip rate remains 28% while force-balanced stays at 0%, confirming the differential result is not driven by guard interactions or coefficient selection. revision: yes

Circularity Check

0 steps flagged

No circularity; central results from new SAE training and behavioral generations

full rationale

The paper derives its claims from a newly trained TopK SAE (validation EV 0.93) on the CLIP layer consumed by LLaVA-1.6-7B and a fresh 3,320-generation baseline that partitions stimuli into regimes. The reported 72% simultaneous activation rate and differential steering outcomes are direct empirical outputs of these steps rather than quantities defined by or fitted from prior self-citations. No load-bearing derivation reduces to self-definition, fitted-input renaming, or imported uniqueness theorems.

Axiom & Free-Parameter Ledger

2 free parameters · 2 axioms · 0 invented entities

The paper relies on standard assumptions about CLIP and LLaVA architectures plus the validity of the SAE as a feature extractor; no new physical entities are postulated. Free parameters include SAE training choices and steering coefficients that are fitted or selected to produce the reported flips.

free parameters (2)
  • TopK SAE sparsity level and training hyperparameters
    Chosen to achieve validation EV 0.93 on the target CLIP layer; directly affects which feature pools are recovered.
  • Causal steering coefficients at layer 22
    Tested values that produce the 33% rabbit-flip rate; selected to demonstrate differential effect across regimes.
axioms (2)
  • domain assumption The CLIP vision tower in LLaVA-1.6-7B produces linearly separable per-aspect features that a TopK SAE can isolate without substantial crosstalk.
    Invoked when claiming simultaneous activation of both pools from SAE outputs.
  • domain assumption The neutral vs forced-choice prompting regimes cleanly partition stimuli into default-dominant, force-dominant, and force-balanced without prompt sensitivity artifacts.
    Used to classify the 83 stimuli and interpret the 72% statistic.

pith-pipeline@v0.9.1-grok · 5765 in / 1782 out tokens · 20789 ms · 2026-06-27T20:14:45.252698+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Vision-Language Asymmetry in Bistable Image Captioning." pith.science (2026). https://pith.science/paper/AQ3OWJ7A

@misc{pith2026260608031,
  author       = {Pith},
  title        = {Pith review of: Vision-Language Asymmetry in Bistable Image Captioning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AQ3OWJ7A}},
  note         = {Machine review of arXiv:2606.08031}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Wittgenstein's duck-rabbit poses a question for vision-language models: when a model captions an ambiguous image, where in the model is the commitment to one aspect made? We address this with a 3,320-generation behavioral baseline over 83 bistable stimuli that surfaces three regimes (default-dominant, force-dominant, force-balanced) under neutral vs forced-choice prompting, then probe the underlying representations using a TopK sparse autoencoder we train on the CLIP layer that LLaVA-1.6-7B actually consumes (validation EV 0.93). Across 69 bistable stimuli with both per-aspect feature pools available, 72% (50/69) show simultaneous activation of both pools at the vision tower, including 12/12 default-dominant duck/rabbit and 7/8 force-balanced young/old. Causal steering at CLIP layer 22 flips captions on default-dominant stimuli (33% rabbit-flip rate under a fluency guard) but cannot flip captions on force-balanced young/old at any tested coefficient, despite their vision-side superposition. The dominance bottleneck lives downstream of the vision tower; the gap between vision-side representation and language-side commitment is an empirical handle on the seeing/seeing-as distinction. We also flag a methodological note: rank-based statistics on TopK SAE outputs require tie-corrected ranking to avoid silent row-order bias.

Figures

Figures reproduced from arXiv: 2606.08031 by Arohan Agate.

Figure 1
Figure 1. Figure 1: Three behavioral regimes over 83 bistable stimuli (LLaVA-1.6-7B, 40 generations each, Qwen3-8B judge). Bottom￾right: default-dominant; top-left: aspect-blind under neutral prompting, splits into force-dominant (7/10, ≥70/30 commitment under forced choice) and force-balanced (3/10, ∼50/50, the Necker cubes). pool on opposite-aspect controls (Section 3) [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Phase 3: per-stimulus aspect-A (x-axis) vs aspect-B (y-axis) pool mean activation across all six groups. Marker color gives the per-stimulus classification (purple = superposition, blue = dominance A, red = dominance B, gray = neither); marker size is proportional to Phase 1 neutral-prompt dominance score. Dashed lines show per-pool thresholds (median activation on opposite-aspect controls). Purple dominat… view at source ↗
Figure 6
Figure 6. Figure 6: Phase 4: steering success (top) and fluency (bot￾tom) vs α. Vision-side steering rescues default-dominant captions (duck rabbit, 33%) and partially the mixed case (hidden face, 50–60%) under the 1.2× fluency guard, but can￾not flip force-balanced captions (young old woman, 0/7). C. Phase 4 full per-α steering results [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
Figure 3
Figure 3. Figure 3: duck rabbit and face vase retained SAE features. Top-3 features per aspect, with top-4 max-activating images across controls and bistable stimuli. Duck-features fire on duck controls, rabbit-features on rabbit controls; face-features fire on profile controls and the canonical Rubin bistable (face-leaning), vase-features on vase controls. striking: across all six analysis groups, the post-AUROC candidate co… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 4 canonical work pages · 2 internal anchors

  1. [1]

    Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics , pages=

    Evaluating vision-language models on bistable images , author=. Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics , pages=

  2. [2]

    Ehinger and Sarah Monazam Erfani and James Bailey , year=

    Xueqi Ma and Yanbei Jiang and Yan Li and Shu Liu and Jiayang Ao and Xingjun Ma and Jey Han Lau and Krista A. Ehinger and Sarah Monazam Erfani and James Bailey , year=. What Do

  3. [3]

    Advances in Neural Information Processing Systems , volume=

    Sparse autoencoders learn monosemantic features in vision-language models , author=. Advances in Neural Information Processing Systems , volume=

  4. [4]

    Steering clip’s vision transformer with sparse autoencoders.arXiv preprint arXiv:2504.08729, 2025

    Steering CLIP's vision transformer with sparse autoencoders , author=. arXiv preprint arXiv:2504.08729 , year=

    work page arXiv

  5. [5]

    Ewington-Pitsos, Louka and Goyal, Ram Rattan , year =

  6. [6]

    Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

    Improved baselines with visual instruction tuning , author=. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages=

  7. [7]

    International conference on machine learning , pages=

    Learning transferable visual models from natural language supervision , author=. International conference on machine learning , pages=. 2021 , organization=

    2021

  8. [8]

    and Stoica, Ion and Xing, Eric P

    Chiang, Wei-Lin and Li, Zhuohan and Lin, Zi and Sheng, Ying and Wu, Zhanghao and Zhang, Hao and Zheng, Lianmin and Zhuang, Siyuan and Zhuang, Yonghao and Gonzalez, Joseph E. and Stoica, Ion and Xing, Eric P. , month =. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90\ url =

  9. [9]

    Qwen3 Technical Report

    Qwen3 technical report , author=. arXiv preprint arXiv:2505.09388 , year=

    work page internal anchor Pith review Pith/arXiv arXiv

  10. [10]

    2010 , publisher=

    Philosophical Investigations , author=. 2010 , publisher=

    2010

  11. [11]

    1958 , publisher=

    Patterns of Discovery: An Inquiry into the Conceptual Foundations of Science , author=. 1958 , publisher=

    1958

  12. [12]

    2012 , publisher=

    The Structure of Scientific Revolutions , author=. 2012 , publisher=

    2012

  13. [13]

    2016 , publisher=

    Aesthetics as Philosophy of Perception , author=. 2016 , publisher=

    2016

  14. [14]

    arXiv preprint arXiv:2405.03207 , year=

    A philosophical introduction to language models-part ii: The way forward , author=. arXiv preprint arXiv:2405.03207 , year=

    work page arXiv

  15. [15]

    Mechanistic Interpretability Needs Philosophy

    Mechanistic Interpretability Needs Philosophy , author=. arXiv preprint arXiv:2506.18852 , year=

    work page internal anchor Pith review Pith/arXiv arXiv