Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision

Hyunjung Shim; Raehyuk Jung; Seungjun Yu

REVIEW 4 major objections 5 minor 33 references

Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that the small trainable projection layer between a frozen vision encoder and a frozen LLM keeps 79–88% of its seen-class accuracy on entirely held-out unseen class labels, and that the frozen language model's feed-forward

desk verdict A solid benchmark for measuring projection-layer generalization to unseen labels; the split-leak concern is real but the OpenImages result largely covers it—needs error bars and split validation. read the letter →

arxiv 2509.00700 v2 pith:ZUVSBJJM submitted 2025-08-31 cs.CV

Raehyuk Jung , Seungjun Yu , Hyunjung Shim This is my paper

classification cs.CV

keywords vision-languagemodelsprojectionlayerunseenclassgeneralizationalignmenttrainingbenchmarkmechanisticinterpretabilitykey-valuememoryVisualGenome

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Vision-language models often join a frozen vision encoder and a frozen LLM with a small trainable projection layer. The paper asks whether that projection layer can recognize visual concepts never seen during alignment training, and builds a controlled benchmark to test it: class labels from Visual Genome are split into disjoint seen and unseen sets, prompts are built from bounding-box annotations, and only the projection layer is trained. Across four vision encoders, several LLMs, and an entirely independent evaluation set from OpenImages, the projection layer keeps 79–88% of seen-class accuracy on unseen classes, roughly 2.7 times random chance. A dataset ablation shows class diversity, not raw example count, is the driver; a mechanistic analysis of the frozen LLM's feed-forward networks indicates seen and unseen visual tokens are stored and retrieved through the same key-value memory machinery. If the conclusion holds, the architectural bet behind modern VLMs—that a tiny bridge can align two huge pretrained spaces—is sound, and alignment data can be far smaller than current practice.

What carries the argument

The central object is the projection layer, a single linear transformation that maps patch-level visual embeddings into the LLM's token embedding space; the benchmark makes it the only trainable component, isolating its contribution. The interpretability apparatus is the FFN key-value memory view: each row of the feed-forward expansion weight acts as a key detecting a pattern in the input prefix, and each column of the compression weight acts as a value spreading probability toward the next token. Applying this view to visual tokens, the paper extracts the most-activated keys and projects the corresponding values into the vocabulary space to read off which labels they predict. This machinery

What would settle it

Compute the average cosine similarity between each unseen class label and its nearest seen label in the same text-embedding space used for the split; if that value is close to the average similarity among random label pairs, the separation is real. If it is high (e.g., beach vs shore), then the benchmark is partially testing near-duplicate recognition rather than genuinely novel concepts; a re-split that explicitly enforces a large inter-cluster similarity margin should reduce the reported unseen accuracy if the claim is being inflated.

Watch

Extended reading notes

Core claim

On its own terms, the central empirical claim is a number: the projection layer's accuracy on held-out unseen classes is between 79% and 88% of its accuracy on seen classes, with a relative gain over random guessing of 137–197%, across every configuration tested. The result holds for self-supervised encoders (DINOv2, MAE), a classification-trained ViT, and text-supervised CLIP; for LLMs from 0.6B to 3B parameters; and when evaluation moves to OpenImages, a dataset with no image or label overlap with the training set. The accompanying mechanistic claim is that the feed-forward networks in the frozen LLM act as unnormalized key-value memories for visual prefixes just as they do for text: a key

Load-bearing premise

The unseen class labels are split from seen labels using text-embedding k-means, and the paper asserts this minimizes semantic overlap; if some unseen labels are near-synonyms of seen labels, the reported 'unseen' accuracy could be inflated.

Editorial extensions

If this is right

If a 4.2M-parameter projection trained on roughly 700k examples generalizes to unseen classes, then alignment data can be collected for diversity of labels rather than exhaustiveness of concepts, and training budgets can shrink without sacrificing open-vocabulary capability.
The ordering of encoders by unseen accuracy—CLIP above DINOv2 above classification ViT above MAE—matches the neighborhood-alignment ordering reported by prior cross-modal representation analyses, suggesting projection-layer generalization is inherited from how well the frozen visual and language spaces already agree.
Because a pretraining-only LLM performs almost identically to the instruction-tuned version, the generalization does not depend on chat-style instruction supervision; it is a property of the base language-model space.
The class-preserving ablation (all labels, fewer examples per label) beats the class-exclusive ablation at every data budget, giving a concrete rule: when building a projection-layer alignment set, prioritize label coverage over per-label instance count.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The split relies on text-embedding k-means to guarantee semantic separation, but the paper does not quantify that separation; a natural check is to measure the nearest-cross-split cosine similarity for each label. If pairs like 'beach' and 'shore' fall on opposite sides, part of the 79–88% could be near-duplicate recognition rather than novelty.
The key-value analysis implies the frozen LLM already has the semantic organization needed to interpret visual tokens; this suggests the projection layer may be learning a near-linear translation of the vision encoder's existing space, in which case even substantially weaker or smaller projections might show partial unseen generalization—a testable prediction.
The same benchmark could be exported to audio or other modalities by using object-detection-style annotations from non-visual sources, testing whether the result is specific to the vision-language pairing or a general property of aligning any frozen perceptual encoder with a frozen language model.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Read the paper. The core result—that a projection layer trained on one label set retains roughly 80–88% of its seen-class accuracy on held-out labels—is genuinely new and I think it holds up better than the reader's conditional verdict suggests. The benchmark design is careful: disjoint label splits, macro-averaged accuracy, ablations across four vision encoders, three LLMs, and an independent OpenImages test. The dataset ablation isolating class diversity is a clean experiment. The code is released. That's real evidence.

The main soft spot is the label split. The authors use spherical k-means with K=2 on MiniLM embeddings and assert this minimizes semantic overlap, but they never quantify the separation. If near-duplicates like 'beach' and 'shore' land on opposite sides, unseen accuracy could be inflated by recognizing seen concepts. This is a legitimate worry, and the paper should report a quantitative measure of inter-cluster distance or a human-labeled validation of the split. But it's not fatal, because the OpenImages experiment uses labels entirely outside Visual Genome, with different label styles, and the model still reaches an RGR of 175%. That result is hard to explain by near-duplicate leakage. So the headline claim of non-trivial generalization to unseen labels stands, even if the exact 79–88% numbers might be a few points optimistic.

The other soft spots are minor. No error bars or repeated seeds, so we don't know the variance. Given the effect sizes (RGR 137–197% vs. 100% random), the qualitative conclusion is unlikely to flip, but for benchmark numbers, they should run multiple seeds. The mechanistic analysis is presented as observation, not causal proof, and the comparison to random baselines is appropriate.

Who gets value: anyone working on efficient VLM alignment, projection-layer design, or cross-modal representation convergence. It's a useful evaluation tool and a reference point for future work. I'd bring it to a reading group and cite it.

Recommendation: send to peer review. Request revisions for error bars and split validation, but the central finding is solid and worth the community's time.

Referee Report

4 major / 5 minor

Summary. The paper proposes a benchmark for evaluating whether the projection layer of a vision-language model (VLM) generalizes to visual concepts whose labels were unseen during alignment training. Detection annotations from Visual Genome are converted into prompt-style samples, and the label space is split into disjoint seen/unseen groups via spherical k-means (K=2) on text embeddings from all-MiniLM-L6-v2. Only the projection layer is trained, keeping the vision encoder and LLM frozen. The main empirical claim is that the projection layer retains 79–88% of seen-class accuracy on unseen classes, measured by macro-averaged accuracy on a 4-choice MCQA benchmark. This claim is supported by ablations over vision encoders and LLMs, and by an out-of-distribution evaluation on OpenImages. A dataset ablation study identifies class diversity as the key factor. A mechanistic analysis, using FFN key-value memory techniques adapted from LLM interpretability, suggests that the LLM processes seen and unseen visual tokens similarly.

Significance. If the main claim holds, this is a useful and nontrivial result: it would show that a lightweight projection layer can transfer visual concepts beyond its alignment supervision, which bears directly on efficient VLM training and on modular design choices. The experimental design has important strengths: held-out label sets, macro-averaged evaluation to counter class imbalance, multiple vision encoders and language models, a separate out-of-distribution dataset, and random baselines in the mechanistic analysis. The released code supports reproducibility. The mechanistic findings are secondary but align with prior work. However, the central measurement depends on the semantic disjointness of the seen/unseen split, and the paper does not quantitatively validate that split; the absence of repeated-seed statistics also weakens the strength of the empirical claims.

major comments (4)

[§3 (Dataset Curation)] The seen/unseen split is load-bearing for every downstream number, yet the assertion that spherical k-means with K=2 on all-MiniLM-L6-v2 embeddings 'minimizes semantic overlap' is not quantified. Please report cross-split semantic separation: e.g., the distribution of cosine similarities between each unseen label and its nearest seen label, the fraction of unseen labels whose nearest neighbor is in the seen set, and a comparison against random label partitions. Also provide examples of the closest cross-split label pairs. Without this, the 79–88% retention could partly reflect recognition of near-duplicate seen concepts rather than generalization to novel concepts. The OpenImages experiment partially mitigates this, but Table 3 reports only absolute accuracy, not a seen-baseline comparison.
[§4, Tables 1–3] All reported numbers come from single runs with no error bars or repeated seeds. The paper uses phrases such as 'consistently' and 'rigorously designed,' but the central quantitative claim (79–88% relative performance) is presented without variance. Since training is one epoch over a single projection layer, repeating each configuration for at least three seeds is inexpensive. Report mean±std over seeds, or at least verify that the main conclusions are stable.
[§4, 'Investigating FFN on unseen label generalization'] The random baselines in the key-value analysis are described loosely: for keys, 'randomly sampling three class labels from within the same label group,' and for values, 'selecting three random values per layer.' It is unclear how many random samples were averaged, whether the same number of keys/values were used, and whether any statistical test supports the claim that the real values 'surpass the baseline by a clear margin' (Figures 6–7). Add explicit sampling details and confidence intervals or significance tests.
[§4, 'Evaluation on Out-of-Distribution Dataset (OpenImages)'] The sentence 'its accuracy on OpenImages reaches 92.5% of that on test-unseen' compares an absolute OpenImages accuracy (68.7) to the Visual Genome test-unseen accuracy (74.2) from a different evaluation set. This is not a relative-performance measurement, because there is no OpenImages seen-baseline. This statement should be rephrased or replaced with a comparison that isolates the effect of the distribution shift.

minor comments (5)

[Abstract/Intro] Typos: 'generalizibility' and 'suituable' should be corrected; 'LLaV A' formatting is inconsistent throughout.
[§3 (Evaluation Dataset and Protocol)] After frequency filtering, the paper reports 600 seen and 468 unseen MCQA classes, but earlier the split produced 1,780 seen and 1,246 unseen classes. Clarify how the filtering and subsampling reduce these numbers, and whether the 20-instance cutoff is applied before or after subsampling.
[§4, Table 1] The vision encoders differ in input resolution (CLIP uses 336×336, others 224×224). This could affect the comparison. State whether all encoders' patch tokens were used identically and whether any resolution normalization was applied.
[§4, Figure 3] The dataset ablation plot shows lines without error bars. Given the claim that class-preserving consistently outperforms class-exclusive, repeated runs would strengthen this figure. At minimum, state the number of runs.
[§4, Table 4] The layout of Table 4 is confusing: the 'Keys Top tokens from value' header spans multiple columns. Also, clarify whether the percentages in the class-label columns refer to the proportion of activating prefixes or the proportion of prefixes in the dataset that activate the key.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the 79-88% unseen retention is a held-out measurement, not a fitted or self-referential quantity; the main risk is unquantified split quality, which is a validity concern rather than circularity.

full rationale

The central claim—that the projection layer retains 79–88% of seen accuracy on unseen labels—is an empirical measurement on disjoint label sets. The model is trained only on train-seen prompts, and test-unseen labels never appear in the training text, so the ratio unseen_accuracy/seen_accuracy is not defined in terms of a fitted parameter nor forced by construction. The only potentially load-bearing construction is the seen/unseen split: the paper states, 'To minimize semantic overlap between the two, we embed all class labels using the all-MiniLM-L6-v2 model and perform spherical k-means clustering with K=2' (Section 3, Dataset Curation), but it does not quantify the separation. If near-duplicate labels straddle the split, unseen accuracy could be inflated. This is a legitimate benchmark-validity risk, but it is not circularity: the accuracy numbers are not derived from the k-means objective, and the OpenImages experiment (Table 3) provides an external dataset with a different label style, yielding 174.8 RGR and 92.5% of test-unseen accuracy, partially mitigating the concern. The mechanistic interpretability analysis compares against random baselines sampled from the same label group, so within-group embedding similarity induced by the clustering is controlled for in the baseline; the finding that FFNs act as key-value memories is an interpretation of observed activations, not a reduction to the paper's own assumptions. There are no load-bearing self-citations: references to prior work (Geva et al. 2021; Neo et al. 2025; Huh et al. 2024) are external and used for methodology or consistency checks. Overall, the derivation chain does not reduce to its inputs; the paper is a self-contained empirical study with a caveat about split semantics that belongs to correctness/validity discussion, not circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper's central measurements rest on several curation and evaluation assumptions rather than on fitted constants. The most important is the label-partition assumption, because a semantically leaky split would undermine the 'unseen' designation. No new physical or conceptual entities are introduced.

free parameters (3)

cosine threshold for OpenImages label deduplication = 0.9
Classes with cosine similarity >0.9 are removed as near-duplicates. This hand-set threshold affects the OpenImages MCQA set composition but is not fitted to the target result.
spherical k-means K = 2
Used to partition Visual Genome labels into seen/unseen groups. K=2 is chosen to create two clusters, but the semantic separation is not quantitatively validated.
class frequency filters = >=10 (dataset), >=20 (evaluation), <=200 (subsample)
Classes with fewer than 10 instances are dropped from curation; evaluation excludes classes with fewer than 20 and subsamples to 200. These thresholds affect test balance but are conventional rather than fitted.

assumptions (5)

domain assumption Visual Genome class labels and bounding boxes are accurate enough to serve as training and evaluation ground truth for VLM alignment.
The entire benchmark is built on Visual Genome annotations; if they are noisy, the measured generalization may be affected. Invoked in Section 3.
domain assumption The MCQA loss-based selection (SeedBench protocol) faithfully measures the model's alignment, with lower loss indicating better answer.
Evaluation assumes the model's choice with minimum negative log-likelihood is its most aligned prediction. Invoked in Section 3 Evaluation Protocol.
domain assumption all-MiniLM-L6-v2 text embeddings capture semantic similarity sufficiently for label partitioning and deduplication.
Used to cluster seen/unseen labels and filter OpenImages near-duplicates. If embeddings poorly reflect visual or semantic similarity, the split may leak related concepts. Invoked in Section 3.
ad hoc to paper Spherical k-means with K=2 partitions labels semantically, minimizing overlap between seen and unseen groups.
The paper asserts the partition 'minimizes semantic overlap' but provides no quantitative measure of separation. This is a load-bearing design choice for the claim that unseen classes are truly novel.
domain assumption Frozen vision encoder and LLM remain compatible with the simple prompt format, and training only the projection layer for one epoch is sufficient to learn meaningful alignment.
The conclusion about projection-layer generalization assumes the frozen components are not limiting and the one-epoch training is adequate. Invoked in Model Architecture and Implementation Details.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision." pith.science (2026). https://pith.science/paper/ZUVSBJJM

@misc{pith2026250900700,
  author       = {Pith},
  title        = {Pith review of: Prompt the Unseen: Evaluating Visual-Language Alignment Beyond Supervision},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZUVSBJJM}},
  note         = {Machine review of arXiv:2509.00700}
}

read the original abstract

Vision-Language Models (VLMs) combine a vision encoder and a large language model (LLM) through alignment training, showing strong performance on multimodal tasks. A central component in this architecture is the projection layer, which maps visual features into the LLM's embedding space. Despite its importance, its ability to generalize to unseen visual concepts has not been systematically evaluated. To address this, we propose a benchmark for evaluating projection-layer generalization. We adapt object detection datasets (rich in fine-grained annotations) into a prompting format and design train/test splits with disjoint label sets, enabling precise control over seen and unseen concept separation. Experimental results show that the projection layer retains about 79 to 88 percent of the performance on unseen classes compared to seen ones across various settings, suggesting a non-trivial level of generalization even without explicit alignment supervision on those concepts. We further analyze this behavior through a mechanistic interpretability lens. Our findings indicate that the feed-forward network in the projection layer functions like a key-value memory, processing seen and unseen tokens in similar ways. This study introduces a new evaluation framework for alignment generalization and highlights the potential for efficient VLM training with limited aligned data.

Figures

Figures reproduced from arXiv: 2509.00700 by the authors.

**Figure 2.** This figure illustrates how an object detection an [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 3.** We present results from the dataset ablation study. [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: This shows violin plots of cosine similarity distri [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Layer-wise violin plots of cosine similarities for [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: This plot shows agreement rate and cosine simi [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]

**Figure 7.** Figure 7: This plot shows the same metrics as in Fig. 6, but [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

33 extracted references · 15 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

2022
[4]

Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

arXiv 2025
[5]

Chen, Z.; Wang, W.; Cao, Y.; Liu, Y.; Gao, Z.; Cui, E.; Zhu, J.; Ye, S.; Tian, H.; Liu, Z.; et al. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

arXiv 2024
[6]

E.; et al

Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3): 6

work page 2023
[7]

Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

arXiv 2020
[8]

Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv e-prints, arXiv--2407

2024

Show all 33 references

[9]

Everingham, M.; Van Gool, L.; Williams, C. K. I.; Winn, J.; and Zisserman, A. 2010. The Pascal Visual Object Classes (VOC) Challenge. International Journal of Computer Vision, 88(2): 303--338

2010
[10]

Geva, M.; Schuster, R.; Berant, J.; and Levy, O. 2021. Transformer Feed-Forward Layers Are Key-Value Memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, ...

2021
[11]

He, K.; Chen, X.; Xie, S.; Li, Y.; Doll \'a r, P.; and Girshick, R. 2022. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 16000--16009

2022
[12]

Huh, M.; Cheung, B.; Wang, T.; and Isola, P. 2024. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987

2024 arXiv
[13]

Krasin, I.; Duerig, T.; Alldrin, N.; Ferrari, V.; Abu-El-Haija, S.; Kuznetsova, A.; Rom, H.; Uijlings, J.; Popov, S.; Veit, A.; et al. 2017. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github. co...

2017
[14]

A.; et al

Krishna, R.; Zhu, Y.; Groth, O.; Johnson, J.; Hata, K.; Kravitz, J.; Chen, S.; Kalantidis, Y.; Li, L.-J.; Shamma, D. A.; et al. 2017. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123: 32--73

2017
[15]

H.; Gurnee, W.; and Tegmark, M

Lad, V.; Lee, J. H.; Gurnee, W.; and Tegmark, M. 2024. The remarkable robustness of llms: Stages of inference? arXiv preprint arXiv:2406.19384

2024 arXiv
[16]

Li, B.; Ge, Y.; Ge, Y.; Wang, G.; Wang, R.; Zhang, R.; and Shan, Y. 2024 a . Seed-bench: Benchmarking multimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13299--13308

2024
[17]

Li, B.; Zhang, Y.; Guo, D.; Zhang, R.; Li, F.; Zhang, H.; Zhang, K.; Zhang, P.; Li, Y.; Liu, Z.; et al. 2024 b . Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326

2024 arXiv
[18]

Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Doll \'a r, P.; and Zitnick, C. L. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, 740--755. Springer

2014
[19]

Liu, H.; Li, C.; Wu, Q.; and Lee, Y. J. 2023. Visual instruction tuning. Advances in neural information processing systems, 36: 34892--34916

2023
[20]

Liu, H.; and other contributors. 2023. LLaVA Official Implementation. https://github.com/haotian-liu/LLaVA. Accessed: 2025-07-31

2023
[21]

Liu, Y.; Zhang, Y.; and Yeung-Levy, S. 2025. Mechanistic Interpretability Meets Vision Language Models: Insights and Limitations. https://d2jud02ci9yv69.cloudfront.net/2025-04-28-vlm-understanding-29/blog/vlm-understanding/. ICLR 2025 Blogpost Track

2025
[22]

Liu, Z.; and He, K. 2024. A decade's battle on dataset bias: Are we there yet? arXiv preprint arXiv:2403.08632

2024 arXiv
[23]

Merullo, J.; Castricato, L.; Eickhoff, C.; and Pavlick, E. 2023. Linearly Mapping from Image to Text Space. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net

2023
[24]

Miller, G. A. 1995. WordNet: a lexical database for English. Communications of the ACM, 38(11): 39--41

1995
[25]

Neo, C.; Ong, L.; Torr, P.; Geva, M.; Krueger, D.; and Barez, F. 2025. Towards Interpreting Visual Information Processing in Vision-Language Models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025 . OpenReview.net

2025
[26]

nostalgebraist. 2020. Interpreting GPT: the logit lens. https://nostalgebraist.tumblr.com/post/627222722064900096/interpreting-gpt-the-logit-lens. Accessed: 2024-07-08

2020
[27]

Oquab, M.; Darcet, T.; Moutakanni, T.; Vo, H.; Szafraniec, M.; Khalidov, V.; Fernandez, P.; Haziza, D.; Massa, F.; El-Nouby, A.; et al. 2023. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193

2023 arXiv
[28]

W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al

Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, 8748--8763. PmLR

2021
[29]

Schuhmann, C.; Beaumont, R.; Vencu, R.; Gordon, C.; Wightman, R.; Cherti, M.; Coombes, T.; Katta, A.; Mullis, C.; Wortsman, M.; et al. 2022. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural information processing systems,...

2022
[30]

Srinivasan, K.; Raman, K.; Chen, J.; Bendersky, M.; and Najork, M. 2021. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, 2443--2449

2021
[31]

Yang, A.; Li, A.; Yang, B.; Zhang, B.; Hui, B.; Zheng, B.; Yu, B.; Gao, C.; Huang, C.; Lv, C.; et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388

2025 arXiv
[32]

Yin, S.; Fu, C.; Zhao, S.; Li, K.; Sun, X.; Xu, T.; and Chen, E. 2023. A Survey on Multimodal Large Language Models. arXiv preprint arXiv:2306.13549

2023 arXiv
[33]

J.; Heo, B.; Han, D.; Choe, J.; and Chun, S

Yun, S.; Oh, S. J.; Heo, B.; Han, D.; Choe, J.; and Chun, S. 2021. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2340--2350

2021

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Alayrac, J.-B.; Donahue, J.; Luc, P.; Miech, A.; Barr, I.; Hasson, Y.; Lenc, K.; Mensch, A.; Millican, K.; Reynolds, M.; et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716--23736

2022

[4] [4]

Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; et al. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923

arXiv 2025

[5] [5]

Chen, Z.; Wang, W.; Cao, Y.; Liu, Y.; Gao, Z.; Cui, E.; Zhu, J.; Ye, S.; Tian, H.; Liu, Z.; et al. 2024. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271

arXiv 2024

[6] [6]

E.; et al

Chiang, W.-L.; Li, Z.; Lin, Z.; Sheng, Y.; Wu, Z.; Zhang, H.; Zheng, L.; Zhuang, S.; Zhuang, Y.; Gonzalez, J. E.; et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90\ See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3): 6

work page 2023

[7] [7]

Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929

arXiv 2020

[8] [8]

Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv e-prints, arXiv--2407

2024