Pith. sign in

REVIEW 4 major objections 7 minor 40 references

NanoVLMs: How small can we go and still make coherent Vision Language Models?

T0 review · 4 major / 7 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that vision-language models can be cut to 25M parameters—50 times smaller than a 1.3B baseline—and still produce coherent image descriptions, if trained on simplified child-like captions.

desk verdict The question is good and the small-VLM architecture is clean, but the Kosmos-2 comparison rests on a 25-sample, circular LLM judge; the central claim is not yet established. read the letter →

arxiv 2502.07838 v2 pith:S5GUI7DO submitted 2025-02-11 cs.CV cs.AI

Mukund Agarwalla , Himanshu Kumar , Raj Dandekar , Rajat Dandekar , Sreedath Panat This is my paper · ORCID
classification cs.CVcs.AI
keywords vision-languagemodelsmodelcompressionimagecaptioningGPT-4oevaluationchild-liketrainingdataefficienttransformersCOCOcaptions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks how small a vision-language model can be and still write fluent, coherent descriptions of images. The authors train three compact models of 5M, 16M, and 25M parameters on two new datasets, ShortDesc and LongDesc, whose captions use the simple vocabulary and syntax of a three- or four-year-old and are generated by GPT-4o from COCO images. On a 25-sample completion task graded by GPT-4o, their 25M-parameter NanoVLM-large scores close to Kosmos-2, a 1.3B-parameter model, while outperforming larger models such as BLIP-base and GIT on creativity and plot. The paper argues that extreme compression is possible without sacrificing coherent output, with the simplification of the training data doing much of the work.

What carries the argument

Three components carry the argument. The data: ShortDesc (20–25 words) and LongDesc (60–70 words), generated by GPT-4o from COCO captions with prompts that restrict output to child-like vocabulary and syntax. The architecture: a ViT-style visual encoder that receives 224×224 images as 196 patches, a single learnable projector layer followed by GELU that maps visual embeddings into the text-embedding space, and a small causal transformer decoder that predicts the next token; most parameters go to the visual encoder. The evaluation: GPT-4o grades each completion on grammar, creativity, consistency, meaningfulness, and plot, giving scores out of 10, which lets the authors measure open-ended text that standard benchmarks cannot handle.

What would settle it

Have human raters, blind to model identity, score the same 25 completions on the same five criteria, or compute an independent metric such as CIDEr against human references. If NanoVLM-large no longer matches Kosmos-2 or beats BLIP-base and GIT, the paper's compression claim does not survive.

Watch

Extended reading notes

Core claim

The central discovery is that a vision-language model with 25 million parameters can complete partial image descriptions at a level the authors call comparable to Kosmos-2, which is roughly fifty times larger (1.3B). On their short prompt task, NanoVLM-large scores 39.39 out of 50 total versus Kosmos-2's 39.86; on the long task it scores 39.84 versus 40.65, and it beats Kosmos-2 on creativity and plot in both settings. The authors position this as evidence that the bottleneck for coherent multimodal text is not raw parameter count but the match between data complexity and model capacity, and they credit the child-like simplicity of ShortDesc and LongDesc for letting a tiny decoder stay fluent and on-topic.

Load-bearing premise

The entire comparison rests on GPT-4o's scores for 25 manually chosen samples, even though GPT-4o also wrote the training captions and no human or independent benchmark check is provided.

Editorial extensions

If this is right

  • If the reported scores hold, coherent image description can be done with a 25M-parameter model, opening edge-device and mobile deployment.
  • The results imply that restricting training captions to child-like vocabulary and syntax may substitute for a much larger decoder, giving data design a central role.
  • The GPT-4o grading protocol extends LLM-as-judge evaluation to open-ended multimodal generation, where fixed benchmarks miss creativity and plot.
  • The same training recipe could carry to other data-scarce domains where simple, controlled language is available.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct test would replace GPT-4o grading with human raters; if human scores do not reproduce the near-parity with Kosmos-2, the coherence claim would be an artifact of judging stylistic similarity to GPT-4o.
  • The small dataset size (28K pairs) raises the question of whether the child-like data simplification or the architecture is responsible; an ablation training the same model on standard adult-level COCO captions would isolate the factor.
  • If robust, the result would soften the scaling assumption for multimodal text generation, suggesting that for narrow tasks, data simplicity can dominate parameter count.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes NanoVLMs, a family of 5M-25M parameter vision-language models trained from scratch on two new datasets, ShortDesc and LongDesc, derived from COCO images and captions rewritten by GPT-4o into simple, child-like vocabulary. The models are trained to complete partial image descriptions conditioned on the image. Evaluation is performed on 25 manually curated samples, with GPT-4o grading outputs on grammar, creativity, consistency, meaningfulness, and plot. The central claim is that NanoVLM-large (25M) performs comparably to Kosmos-2 (1.3B) despite being roughly 50x smaller, based on Table 3 scores (ShortDesc: 39.39 vs 39.86; LongDesc: 39.84 vs 40.65).

Significance. If the quantitative claim were valid, the paper would be significant: it would show that extreme parameter reduction in VLMs is possible when training data and task are deliberately simplified, extending the TinyStories finding to the multimodal setting. The introduced datasets and the transparent architecture are also potentially useful building blocks. However, the current evidence does not support the central claim: the evaluation is circular (GPT-4o both generated the training captions and graded the outputs), rests on 25 samples with no uncertainty quantification, and compares NanoVLMs against baselines evaluated zero-shot on a task distribution the NanoVLMs were explicitly trained for. The significance is therefore conditional on a substantially stronger evaluation that the manuscript does not provide.

major comments (4)
  1. [Section 2.1.2 and Section 3 (Figure 2, Prompts 1-3)] The evaluation is circular: GPT-4o generated the ShortDesc and LongDesc training captions (Prompts 1 and 2) and also acts as the judge for the generated completions (Prompt 3). Since the training target is itself GPT-4o text, high scores may simply measure how closely a NanoVLM imitates GPT-4o's style, not an independent property such as coherence or meaningfulness. No human ratings, no alternative judge model, and no agreement statistics are provided to break this circularity. This is load-bearing because the entire comparative claim in Table 3 rests on these scores. The authors state in the conclusion that a more extensive human evaluation could help, but no such calibration is presented in the current manuscript.
  2. [Section 3 and Table 3] All quantitative results are averages over 25 manually curated samples, reported without confidence intervals, significance tests, or inter-annotator agreement. The headline differences—NanoVLM-large vs Kosmos-2: 39.39 vs 39.86 for ShortDesc and 39.84 vs 40.65 for LongDesc—are 0.47 and 0.81 points, respectively, on a scale with per-sample score ranges that are not reported. These gaps are within plausible sampling noise, so the claim that NanoVLM-large is 'nearly comparable' or 'comparable' to Kosmos-2 is not statistically supported. The manuscript needs a larger evaluation set, error bars, and appropriate paired significance tests before this comparison can be accepted.
  3. [Section 4, Table 3] The comparison to BLIP-base, GIT, and Kosmos-2 is distributionally unfair. NanoVLMs are trained on partial-description completion with the same prompt lengths (6-7 words for short, 18-20 for long) and the same simplified vocabulary as the test set. The baseline models are general image captioners evaluated zero-shot on this unfamiliar completion task, with no description of how their inputs were formatted. Thus the observed score gap may reflect task-specific overfitting rather than parameter efficiency. A fair comparison would require fine-tuning the baselines on the same training data or, alternatively, evaluating NanoVLMs on standard captioning benchmarks (e.g., the COCO Karpathy split with CIDEr, SPICE, or BLEU) to test general-purpose capability.
  4. [Section 4, Figure 8] The ROUGE-1 analysis is used to claim that NanoVLMs produce original output rather than memorizing training data, but low n-gram overlap with training captions (less than 0.5) is also consistent with generating off-topic or incoherent text. The manuscript reports no Rouge-2 or Rouge-L scores, no details on which references were compared, and no baselines (e.g., ROUGE of a random or degenerate generator). As presented, this evidence does not support the originality claim, and the claim should be either substantiated with additional diversity/quality metrics or removed.
minor comments (7)
  1. [Abstract] The abstract claims 'up to 10 times smaller than SOTA small VLMs,' but Table 3 compares against models that are 10x to 50x larger; please state the exact comparison range or clarify the definition of 'SOTA small VLMs.'
  2. [Section 2.1.1] The description 'approximately 28K image-caption pairs' is ambiguous: COCO provides five captions per image, so please clarify whether 28K refers to images, captions, or unique image-caption pairs, and how the 90/10 split is performed.
  3. [Section 3] The list of benchmarks in the text (grammatical correctness, consistency, creativity, meaningfulness, plot) does not match the column order in Table 3 (Grammar, Creativity, Consistency, Meaningfulness, Plot); please align the order.
  4. [Figure 2] The exact prompt texts for Prompts 1, 2, and 3 are only shown in a figure; please reproduce them verbatim in the text or an appendix for reproducibility.
  5. [Section 4] The sentence 'the gap between training and validation losses remains minimal ... with a maximum observed difference of only 0.08 to 0.1' should specify whether this is the final-epoch gap or the maximum over training, and for which variant.
  6. [Section 2.3] Important training details are missing, including optimizer, learning-rate schedule, batch size, number of epochs, and the number of training steps; these should be reported to make the experiments reproducible.
  7. [References] Several citations appear incomplete or informal, such as 'AI, 2024' for Llama 3 and 'Face, 2023' for the Hugging Face blog; these should be replaced with standard technical-report or blog citations with URLs and access dates.

Circularity Check

1 steps flagged · score 4.0 of 10

GPT-4o both generates the training captions and grades the outputs, making the central near-parity claim partly self-referential.

  1. self definitional [Section 2.1.2 (dataset generation with Prompts 1–2) and Section 3 / Table 3 (evaluation with Prompt 3)]
    "For generating these descriptions, we employed OpenAI’s GPT-4o, a SOTA text generation model capable of producing high-quality synthetic content. Combined captions for each image (all captions) along with the respective prompt is passed to GPT-4o(OpenAI, 2024), where Prompt1 shown in Figure 2 is used to generate ShortDesc dataset and Prompt2 shown in Figure 2 is used to generate LongDesc dataset. ... we employ an LLM-based evaluation approach that leverages GPT-4o to assess generated text quality."

    The training targets for NanoVLM are GPT-4o-generated captions, and the evaluation scores in the central comparison (Table 3) are also GPT-4o judgments on the same style of task. Thus 'coherence,' 'meaningfulness,' and 'creativity' are operationally defined by the same system that created the training distribution. A model trained to imitate GPT-4o’s captions is therefore rewarded by the GPT-4o judge, so the near-parity with Kosmos-2 (39.39 vs 39.86 short; 39.84 vs 40.65 long) is partly a measure of GPT-4o-likeness rather than independent text quality. This is a self-referential evaluation loop by construction, although the scores themselves are empirical and could have come out differently.

full rationale

The paper’s architecture, training, and qualitative examples are independent contributions, and the ROUGE analysis provides some evidence that outputs are not memorized. However, the central quantitative claim depends on a single self-referential evaluation design: GPT-4o generated the ShortDesc/LongDesc training captions (Section 2.1.2) and GPT-4o graded the completions (Section 3, Prompt 3, Table 3). No human ratings, confidence intervals, inter-annotator agreement, or external benchmarks anchor these scores, so the reported near-parity with larger models is not independently established. This is a genuine, but partial, circularity: the result is not equivalent to its inputs, but the metric conflates quality with stylistic similarity to the generator. The authors’ own conclusion concedes that 'a more extensive human evaluation could help assess fluency, coherence, and real-world applicability,' underscoring the absence of an independent yardstick. The distributional unfairness of comparing zero-shot baselines against models trained on the same task is a separate validity concern, not a circularity. Overall score 4: the central claim retains independent content but is partly validated by a self-referential judge.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central capability claim rests on several unvalidated assumptions: the transfer of tiny-language-model findings to vision, the validity of GPT-4o as a judge without human calibration, the sufficiency of 28K training pairs and 25 test samples, the fairness of the baseline comparisons, and the interpretation of ROUGE-1 as an originality measure. These are domain assumptions chosen by the authors rather than externally established.

free parameters (3)
  • Target description length in ShortDesc and LongDesc = 20-25 words and 60-70 words
    Hand-chosen in GPT-4o prompts (Figure 2, Prompt1/Prompt2) to define the data distribution. The model cannot be evaluated outside this length/style regime, so the central capability claim is scoped to these choices.
  • Test set size = 25 samples
    Manually curated 25 samples for evaluation (Section 3). No power analysis or selection criteria are stated, yet all quantitative comparisons depend on this tiny set.
  • Evaluation rubric weighting = Equal average of grammar, consistency, creativity, meaningfulness, plot scores
    Prompt 3 asks GPT-4o to grade these five dimensions and Table 3 reports their unweighted total; no justification for equal weighting or validation against human ratings.
assumptions (5)
  • ad hoc to paper The TinyStories finding, that restricting language to a child-like vocabulary lets small models produce coherent text, transfers to vision-language models when images supply context.
    Section 1 and 2 state the 3-4 year old analogy as the design motivation; this is a heuristic transfer, not a proven principle.
  • domain assumption GPT-4o scores on 25 samples are a valid proxy for human judgment of text quality on five dimensions without calibration.
    Section 3 substitutes GPT-4o grading for human evaluation and explicitly says this 'addresses limitations of standard benchmarks', but provides no correlation with human ratings.
  • domain assumption About 28K GPT-4o-rewritten COCO caption pairs are sufficient training data for a generalizable tiny VLM.
    Section 2.1.1 selects approximately 28K pairs; the paper provides no data scaling study and the conclusion admits larger datasets could improve generalization.
  • domain assumption The three baseline VLMs were evaluated fairly with appropriate task adaptation (prompting, decoding).
    Section 4 compares against BLIP-base, GIT, and Kosmos-2 on a partial-text completion task, but no details are given for how these models were prompted or decoded; a mismatch would bias the comparison.
  • domain assumption ROUGE-1 scores below 0.5 indicate originality rather than memorization.
    Section 4 interprets low ROUGE-1 as evidence the models do not memorize training data, but the threshold is arbitrary and no comparison to a memorization baseline is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NanoVLMs: How small can we go and still make coherent Vision Language Models?." pith.science (2026). https://pith.science/paper/S5GUI7DO

@misc{pith2026250207838,
  author       = {Pith},
  title        = {Pith review of: NanoVLMs: How small can we go and still make coherent Vision Language Models?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5GUI7DO}},
  note         = {Machine review of arXiv:2502.07838}
}
read the original abstract

Vision-Language Models (VLMs), such as GPT-4V and Llama 3.2 vision, have garnered significant research attention for their ability to leverage Large Language Models (LLMs) in multimodal tasks. However, their potential is constrained by inherent challenges, including proprietary restrictions, substantial computational demands, and limited accessibility. Smaller models, such as GIT and BLIP, exhibit marked limitations, often failing to generate coherent and consistent text beyond a few tokens, even with extensive training. This underscores a pivotal inquiry: how small can a VLM be and still produce fluent and consistent text? Drawing inspiration from the exceptional learning process of 3-4 year old children, who rely heavily on visual cues for understanding and communication, we introduce two novel datasets: ShortDesc (featuring concise image descriptions) and LongDesc (containing more detailed image descriptions). These datasets consist of image-text pairs where the text is restricted to the simple vocabulary and syntax typically used by young children, generated with a scaled-down model, GPT-4o. Using these datasets, we demonstrate that it is possible to train VLMs that are significantly smaller, up to 10 times smaller than state of the art(SOTA) small VLMs while maintaining architectural simplicity. To evaluate the outputs, we leverage GPT-4o to grade the text, as if stories written by students, on creativity, meaningfulness, and consistency, assigning scores out of 10. This method addresses limitations of standard benchmarks by accommodating unstructured outputs and providing a multidimensional evaluation of the model capabilities. Our findings contribute to the development of lightweight, accessible multimodal models for resource constrained environments.

Figures

Figures reproduced from arXiv: 2502.07838 by the authors.

Figure 1
Figure 1. Root level architecture of VLM. 1. Introduction LLMs(Zheng et al., 2023; Zhao et al., 2024; OpenAI, 2023a;b) have significantly advanced natural language pro￾cessing (NLP), demonstrating strong capabilities in rea￾soning, long-form content generation and in-context learn￾ing (ICL). While models like GPT-3, LLaMA(AI, 2024), and Claude have achieved SOTA performance across text￾based tasks, their unimodal nature limit… view at source ↗
Figure 2
Figure 2. Prompts to GPT-4o for dataset creation and evaluation. et al., 2024), LLaVA(Liu et al., 2023), Flamingo(Alayrac et al., 2022), Qwen2.5-VL-7B-Instruct(Wang et al., 2024), MiniGPT-4(Zhu et al., 2024), and InstructBLIP(Ghosh et al., 2024). These models typically comprise three core archi￾tectural components: (1) a visual encoder(Kar et al., 2024; Jain et al., 2023), responsible for transforming raw images into feature-… view at source ↗
Figure 3
Figure 3. Process for creation of datasets. including people, animals, food, vehicles, and outdoor set￾tings—perfectly aligning with the learning analogy of 3–4 year old children. For our work, we specifically leveraged the image-captioning component of COCO, where each im￾age is paired with five natural language captions describing the scene and its objects. From this dataset, we selected approximately 28K image-caption pair… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: Feature extraction from an image [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]
Figure 5
Figure 5. Figure 5: Vision Transformer [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]
Figure 6
Figure 6. Figure 6: Training and validation losses of NanoVLMs. 2.2.3. DECODER BLOCK The decoder block in NanoVLM transforms fused visual￾textual embeddings into coherent text using a transformer￾based architecture, ensuring text generation. It begins by passing the multimodal token embed…
Figure 7
Figure 7. Figure 7: Sample output text and evaluation scores of various models on short and long partial text completion task. to NanoVLMs. The model then completes the partial text while attending to the image, and its output is subsequently graded using Prompt 3 (shown in [PITH_FULL_IM…
Figure 8
Figure 8. Figure 8: Histogram plot of rouge scores across each model. contrast, our three NanoVLM models maintain a more bal￾anced performance across all benchmarks, ensuring better coherence and overall stability in text completion. For long text completion, a similar trend is observed, …
Figure 9
Figure 9. Figure 9: Qualitative comparison of image descriptions generated by NanoVLMs and other VLMs. 11 [PITH_FULL_IMAGE:figures/full_fig_p011_9.png]
Figure 10
Figure 10. Figure 10: Qualitative comparison of image descriptions generated by NanoVLMs and other VLMs. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]
Figure 11
Figure 11. Figure 11: Qualitative comparison of image descriptions generated by NanoVLMs and other VLMs. 13 [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]
Figure 12
Figure 12. Figure 12: Qualitative comparison of image descriptions generated by NanoVLMs and other VLMs. 14 [PITH_FULL_IMAGE:figures/full_fig_p014_12.png]
Figure 13
Figure 13. Figure 13: Qualitative comparison of image descriptions generated by NanoVLMs and other VLMs. 15 [PITH_FULL_IMAGE:figures/full_fig_p015_13.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

40 extracted references · 9 canonical work pages

  1. [1]

    Agarap, A. F. Deep learning using rectified linear units (relu), 2019. URL https://arxiv.org/abs/1803.08375

    arXiv 2019

  2. [2]

    L., and Parikh, D

    Agrawal, A., Lu, J., Antol, S., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp.\ 2425--2433, 2015. doi:10.1109/ICCV.2015.279

    doi:10.1109/iccv.2015.279 2015

  3. [3]

    H., Patra, B., Aggarwal, K., Corro, L

    Ahuja, S., Tanmay, K., Chauhan, H. H., Patra, B., Aggarwal, K., Corro, L. D., Mitra, A., Dhamecha, T. I., Awadallah, A., Choudhary, M., Chaudhary, V., and Sitaram, S. sphinx: Sample efficient multilingual instruction fine-tuning through n-shot guided prompting, 2024. URL https://arxiv.org/abs/2407.09879

    arXiv 2024

  4. [4]

    Llama 3: Next-generation open-source language models, 2024

    AI, M. Llama 3: Next-generation open-source language models, 2024. URL https://ai.meta.com/blog/llama-3-release. Accessed: 31-Jan-2025

    work page 2024

  5. [5]

    Multimodal deep learning, 2023

    Akkus, C., Chu, L., Djakovic, V., Jauch-Walser, S., Koch, P., Loss, G., Marquardt, C., Moldovan, M., Sauter, N., Schneider, M., Schulte, R., Urbanczyk, K., Goschenhofer, J., Heumann, C., Hvingelby, R., Schalk, D., and Aßenmacher, M. Multimodal deep learning, 2023. URL https://arxiv.org/abs/2301.04856

    arXiv 2023

  6. [6]

    Flamingo: a visual language model for few-shot learning, 2022

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., Brock, A., Nematzadeh, A., Sharifzadeh, S., Binkowski, M., Barreira, R., Vinyals, O., Zisserman, A., and Simonyan, K. Flaming...

    arXiv 2022

  7. [7]

    L., Kiros, J

    Ba, J. L., Kiros, J. R., and Hinton, G. E. Layer normalization, 2016. URL https://arxiv.org/abs/1607.06450

    arXiv 2016

  8. [8]

    Honeybee: Locality-enhanced projector for multimodal llm, 2024

    Cha, J., Kang, W., Mun, J., and Roh, B. Honeybee: Locality-enhanced projector for multimodal llm, 2024. URL https://arxiv.org/abs/2312.06742

    arXiv 2024

Show all 40 references
  1. [9]

    Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Doll \'a r, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    2015 arXiv

  2. [10]

    and Li, Y

    Eldan, R. and Li, Y. Tinystories: How small can language models be and still speak coherent english?, 2023. URL https://arxiv.org/abs/2305.07759

    2023 arXiv

  3. [11]

    Vision-language pretraining: Bridging vision and language with transformers, 2023

    Face, H. Vision-language pretraining: Bridging vision and language with transformers, 2023. URL https://huggingface.co/blog/vision_language_pretraining. Accessed: 2025-01-31

    2023

  4. [12]

    Exploring the frontier of vision-language models: A survey of current methodologies and future directions, 2024

    Ghosh, A., Acharya, A., Saha, S., Jain, V., and Chadha, A. Exploring the frontier of vision-language models: A survey of current methodologies and future directions, 2024. URL https://arxiv.org/abs/2404.07214

    2024

  5. [13]

    Vcoder: Versatile vision encoders for multimodal large language models

    Jain, J., Yang, J., and Shi, H. Vcoder: Versatile vision encoders for multimodal large language models. arXiv preprint arXiv:2312.14233, 2023

    2023 arXiv

  6. [14]

    The illustrated gpt-2, 2019

    Jalammar, J. The illustrated gpt-2, 2019. URL https://jalammar.github.io/illustrated-gpt2/. Accessed: 2025-01-31

    2019

  7. [15]

    F., Tonioni, A., Poklukar, P., Kulshrestha, A., Zamir, A., and Tombari, F

    Kar, O. F., Tonioni, A., Poklukar, P., Kulshrestha, A., Zamir, A., and Tombari, F. Brave: Broadening the visual encoding of vision-language models. arXiv preprint arXiv:2404.07204, 2024

    2024 arXiv

  8. [16]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022

    Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022. URL https://arxiv.org/abs/2201.12086

    2022 arXiv

  9. [17]

    Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Doll \'a r, P., and Zitnick, C. L. Microsoft coco: Common objects in context. European conference on computer vision (ECCV), pp.\ 740--755, 2014

    2014

  10. [18]

    Forgetful causal masking makes causal language models better few-shot learners

    Liu, H., Geng, X., Lee, L., Mordatch, I., Levine, S., Narang, S., and Abbeel, P. Forgetful causal masking makes causal language models better few-shot learners. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=YrZEKNLWhlp

    2022

  11. [19]

    Liu, H., Li, C., Wu, Q., and Lee, Y. J. Visual instruction tuning, 2023. URL https://arxiv.org/abs/2304.08485

    2023 arXiv

  12. [20]

    Vividmed: Vision language model with versatile visual grounding for medicine, 2024

    Luo, L., Tang, B., Chen, X., Han, R., and Chen, T. Vividmed: Vision language model with versatile visual grounding for medicine, 2024. URL https://arxiv.org/abs/2410.12694

    2024 arXiv

  13. [21]

    Chatgpt: A language model for conversational ai

    OpenAI. Chatgpt: A language model for conversational ai. Tech. rep., OpenAI, 2023 a . [Online]. Available: https://www.openai.com/research/chatgpt

    2023

  14. [22]

    Gpt-4 technical report

    OpenAI. Gpt-4 technical report. arXiv, 2303.08774, 2023 b . [Online]. Available: https://arxiv.org/abs/2303.08774

    2023 arXiv

  15. [23]

    Gpt-4o system card, 2024

    OpenAI. Gpt-4o system card, 2024. URL https://openai.com/index/gpt-4o-system-card/. Accessed: 2025-01-31

    2024

  16. [24]

    Kosmos-2: Grounding multimodal large language models to the world, 2023

    Peng, Z., Wang, W., Dong, L., Hao, Y., Huang, S., Ma, S., and Wei, F. Kosmos-2: Grounding multimodal large language models to the world, 2023. URL https://arxiv.org/abs/2306.14824

    2023 arXiv

  17. [25]

    A., Wang, L., Cervantes, C

    Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockenmaier, J., and Lazebnik, S. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp....

    2015 doi

  18. [26]

    W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I

    Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision, 2021. URL https://arxiv.org/abs/2103.00020

    2021 arXiv

  19. [27]

    An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition

    Shi, B., Bai, X., and Yao, C. An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 1532--1540, 2017. doi:10.1109/...

    2017 doi

  20. [28]

    L., and Zou, J

    Thapa, R., Chen, K., Covert, I., Chalamala, R., Athiwaratkun, B., Song, S. L., and Zou, J. Dragonfly: Multi-resolution zoom-in encoding enhances vision-language models, 2024. URL https://arxiv.org/abs/2406.00977

    2024 arXiv

  21. [29]

    N., Kaiser, L., and Polosukhin, I

    Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I. Attention is all you need, 2023. URL https://arxiv.org/abs/1706.03762

    2023 arXiv

  22. [30]

    Show and tell: A neural image caption generator

    Vinyals, O., Toshev, A., Bengio, S., and Erhan, D. Show and tell: A neural image caption generator. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 3156--3164, 2015. doi:10.1109/CVPR.2015.7298935

    2015

  23. [31]

    Git: A generative image-to-text transformer for vision and language, 2022

    Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. Git: A generative image-to-text transformer for vision and language, 2022. URL https://arxiv.org/abs/2205.14100

    2022 arXiv

  24. [32]

    Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., and Lin, J. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024. URL h...

    2024 arXiv

  25. [33]

    Efficient vision-language models by summarizing visual tokens into compact registers, 2024

    Wen, Y., Cao, Q., Fu, Q., Mehta, S., and Najibi, M. Efficient vision-language models by summarizing visual tokens into compact registers, 2024. URL https://arxiv.org/abs/2410.14072

    2024 arXiv

  26. [34]

    mplug-owl: Modularization empowers large language models with multimodality, 2024

    Ye, Q., Xu, H., Xu, G., Ye, J., Yan, M., Zhou, Y., Wang, J., Hu, A., Shi, P., Shi, Y., Li, C., Xu, Y., Chen, H., Tian, J., Qian, Q., Zhang, J., Huang, F., and Zhou, J. mplug-owl: Modularization empowers large language models with multimodality, 2024. URL https://arxiv.org/abs/...

    2024 arXiv

  27. [35]

    Stablemask: Refining causal masking in decoder-only transformer

    Yin, Q., He, X., Zhuang, X., Zhao, Y., Yao, J., Shen, X., and Zhang, Q. Stablemask: Refining causal masking in decoder-only transformer. arXiv preprint arXiv:2402.04779, 2024. URL https://arxiv.org/abs/2402.04779

    2024 arXiv

  28. [36]

    Tinygpt-v: Efficient multimodal large language model via small backbones, 2024

    Yuan, Z., Li, Z., Huang, W., Ye, Y., and Sun, L. Tinygpt-v: Efficient multimodal large language model via small backbones, 2024. URL https://arxiv.org/abs/2312.16862

    2024 arXiv

  29. [37]

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., Liu, P., Nie, J.-Y., and Wen, J.-R. A survey of large language models, 2024. URL https://ar...

    2024 arXiv

  30. [38]

    P., Zhang, H., Gonzalez, J

    Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E. P., Zhang, H., Gonzalez, J. E., and Stoica, I. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023. URL https://arxiv.org/abs/2306.05685

    2023 arXiv

  31. [39]

    Mini GPT -4: Enhancing vision-language understanding with advanced large language models

    Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Mini GPT -4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=1tZbq88f27

    2024

  32. [40]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.