Pith. sign in

REVIEW 4 major objections 6 minor 21 references

Multimodal Instruction Tuning with Hybrid State Space Models

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that a hybrid transformer-Mamba backbone lets a multimodal model handle over 100k visual tokens and run about four times faster than current open-source models at high resolutions and frame rates.

desk verdict A promising train-short/infer-long recipe for hybrid SSM multimodal LLMs, but the evaluation is not trustworthy because the LLaVA-NeXT baselines are broken and the scaling tables contradict the abstract. read the letter →

arxiv 2411.08840 v1 pith:TTBVR45U submitted 2024-11-13 cs.CV

Jianing Zhou , Han Li , Shuai Zhang , Ning Xie , Ruijie Wang , Xiaohan Nie , Sheng Liu , Lingyun Wang This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelhybridstatespaceMambalong-contextvisionhigh-resolutionimagesvideounderstandinginstructiontuningtrain-short-infer-long
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal models that read high-resolution images or many video frames pay a quadratic price because self-attention scales with the number of visual tokens. This paper argues that swapping the attention-only backbone for a hybrid transformer-Mamba model removes that quadratic penalty, allowing a model trained on short, low-cost inputs to process more than 100k tokens at inference. The two concrete promises are that the resulting model, MMJamba, outperforms existing open models across the benchmarks tested and runs about four times faster than current open-source models at very high resolutions and frame rates. If true, high-resolution and high-frame-rate multimodal understanding becomes available without the usual training cost for long contexts.

What carries the argument

The load-bearing mechanism is the hybrid Jamba backbone: Mamba selective state-space layers, whose hidden state updates every token at linear cost, interleaved with transformer attention layers and mixture-of-experts. The recurrent state is what lets the model carry information over sequences far longer than those seen in training, and the paper's train-on-short-infer-on-long recipe is built directly on that recurrence. The recipe freezes the vision encoder, trains only the adapter and LLM on short sequences (at most 4 tiles, 672x672 images, and 8 frames), and then at inference feeds much larger tile counts and frame counts through the same recurrent layers.

What would settle it

Feed MMJamba a single multimodal input over 100k tokens, such as 64 high-resolution frames plus a long prompt, and measure time-to-first-token together with accuracy on a question about one of the earliest frames. The central claim predicts near-linear latency growth and stable accuracy; quadratic latency scaling or a sharp accuracy drop for the early-frame question would refute it.

Watch

Extended reading notes

Core claim

The paper's central claim is that a multimodal instruction-tuned model named MMJamba can train on short visual contexts and then infer on much longer ones. MMJamba uses a frozen CLIP vision encoder, an AnyRes image-tiling scheme that allows at most 4 tiles during training but up to 64 during inference, an AnyFrame video sampler that trains on 8 frames and infers on up to 64, and a Jamba-52B hybrid decoder that interleaves Mamba state-space layers with transformer layers and mixture-of-experts. The authors report top results among the open models they compare with on image benchmarks, gains over open-source video baselines with occasional matches or wins against proprietary models, and an efficiency analysis showing throughput about four times higher than the primary open-source baseline at 4368x4368 resolution, with the gap widening as resolution or frame count rises. They also claim to be the first to train on low-resolution images or low-frame-rate videos while inferring on high-resolution and high-frame-rate inputs.

Load-bearing premise

The load-bearing premise is that a model trained on at most 4 image tiles and 8 video frames will keep working when asked to reason over 64 tiles or 64 frames at inference, because the Mamba layers' recurrent state is expected to generalize to sequence lengths it never saw in training.

Editorial extensions

If this is right

  • If the recipe generalizes, long-context visual pretraining becomes unnecessary for this architecture, cutting the training cost of high-resolution and high-frame-rate models.
  • The efficiency gap over attention-based models should widen as resolution or frame count grows, because attention cost is quadratic while Mamba's recurrence is linear in token count.
  • Visual token pruning and downsampling become less necessary, which the paper argues avoids the information loss that such compression introduces.
  • Deploying the same checkpoint at different resolutions or frame counts becomes a runtime choice between speed and fidelity, without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I infer that the same train-short-infer-long recipe would likely transfer to other long-form modalities, such as audio or long interleaved documents, because the bottleneck the paper removes is architectural rather than visual; the paper does not test this.
  • I infer that the recurrent state's ability to compress visual tokens is the real precondition for the recipe, so a direct probe of hidden-state consistency across 1, 4, 16, and 64 tiles would be a sharper test than benchmark averages; the paper does not report such a probe.
  • I infer that the observed accuracy drops on EgoSchema and VideoMME at 64 frames, which the paper attributes to benchmark characteristics, could also signal a length ceiling in the Mamba state; distinguishing those requires a frame-order perturbation experiment.
  • I infer that the throughput comparison may understate the real-world speedup for long generations, since the paper measures average tokens processed over 1024 generated tokens rather than complete responses.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes MMJamba, a multimodal large language model built on the hybrid transformer-Mamba backbone Jamba, with the stated goal of efficiently handling long visual contexts arising from high-resolution images and high-frame-rate videos. The method uses an AnyRes/AnyFrame visual encoding scheme, trains on short contexts (at most 4 image tiles and 8 video frames), and then performs inference on much longer contexts (up to 2688x2688 resolution or 64 frames). The authors report results on 18 image and video benchmarks, claim state-of-the-art performance relative to open-source and some proprietary models, and report roughly 4x throughput gains over LLaVA-NeXT at high resolution. The paper is empirical: there is no theoretical derivation, and the central claims are measurements of accuracy and efficiency.

Significance. If the empirical claims were sound, the train-short-infer-long recipe would be a practically interesting contribution, since it would decouple training cost from inference-time context length and could make long-context multimodal inference more affordable. The efficiency analysis in Figure 2 also addresses a real bottleneck in high-resolution multimodal processing. However, the paper's central comparative claims rest on baseline numbers in Tables 5 and 7 that are internally inconsistent and implausible, and the paper's own scaling results in Tables 5 and 6 show nontrivial degradations on several benchmarks as context length increases. Without corrected baseline evaluations and a more careful characterization of the length-scaling behavior, the stated conclusions are not supported. No code or model weights are released, which further limits the reproducibility of the empirical claims.

major comments (4)
  1. [Section 4.2, Table 7] Table 7, which reports results 'without AnyRes', shows even more extreme baseline collapses: LLaVA-NeXT-7B scores 6 on MME, 3.3 on MMB-EN, 0.48 on TextVQA and 0.27 on GQA at 1344x1344. These numbers are not informative about the models' capabilities; they indicate that the evaluation harness is feeding inputs outside the operating range of the baseline models. If the intent is to show that LLaVA-NeXT cannot process such inputs without AnyRes, that is a known and unremarkable fact; it does not support the paper's claim that MMJamba outperforms these models under a fair protocol. The authors must clarify exactly what preprocessing was applied to baseline inputs and provide results under each baseline's supported protocol.
  2. [Section 5.2, Table 6] The central claim that the model 'can effectively process long context input' is undermined by the paper's own frame-scaling results. Increasing the number of inference frames from 8 to 64 causes EgoSchema to drop from 58.7 to 52.45, MVBench from 61.0 to 58.8, and VideoMME from 50.1 to 46.3. The text describes these as 'slight declines', but a 6.3-point drop on EgoSchema is substantial. At 64 frames, VideoMME is worse than at 8 frames, which contradicts the narrative that longer visual context improves effectiveness. The authors should either explain these degradations with analysis or revise the claim to state that the train-short-infer-long recipe improves performance on some benchmarks but degrades it on others.
  3. [Section 6, Table 5] The resolution scaling for MMJamba itself is also mixed. While MM-Vet, TextVQA, and MMMU improve from 672 to 1344, other benchmarks decline: MME falls from 1655 to 1640, LLaVA-Wild from 83.9 to 80.0, and VizWiz from 57.6 to 54.3 as resolution increases to 2688. The text attributes these to 'the inherent nature and characteristics of different benchmarks', but no evidence is provided for this explanation. Since the paper's headline claim is that training on short contexts and inferring on long contexts yields better performance, the authors need to quantify how often the recipe helps versus hurts and avoid blanket statements that higher inference resolution 'enhances the performance'.
  4. [Section 6, Figure 2] The efficiency claim of 'about 4 times faster' is presented without sufficient experimental detail. Figure 2 reports first-inference latency and throughput, but the hardware, batch size, tensor-parallel configuration, tokenizer, and generation length are not specified. The text says throughput is 'calculated as average tokens processed when generating 1024 tokens', but no numerical values are given in the text or figure. To support the efficiency claim, the authors should provide a table with exact latency and throughput numbers for each model and configuration, including the 4368x4368 setting, and specify the evaluation environment.
minor comments (6)
  1. [Section 1] There is a typo in the contributions list: 'We conducte experiments' should be 'We conduct experiments'.
  2. [Section 4.1] The evaluation section lists MathVista among the benchmarks, but no MathVista results appear in any table. Either add the results or remove the mention.
  3. [Section 5.1] The video training data description says 'LLaVA-665K' is used, but the same name is used for the image training set. Please clarify whether these are the same or different collections.
  4. [Section 6, Table 5] The table header for Table 5 is misaligned: the column label 'LLaVA' is separated from 'Wild' and 'Vet', which makes the table hard to read. Please fix the header formatting.
  5. [Section 1] The abstract and introduction state that the model is 'the first to be trained on low-resolution images or low-frame-rate videos while being capable of inference on high-resolution images and high-frame-rate videos.' This claim should be qualified: the ability to run at longer contexts does not imply that performance is maintained or improved, as Tables 5 and 6 show declines on several benchmarks.
  6. [Appendix A.2] The video case studies are referenced in the main text but appear only in the appendix. Please add explicit references to Figure 4 in the appropriate section of the main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the paper reports empirical measurements and architectural choices, with no fitted parameter renamed as a prediction and no load-bearing self-citation chain.

full rationale

MMJamba's central claims are empirical: benchmark accuracies in Tables 1-3, 5, and 6 are measured outcomes, and the efficiency gain in Figure 2 is a measured latency/throughput comparison. The train-short-infer-long recipe is a training-protocol design choice, not a quantity derived from or fitted to the evaluation targets; the paper even reports performance declines at the longest inference settings (e.g., EgoSchema dropping from 58.7 at 8 frames to 52.45 at 64 frames in Table 6), which shows the recipe is not assumed into existence. No equations define a target result in terms of an input in a way that forces the conclusion, and no fitted constants are used to produce the reported numbers. The references are external prior works (Jamba, LLaVA-NeXT, Mamba, AnyRes) used as architecture and baseline sources, and no load-bearing self-citation chain was found. The skeptic's concern about implausibly low LLaVA-NeXT scores at high resolutions is a benchmark-protocol validity issue, not circularity, and therefore does not affect this score.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no mathematical derivation and no fitted physical parameters. Its central claims rest on empirical generalization assumptions, primarily that a model trained on short sequences (4 tiles, 8 frames) can be applied to much longer sequences at inference. The suspiciously low baseline numbers in Table 5 suggest an additional unstated assumption about how baselines were evaluated.

assumptions (4)
  • domain assumption Mamba layers trained on sequences up to 4096 tokens will correctly process sequences of 40k+ tokens at inference without additional adaptation.
    The train-short-infer-long recipe relies on this. Section 3.3 presents the recipe; Tables 5 and 6 provide the evidence, which is mixed: several benchmarks decline at high resolution or high frame counts.
  • domain assumption Visual features from AnyRes tiles remain semantically aligned when the number of tiles used at inference (up to 64) greatly exceeds the number used in training (4).
    Each tile is encoded independently and then concatenated, so the LLM must handle a significantly longer sequence and a spatial layout that was not seen during training. The global low-resolution image is meant to preserve layout, but this is not proven. Section 3.2 and Appendix A.1.
  • domain assumption Baseline evaluations use each model's intended inference behavior, so that the reported comparisons are fair.
    Table 5 gives LLaVA-NeXT-7B a TextVQA score of 19.4 at 1344x1344 resolution, while the same paper reports 65.7 at 672x672 in Table 2. This suggests the baseline protocol is not the intended use. Section 6 and Table 5.
  • domain assumption The pretrained Jamba-52B backbone behaves as described in its original paper and is unchanged apart from multimodal training.
    The paper builds directly on the public Jamba-52B model and does not audit its implementation; any quirk in the backbone transfers to MMJamba. Section 3.2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multimodal Instruction Tuning with Hybrid State Space Models." pith.science (2026). https://pith.science/paper/TTBVR45U

@misc{pith2026241108840,
  author       = {Pith},
  title        = {Pith review of: Multimodal Instruction Tuning with Hybrid State Space Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TTBVR45U}},
  note         = {Machine review of arXiv:2411.08840}
}
read the original abstract

Handling lengthy context is crucial for enhancing the recognition and understanding capabilities of multimodal large language models (MLLMs) in applications such as processing high-resolution images or high frame rate videos. The rise in image resolution and frame rate substantially increases computational demands due to the increased number of input tokens. This challenge is further exacerbated by the quadratic complexity with respect to sequence length of the self-attention mechanism. Most prior works either pre-train models with long contexts, overlooking the efficiency problem, or attempt to reduce the context length via downsampling (e.g., identify the key image patches or frames) to decrease the context length, which may result in information loss. To circumvent this issue while keeping the remarkable effectiveness of MLLMs, we propose a novel approach using a hybrid transformer-MAMBA model to efficiently handle long contexts in multimodal applications. Our multimodal model can effectively process long context input exceeding 100k tokens, outperforming existing models across various benchmarks. Remarkably, our model enhances inference efficiency for high-resolution images and high-frame-rate videos by about 4 times compared to current models, with efficiency gains increasing as image resolution or video frames rise. Furthermore, our model is the first to be trained on low-resolution images or low-frame-rate videos while being capable of inference on high-resolution images and high-frame-rate videos, offering flexibility for inference in diverse scenarios.

Figures

Figures reproduced from arXiv: 2411.08840 by the authors.

Figure 1
Figure 1. Overview of MMJAMBA. Image representations of all divided patches are flattened and concatenated. The hybrid state space model consists of interleaved Mamba and Transformer layers. approach can enhance performance, it also results in a quadratic increase in computational cost due to the extended context length. MLLMs for Videos. Several works, such as VideoChat (Li et al., 2023d) and Video-LLaMA (Zhang et al., 2023)… view at source ↗
Figure 2
Figure 2. Efficiency analysis across different models and resolutions. First inference latency is [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Dialogues between the user and MLLMs. Examples on videos are presented in Sec A.2. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative Video Understanding Cases from different perspectives of MMJ [PITH_FULL_IMAGE:figures/full_fig_p017_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 4 canonical work pages

  1. [1]

    More specifically, we dynamically match the optimal aspect ratio from a pre-defined set of aspect ratios

    Our method segments the image into smaller patches, while trying to maintain the integrity of the original image’s aspect ratio. More specifically, we dynamically match the optimal aspect ratio from a pre-defined set of aspect ratios. Due to limited computational resources, we allow a maximum of 4 tiles during training. Consequently, this set includes all...

    work page 2021

  2. [3]

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al

    URL https://openreview.net/forum?id=vvoWPYqZJA. Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. Internlm-xcomposer2-4khd: A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.06512,

    arXiv

  3. [4]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,

    arXiv 2010

  4. [9]

    Video-llava: Learning united visual representation by alignment before projection

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122, 2023a. Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2...

    arXiv

  5. [10]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101,

    arXiv

  6. [11]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255,

    arXiv

  7. [12]

    A survivor in the era of large- scale pretraining: An empirical study of one-stage referring expression comprehension

    Gen Luo, Yiyi Zhou, Jiamu Sun, Xiaoshuai Sun, and Rongrong Ji. A survivor in the era of large- scale pretraining: An empirical study of one-stage referring expression comprehension. IEEE Transactions on Multimedia, 2023a. 13 Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient vision-language instructio...

    arXiv

  8. [14]

    Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty

    URL https://arxiv.org/ abs/2403.09611. Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947–952. IEEE,

    arXiv 2019

Show all 21 references
  1. [15]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al

    URL https://arxiv.org/abs/2403.05530. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805,

    arXiv

  2. [16]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03079,

    arXiv

  3. [17]

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui

    URL https:// arxiv.org/abs/2409.02889. Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video de- scription dataset for bridging video and language. IEEE International Con- ference on Computer Vision and Pattern Recognition (CVPR), June

  4. [19]

    Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities.arXiv preprint arXiv:2308.02490,

    arXiv

  5. [20]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations , pp. 543–553,

    2023

  6. [2016]

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg

    URL https://www.microsoft.com/en-us/research/publication/ msr-vtt-a-large-video-description-dataset-for-bridging-video-and-language/ . Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision–ECC...

    2016

  7. [2017]

    Otterhd: A high-resolution multi-modality model

    Bo Li, Peiyuan Zhang, Jingkang Yang, Yuanhan Zhang, Fanyi Pu, and Ziwei Liu. Otterhd: A high-resolution multi-modality model. arXiv preprint arXiv:2311.04219, 2023a. Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal l...

    2022 arXiv

  8. [2018]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th Euro- pean Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pp. 235–251....

    2016

  9. [2019]

    Chartqa: A bench- mark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Xuan Long Do, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A bench- mark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022 , pp. 2263–2279,

    2022

  10. [2020]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a

    11 Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a. URL https://arxiv.org/abs/ 2306.13394. C...

    arXiv

  11. [2022]

    Cumo: Scaling multimodal llm with co-upcycled mixture-of-experts, 2024a

    Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, and Longyin Wen. Cumo: Scaling multimodal llm with co-upcycled mixture-of-experts, 2024a. URL https://arxiv.org/abs/2405.05949. 12 Junnan Li, Dongxu Li, Silvio Savarese, and Steven Ho...

    arXiv

  12. [2023]

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang

    URL https://arxiv.org/abs/2308.12966. Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhi- hong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for a lite vision-language model. arXiv preprint arXi...

    arXiv

  13. [2024]

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi

    URL https://arxiv.org/abs/2406.07476. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general-purpose vision-language models with instruction tuning. In Thirty-seventh Conference on Ne...

    arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.