Pith. sign in

REVIEW 3 major objections 6 minor 8 cited by

Valley2: Exploring Multimodal Models with Scalable Vision-Language Design

T0 review · 3 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Valley2 claims a 7B model beats Qwen2-VL-7B by 6.9 points on e-commerce VQA and ranks second on OpenCompass among sub-10B models.

desk verdict OpenCompass claim is solid and the model release is useful, but the e-commerce SOTA is built on a tiny unreleased benchmark used for model selection. read the letter →

arxiv 2501.05901 v2 pith:4GY6E7E5 submitted 2025-01-10 cs.CV

Ziheng Wu , Zhenghao Chen , Ruipu Luo , Can Zhang , Yuan Gao , Zhentao He , Xian Wang , Haoran Lin
show 1 more author
Minghui Qiu
This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelvision-languagee-commercebenchmarkConvAdapterEagleModulechain-of-thoughtpost-trainingOpenCompasstokencompression
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Valley2, a 7B-parameter multimodal large language model that accepts text, images, and videos. The authors claim it achieves state-of-the-art accuracy on e-commerce question-answering benchmarks, 79.66 versus 72.76 for Qwen2-VL-7B, and ranks second among models under 10B parameters on the OpenCompass leaderboard with an average score of 67.4. The purpose of the work is to show that careful token compression and a second vision encoder can let a small open model match or exceed much larger systems on practical, real-world tasks. The authors also open-source code and weights so others can verify these results.

What carries the argument

The load-bearing mechanism is the token-budget compromise: a ConvAdapter that halves the spatial grid of vision tokens while preserving channel dimension, a large hidden-size MLP that benefits from that compact representation, and the Eagle Module that adds a second vision encoder's tokens only when needed. This trio keeps the average training sequence length under 1,000 tokens while allowing single-image inference to scale to 1,000-2,000 tokens, matching models that use 4,000-8,000 tokens per image.

What would settle it

Releasing Ecom-VQA and having an independent lab re-run Valley2 alongside Qwen2-VL-7B and InternVL2-8B under identical settings would settle it; if the 79.66 versus 72.76 margin shrinks to near parity or the benchmark leaks training data, the claim loses its basis.

Watch

Extended reading notes

Core claim

Valley2's central claim is that a 7B model can reach competitive general multimodal performance and best-in-class e-commerce understanding without paying the token cost of high-resolution tiling on every input. The design pairs a large two-layer MLP projector with a ConvAdapter that uses a 2x2 convolution to fold four spatial tokens into one, keeping the MLP input dimension at the vision encoder's width instead of quadrupling it as PixelShuffle would. An additional parallel vision encoder, the Eagle Module, supplies extra tokens for extreme aspect ratios and OCR-heavy inputs while remaining constrained to the same token budget during training. On the authors' in-house Ecom-VQA benchmark Valley2 scores 79.66, and on OpenCompass it averages 67.4 across eight benchmarks. The paper further shows that chain-of-thought post-training adds about 1.3 points to the OpenCompass average.

Load-bearing premise

The headline e-commerce result rests on the validity of the in-house Ecom-VQA benchmark, which contains 536 multiple-choice pairs, has not been publicly released, and was used to tune the very hyperparameters the paper then evaluates.

Editorial extensions

If this is right

  • Valley2 reaches state-of-the-art accuracy on e-commerce VQA among open models of comparable size, as measured by the authors' Ecom-VQA benchmark.
  • It ranks second on the OpenCompass leaderboard among models with fewer than 10B parameters, with an average score of 67.4.
  • The ConvAdapter and Eagle Module design keeps average training sequence length under 1,000 tokens, and packing improves training efficiency by a factor of 2.2.
  • Chain-of-thought post-training raises the OpenCompass average from 66.11 to 67.40, showing that systematic reasoning data helps even on general benchmarks.
  • Code and model weights are open-sourced, allowing independent reproduction and application to other domains.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the token-budget compromise generalizes, the same ConvAdapter-style compression could extend to audio inputs or longer video sequences, where token count is the main bottleneck.
  • The Ecom-VQA benchmark's four-way cyclic testing checks answer-position consistency, but an independent evaluation without the authors' hyperparameter tuning on the same data would clarify whether the margin over Qwen2-VL-7B is robust.
  • A testable extension would be to apply Valley2's ConvAdapter to other vision encoders while omitting the Eagle Module, isolating whether the compression or the extra encoder drives the e-commerce gains.
  • The paper's planned omni-model incorporating audio suggests the authors expect the same architectural pattern to transfer across modalities, but that claim is not yet supported by results here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents Valley2, a 7B-parameter multimodal large language model built on Qwen2.5 and SigLIP, with a large-MLP projector combined with a ConvAdapter for token compression, an optional Eagle Module for high-resolution inputs, and a chain-of-thought (CoT) post-training stage. The authors claim state-of-the-art performance on an in-house e-commerce benchmark (Ecom-VQA 79.66 vs. Qwen2-VL-7B 72.76), second place among sub-10B models on the OpenCompass leaderboard (average 67.4), and an average training sequence length under 1,000 tokens. The paper also describes data curation for e-commerce and short-video domains, a packing strategy for training efficiency, and ablations for each architectural component. Code and model weights are released.

Significance. If the claims hold, Valley2 is a competitive and efficient design for practical multimodal applications, and the open-source release allows independent verification of the OpenCompass ranking, which is a concrete strength. The architectural contributions (ConvAdapter, large MLP, Eagle Module) are clearly described and ablated. However, the headline e-commerce SOTA rests on a small, unreleased 536-item benchmark that is also used for model selection, so the e-commerce claim is not currently established to the standard expected for a journal publication. The external OpenCompass result is the more defensible claim and is readily checkable.

major comments (3)
  1. [Section 3.2 and Table 4] The headline Ecom-VQA result is based on only 536 QA pairs, and the benchmark is not publicly released. For 536 independent binary items, the standard error of the difference between two proportions near 0.75 is roughly 2.6 percentage points, so the reported margin of 6.9 points (79.66 vs. 72.76) is not robust to modest contamination or selection effects. Please report confidence intervals (or bootstrap replicates) and provide the benchmark, or at minimum a detailed statistical analysis, to support the claimed margin.
  2. [Section 6.1 and Table 5] The ablation study selects hyperparameters (MLP hidden size, ConvAdapter vs. PixelShuffle, Eagle Module, and annealing) using the same Ecom-VQA and OpenCompass benchmarks that are later reported as headline results. This makes the reported Ecom-VQA score an optimistically biased estimate of performance on new data. The authors should either use a held-out validation set for model selection or report selection-adjusted estimates, such as nested cross-validation or a clear statement of how many configurations were tried.
  3. [Section 3.2 and Table 1] The manuscript states that the proposed benchmarks are used to track and assess the development of capabilities, while Table 1 lists Inhouse Ecom-Data in every training category, including General, Reasoning, OCR, and OneVision tasks. The paper does not demonstrate that Ecom-VQA questions and images are disjoint from the in-house training data, so the comparison against Qwen2-VL-7B could be contaminated by training on the same distribution or even the same instances. Please provide a detailed overlap/leakage analysis and clarify the provenance of the benchmark items relative to the training corpus.
minor comments (6)
  1. [Section 5.1 and Table 3] The average score for Valley2 is listed as 67.40, but the table column header includes an unused 'A VG' column; please clarify how the average is computed and whether it matches the OpenCompass aggregate.
  2. [Section 6.4 and Table 8] The CoT ablation row reports an OpenCompass improvement but leaves the E-VQA cell blank; please state explicitly whether CoT was evaluated on Ecom-VQA and, if not, why.
  3. [Section 6.1] The text says the intermediate MLP dimension (Dh) is set to '1/5 of the LLM vocabulary size', but Table 5 reports a default MLP hidden size of 3584, which equals the Qwen2.5-7B hidden dimension rather than one-fifth of its vocabulary size. This appears to be a typo and should be corrected.
  4. [Section 4.2 and 6.3] The packing ablation reports a 2.2x training speedup, while the text in Section 6.3 says training time is reduced 'by a factor of approximately 2–3'; please reconcile these numbers for consistency.
  5. [References] Several references contain 'arXiv preprint arXiv:unknown' (e.g., [24], [46], [73], [87]); these entries are incomplete and should be updated with the correct identifiers or venues.
  6. [Section 8] The 'Coming Soon' section listing future work is unusual for a journal article; consider removing it or moving it to a separate document to keep the paper focused.

Circularity Check

1 steps flagged · score 3.0 of 10

No formal derivation-level circularity; the headline Ecom-VQA score (79.66) is the same ablation-selected value used to choose the architecture, while OpenCompass provides independent external support.

  1. fitted input called prediction [Section 6.1, Table 5; Section 5.2, Table 4; Abstract]
    "To evaluate the contribution of each component, we conducted a series of ablation studies, with results summarized in Table 5. ... Ecom-VQA and OpenCompass evaluate the performance of the model after stage-2 training. ... Exp 6 ✓ ✓ ✓ 9× 37.45 79.66 66.1"

    The abstract and Table 4 report Valley-7B's 79.66 on Ecom-VQA as the state-of-the-art result. Table 5 shows that the same 79.66 is the E-VQA score of the configuration selected by the ablation (Eagle Module enabled, MLP hidden size 9x, ConvAdapter, Qwen2.5). The hyperparameters were therefore chosen using the same 536-pair in-house benchmark on which the headline number is reported, making that number a selection score rather than an independent prediction. The paper provides no separate holdout split, no error bars, and no leakage analysis, so the 6.9-point margin over Qwen2-VL-7B is not protected against selection effects. The OpenCompass 67.40 score is an external leaderboard result and remains independent, which keeps the circularity partial and empirical rather than definitional.

full rationale

This paper has no formal mathematical derivation chain, so there are no equations whose outputs are identical to their inputs by construction. The self-citations to Valley [54] are contextual and not load-bearing; the Eagle Module [66] is an external citation. The independent OpenCompass evaluation (average 67.4, rank 2 under 10B) is externally checkable from released weights, and the packing and CoT ablations on OpenCompass retain independent content. The demonstrated circular element is the Ecom-VQA headline: Section 6.1's Table 5 uses E-VQA to select the final architecture, and the same 79.66 value is then reported as the SOTA in Table 4 and the abstract. This is a model-selection-on-the-evaluation-benchmark problem, not a definitional identification, so it warrants a moderate score rather than 6+. Additionally, the paper does not establish that Ecom-VQA is disjoint from the in-house Ecom training data listed in Table 1, especially given Section 3.2's statement that the proposed benchmarks are used to track and assess capability development; I treat that as an evaluation-validity risk rather than a separately proven circular step.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on benchmark validity and data representativeness rather than mathematical axioms. No new physical entities are introduced; architectural modules are either borrowed or simple variants.

free parameters (4)
  • MLP hidden size (Dh) = 3584 (1/5 of LLM vocabulary size)
    Selected via ablation on the in-house Ecom-Caption, Ecom-VQA, and OpenCompass benchmarks; affects all final scores.
  • Tiling max slices = 9
    Hand-chosen cap on single-image tiles to balance token count and detail; affects accuracy on high-resolution inputs.
  • ConvAdapter downsampling factor = 2 (2x2 convolution)
    Design choice replacing PixelShuffle to reduce token count while preserving dimension; central to the projector's efficiency.
  • CoT mixing ratio = 1:1 CoT to non-CoT samples
    Set in Stage-3 to balance reasoning mode and direct response mode; affects final OpenCompass average.
assumptions (3)
  • domain assumption In-house Ecom data is representative of e-commerce understanding and similar to the Ecom-VQA benchmark distribution.
    Invoked in Section 3.2 where the benchmark is constructed from the same domain; underpins the validity of the 79.66 SOTA claim.
  • domain assumption The OpenCompass benchmark suite is a valid external measure of general multimodal capability.
    Invoked in Section 5.1; the paper relies on this to support the second-place ranking claim.
  • domain assumption Chain-of-thought post-training on LLaVA-CoT data transfers to general multimodal reasoning.
    Invoked in Section 3.3 and 6.4; the CoT Stage-3 gain on OpenCompass assumes this transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Valley2: Exploring Multimodal Models with Scalable Vision-Language Design." pith.science (2026). https://pith.science/paper/4GY6E7E5

@misc{pith2026250105901,
  author       = {Pith},
  title        = {Pith review of: Valley2: Exploring Multimodal Models with Scalable Vision-Language Design},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4GY6E7E5}},
  note         = {Machine review of arXiv:2501.05901}
}
read the original abstract

Recently, vision-language models have made remarkable progress, demonstrating outstanding capabilities in various tasks such as image captioning and video understanding. We introduce Valley2, a novel multimodal large language model designed to enhance performance across all domains and extend the boundaries of practical applications in e-commerce and short video scenarios. Notably, Valley2 achieves state-of-the-art (SOTA) performance on e-commerce benchmarks, surpassing open-source models of similar size by a large margin (79.66 vs. 72.76). Additionally, Valley2 ranks second on the OpenCompass leaderboard among models with fewer than 10B parameters, with an impressive average score of 67.4. The code and model weights are open-sourced at https://github.com/bytedance/Valley.

Figures

Figures reproduced from arXiv: 2501.05901 by the authors.

Figure 1
Figure 1. Overview of Valley The structure of Valley2 is shown in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The data distribution of Ecom-VQA Bench To robustly address these benchmarks and validate our model’s capabilities, we build a large-scale training corpus consisting of 2M multimodal alignment data, 5M knowledge-injected instances (covering images, videos, OCR tasks, and localization challenges), and 1M high-quality instruction data. These datasets are evenly distributed across training phases, with the proposed ben… view at source ↗
Figure 3
Figure 3. Comparison of Problem-Solving Approaches With and Without Chain-of-Thought Training [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Training Stages Stage-1 Stage-1.5 Stage-2 Stage-3 (CoT) Resolution 384 384×{{1×1}, · · · , {3×3}} 384×{{1×1}, · · · , {3×3}} 384×{{1×1}, · · · , {3×3}} #Tokens Max 196×10 + (EAGLE) Max 196×10 + (EAGLE) Max 196×10 + (EAGLE) Max 196×10 + (EAGLE) Dataset 7.5M 8M 11.5M 0.2…
Figure 5
Figure 5. Figure 5: Comparison of token count distributions per sample before and after packing. [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Data of Ecom-VQA Benchmark 18 [PITH_FULL_IMAGE:figures/full_fig_p018_6.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VADER: Adaptive Debiasing for Hallucination Mitigation in Video Large Language Models

    cs.CV 2026-08 conditional novelty 7.0 of 10

    A training-free decoding framework that adaptively reweights attention toward video tokens and erases key visual evidence per frame to suppress hallucinated predictions, achieving 72.60% accuracy on EventHallusion wit...

  2. R1-SyntheticVL: Is Synthetic Data from Generative Models Ready for Multimodal Large Language Model?

    cs.LG 2026-02 conditional novelty 6.0 of 10

    A collective adversarial data-synthesis pipeline produces 20K synthetic multimodal training examples whose GRPO-trained 7B model beats several listed open-source MLLMs on reasoning benchmarks.

  3. VFaith: Do Large Multimodal Models Really Reason on Seen Images Rather than Previous Memories?

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A new benchmark with edited-image question pairs shows that multimodal reasoning models lose accuracy when visual cues change, suggesting their reasoning is often not faithfully tied to the image.

  4. GThinker: Towards General Multimodal Reasoning via Cue-Guided Rethinking

    cs.CV 2025-06 conditional novelty 6.0 of 10

    A cue-tagging and rethinking training recipe lifts a 7B multimodal model to 81.5% on M3CoT, though the gain may reflect training on the same benchmark.

  5. R1-ShareVL: Incentivizing Reasoning Capability of Multimodal Large Language Models via Share-GRPO

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Share-GRPO creates paraphrased and visually augmented versions of reasoning questions and shares answers and reward signals across versions, improving multimodal reasoning without cold-start SFT.

  6. Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmark

    cs.CV 2025-04 conditional novelty 6.0 of 10

    Video-MMLU: a 1,065-video lecture benchmark where most AI video models score 10-50%, but text-only models answer 40% of quiz questions without video.

  7. Interpretable Open-Vocabulary Referring Object Detection with Reverse Contrast Attention

    cs.CV 2025-07 reject novelty 5.0 of 10

    A training-free attention reweighting method, Reverse Contrast Attention, is claimed to improve referring object detection in 11 of 15 VLMs, but the custom FitAP metric ranks boxes by IoU with ground truth, which make...

  8. Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models

    cs.AI 2025-04 conditional novelty 4.0 of 10

    A structured survey of reinforcement-learning-based reasoning methods for multimodal large language models, including a taxonomy, reward design review, benchmark tables, and open challenges.

Reference graph

Works this paper leans on

92 extracted references · 32 canonical work pages · cited by 8 Pith papers

  1. [1]

    Tallyqa: Answering complex counting questions

    Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. arXiv preprint arXiv:1810.12440, 2019. 10

    arXiv 2019

  2. [2]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Math-llava: Bootstrapping mathematical reasoning for multimodal large language models

    Yi Bin et al. Math-llava: Bootstrapping mathematical reasoning for multimodal large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024 , 2024. URL https://aclanthology.org/2024.findings-emnlp.268.pdf

    2024

  4. [4]

    Scene text visual question answering

    Ali Furkan Biten, Ruben Tito, Andres Mafla Mafla, Lluis Gomez, Marcal Rusinol, Ernest Valveny, CV Jawahar, Minesh Mathew, Yash Zhu, Michael Stamatatos, et al. Scene text visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4291–4301, 2019

    2019

  5. [5]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877–1901, 2020

    1901

  6. [6]

    Coyo-700m: Image-text pair dataset

    Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Sae- hoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/kakaobrain/ coyo-dataset, 2022

    2022

  7. [7]

    Allava: Harnessing gpt4v- synthesized data for a lite vision-language model

    Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v- synthesized data for a lite vision-language model. arXiv preprint arXiv:2402.11684, 2024

    arXiv 2024

  8. [8]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023

    arXiv 2023

Show all 92 references
  1. [9]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    2024 arXiv

  2. [10]

    Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling

    Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024

    2024 arXiv

  3. [11]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences , 67(12):220101, 2024

    2024

  4. [12]

    Opencompass: A universal evaluation platform for foundation models

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompass, 2023

    2023

  5. [13]

    Patch n’ pack: Navit, a vision transformer for any aspect ratio and resolution, 2023

    Mostafa Dehghani, Basil Mustafa, Josip Djolonga, Jonathan Heek, Matthias Minderer, Mathilde Caron, Andreas Steiner, Joan Puigcerver, Robert Geirhos, Ibrahim Alabdulmohsin, Avital Oliver, Piotr Padlewski, Alexey Gritsenko, Mario Luˇci´c, and Neil Houlsby. Patch n’ pack: Navit, ...

    2023

  6. [14]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models. arXiv preprint arXiv:2411.14432, 2024

    2024 arXiv

  7. [15]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  8. [16]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Dusty Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 6904–6913, 2017. 11

    2017

  9. [18]

    Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410.18558, 2024. URL http...

    arXiv

  10. [19]

    Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Magpie: Alignment data synthesis from scratch by prompting aligned llms with nothing. arXiv preprint arXiv:2406.08464, 2024. URL https://arxiv....

    2024 arXiv

  11. [20]

    Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large vision-language models

    Tianrui Guan, Fuxiao Liu, Xiyang Wu, Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang, Lichang Chen, Furong Huang, Yaser Yacoob, Dinesh Manocha, and Tianyi Zhou. Hallusion- bench: An advanced diagnostic suite for entangled language hallucination and visual illusion in large visi...

    2024

  12. [21]

    Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images

    Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an lmm perceiving any aspect ratio and high- resolution images. In European Conference on Computer Vision , pages 390–406. Springer, 2025

    2025

  13. [22]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qixing Li, Anne Stangl, Anhong Guo, Chuanqi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3608–...

    2018

  14. [23]

    Minicpm: Unveiling the potential of small language models with scalable training strategies

    Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06395, 2024

    2024 arXiv

  15. [24]

    et al. Hudson. Gqa: A new dataset for real-world visual reasoning and compositional question answering. arXiv preprint arXiv:unknown, 2019

    2019

  16. [25]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognitio...

    2017

  17. [26]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In European conference on computer vision , pages 235–251. Springer, 2016

    2016

  18. [27]

    The hateful memes challenge: Detecting hate speech in multimodal memes

    Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. In Advances in Neural Information Processing Systems , pages 2611–2624, 2020

    2020

  19. [28]

    G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park. Ocr-free document understanding transformer. In European Conference on Computer Vision, 2022

    2022

  20. [29]

    Shamma, Michael S

    Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In...

    2017

  21. [30]

    Vqa-rad: Visual question answering dataset for radiology

    Jason J Lau, Swarnajay Gayen, Zhi Huang, Siqi Liu, and Daniel L Rubin. Vqa-rad: Visual question answering dataset for radiology. In Medical Imaging 2018: Imaging Informatics for Healthcare, Research, and Applications, volume 10579, page 105790V . SPIE, 2018. 12

    2018

  22. [32]

    Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning

    Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2023

  23. [33]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer, 2014

    2014

  24. [34]

    Clevr-math: A dataset for compositional language, visual and mathematical reasoning

    Adam Dahlgren Lindström and Savitha Sam Abraham. Clevr-math: A dataset for compositional language, visual and mathematical reasoning. arXiv preprint arXiv:2208.05358, 2022

    2022 arXiv

  25. [35]

    Visual spatial reasoning

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. arXiv preprint arXiv:2205.00363, 2022

    2022 arXiv

  26. [36]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu et al. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2408.03326, 2024. URL https://fuxiaoliu.github.io/LRV/

    2024 arXiv

  27. [38]

    Llava: Large language and vision assistant

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Llava: Large language and vision assistant. arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  28. [39]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024. URL https: //llava-vl.github.io/blog/2024-01-30-llava-next/

    2024

  29. [40]

    Visual instruction tuning.Advances in neural information processing systems , 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems , 36, 2024

    2024

  30. [41]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2405.07895, 2024

    2024 arXiv

  31. [42]

    Kangaroo: A powerful video-language model supporting long-context video input

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542, 2024

    2024 arXiv

  32. [43]

    Yuan Liu, Le Tian, Xiao Zhou, Xinyu Gao, Kavio Yu, Yang Yu, and Jie Zhou. Points1. 5: Build- ing a vision-language model towards real world applications. arXiv preprint arXiv:2412.08443, 2024

    2024 arXiv

  33. [44]

    Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216–233. Springer, 2025

    2025

  34. [45]

    Ocrbench: on the hidden mystery of ocr in large multimodal models

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences , 67(12), December 2024. ISSN 1869-1919. doi:...

    2024 doi

  35. [46]

    et al. Lu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:unknown, 2023

    2023

  36. [47]

    Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action

    Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023. 13

    2023 arXiv

  37. [48]

    Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning.arXiv preprint arXiv:2109.05014, 2021

    Pan Lu, Hongming Zhang, Ziyu Jiang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning.arXiv preprint arXiv:2109.05014, 2021

    2021 arXiv

  38. [49]

    Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610, 2022

    2022 arXiv

  39. [50]

    Scienceqa: A challenging dataset for multi-modal reasoning

    Pan Lu, Hongming Zhang, Ziyu Jiang, and Song-Chun Zhu. Scienceqa: A challenging dataset for multi-modal reasoning. In Advances in Neural Information Processing Systems , 2022

    2022

  40. [51]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. In International Conference on Learning Representat...

    2024

  41. [52]

    Ovis: Structural embedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural embedding alignment for multimodal large language model. arXiv preprint arXiv:2405.20797, 2024

    2024 arXiv

  42. [53]

    Bluelm-v-3b: Algorithm and system co-design for multimodal large language models on mobile devices

    Xudong Lu, Yinghao Chen, Cheng Chen, Hui Tan, Boheng Chen, Yina Xie, Rui Hu, Guanxin Tan, Renshou Wu, Yan Hu, et al. Bluelm-v-3b: Algorithm and system co-design for multimodal large language models on mobile devices. arXiv preprint arXiv:2411.10640, 2024

    2024 arXiv

  43. [54]

    Valley: Video assistant with large language model enhanced ability

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207, 2023

    2023 arXiv

  44. [55]

    Ok-vqa: A benchmark for visual question answering using external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A benchmark for visual question answering using external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3195–3204, 2019

    2019

  45. [56]

    Ok-vqa: A benchmark for visual question answering using external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A benchmark for visual question answering using external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3195–3204, 2021

    2021

  46. [57]

    Marti and H

    U.-V . Marti and H. Bunke. The iam database: An english sentence database for offline handwriting recognition. In International Journal on Document Analysis and Recognition , pages 199–210, 2002

    2002

  47. [58]

    ChartQA: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In Findings of the Association for Computational Linguistics: ACL 2022 , pages 2263–2279, Dublin, Ireland, May

    2022

  48. [59]

    Mishra, K

    A. Mishra, K. Alahari, and C. V . Jawahar. Scene text recognition using higher order language priors. In Proceedings of the British Machine Vision Conference , pages 1–11, 2012

    2012

  49. [60]

    Icdar 2019 crohme + tfd: Competition on recognition of handwritten mathematical expressions and typeset formula detection

    Harold Mouchère, Richard Zanibbi, Utpal Garain, and Christian Viard-Gaudin. Icdar 2019 crohme + tfd: Competition on recognition of handwritten mathematical expressions and typeset formula detection. In 2019 International Conference on Document Analysis and Recognition (ICDAR),...

    2019

  50. [61]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:277...

    2022

  51. [62]

    Image textualization: An automatic framework for creating accurate and detailed image descrip- tions

    Renjie Pi, Jian Zhang, Shunian Chen, Benyou Wang, Song-Chun Zhu, and Hongming Zhang. Image textualization: An automatic framework for creating accurate and detailed image descrip- tions. arXiv preprint arXiv:2406.07502, 2024. 14

    2024 arXiv

  52. [63]

    Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models

    Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision...

    2015

  53. [64]

    Improving language understanding by generative pre-training

    Alec Radford. Improving language understanding by generative pre-training. 2018

    2018

  54. [65]

    Aritra Roy Chowdhury, Arindam Mitra, Bhuwan Dhingra, and William W. Cohen. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. InProceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Linguis...

    2019

  55. [66]

    Eagle: Exploring the design space for multimodal llms with mixture of encoders

    Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, Bryan Catanzaro, Andrew Tao, Jan Kautz, Zhiding Yu, and Guilin Liu. Eagle: Exploring the design space for multimodal llms with mixture of ...

    2024 arXiv

  56. [67]

    Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network

    Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE conference on compute...

    2016

  57. [68]

    Textcaps: a dataset for image captioning with reading comprehension

    Nikita Sidorov, Amanpreet Singh, Stanislaw Antol, Dhruv Batra, Devi Parikh, Marcus Rohrbach, Ronghang Hu, Yuting Wang, and Alexander Schwing. Textcaps: a dataset for image captioning with reading comprehension. arXiv preprint arXiv:2003.12462, 2020

    2003 arXiv

  58. [69]

    Singh, V

    A. Singh, V . Natarajan, D. Batra, D. Parikh, S. Lee, and R. Krishna. Towards vqa models that can read. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019

    2019

  59. [70]

    Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text

    Amanpreet Singh, Vivek Natarajan, Dhruv Batra, Devi Parikh, Stefan Lee, and Ranjay Kr- ishna. Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8802–8812, 2021

    2021

  60. [71]

    Generative multimodal models are in-context learners

    Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Zhengxiong Luo, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. 2023

    2023

  61. [72]

    Bluelm: An open multilingual 7b language model

    BlueLM Team. Bluelm: An open multilingual 7b language model. https://github.com/ vivo-ai-lab/BlueLM, 2023

    2023

  62. [73]

    G-llava: Solving geometric problem with multi-modal large language model

    Unknown. G-llava: Solving geometric problem with multi-modal large language model. arXiv preprint arXiv:unknown, 2024

    2024

  63. [74]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  64. [75]

    Enhancing the reasoning ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reasoning ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442, 2024

    2024 arXiv

  65. [76]

    Llava-cot: Let vision language models reason step-by-step, 2024

    Guowei Xu, Peng Jin, Hao Li, Yibing Song, Lichao Sun, and Li Yuan. Llava-cot: Let vision language models reason step-by-step, 2024. URL https://arxiv.org/abs/2411.10440

    2024 arXiv

  66. [77]

    Vision-flan: Scaling human-labeled tasks in visual instruction tuning

    Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. arXiv preprint arXiv:2402.11690, 2024. 15

    2024 arXiv

  67. [78]

    xgen-mm(blip-3): A family of open large multimodal models

    Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, Shrikant Kendre, Jieyu Zhang, Can Qin, Shu Zhang, Chia-Chih Chen, Ning Yu, Juntao Tan, Tulika Manoj Awalgaonkar, Shelby Heinecke, Huan Wang, Yejin C...

    2024

  68. [79]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  69. [80]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    2024 arXiv

  70. [81]

    Modeling context in referring expressions

    Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Proceedings of the European Conference on Computer Vision (ECCV), pages 69–85. Springer, 2016

    2016

  71. [82]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In International conference on machine learning . PMLR, 2024

    2024

  72. [83]

    Syntax-aware network for handwritten mathematical expression recognition

    Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9816–9825, 2022

    2022

  73. [84]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: A m...

    2024

  74. [85]

    Sigmoid loss for language image pre-training, 2023

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, 2023

    2023

  75. [86]

    Raven: A dataset for relational and analogical visual reasoning

    Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5317–5327, 2019

    2019

  76. [87]

    Unimath: A foundational and multimodal mathematical reasoner

    Haipeng Zhang et al. Unimath: A foundational and multimodal mathematical reasoner. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ,

    2023

  77. [88]

    Infinity-instruct: Scaling multimodal performance with large-scale and high-quality instruction data

    Haipeng Zhang et al. Infinity-instruct: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2408.07089 , 2024. URL https: //arxiv.org/abs/2408.07089

    2024 arXiv

  78. [89]

    Video-llama: An instruction-tuned audio-visual language model for video understanding

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  79. [90]

    Mavis: Mathematical visual instruction tuning with an automatic data engine

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, Chunyuan Li, and Hongsheng Li. Mavis: Mathematical visual instruction tuning with an automatic data engine. arXiv preprint a...

    2024 arXiv

  80. [91]

    Improve vision language model chain-of-thought reasoning

    Ruohong Zhang, Bowen Zhang, Yanghao Li, Haotian Zhang, Zhiqing Sun, Zhe Gan, Yinfei Yang, Ruoming Pang, and Yiming Yang. Improve vision language model chain-of-thought reasoning. arXiv preprint arXiv:2410.16198, 2024

    2024 arXiv

  81. [92]

    Multimodal C4: An open, billion-scale corpus of images interleaved with text

    Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal C4: An open, billion-scale corpus of images interleaved with text. arXiv preprint arXiv:2304.06939, 2023. 16

    2023 arXiv

  82. [93]

    Visual7w: Grounded question answering in images

    Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4995–5004, 2016. 17 A Detail of Ecom-VQA benchmark Figure 6: Data of Ecom-VQA Ben...

    2016

  83. [2022]

    doi: 10.18653/v1/2022.findings-acl.177

    Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.177. URL https://aclanthology.org/2022.findings-acl.177

    2022 doi

  84. [2023]

    URL https://aclanthology.org/2023.emnlp-main.440.pdf

    2023

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.