Pith. sign in

REVIEW 3 major objections 6 minor 3 cited by

Chimera: Improving Generalist Model with Domain-Specific Experts

T0 review · 3 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fusing frozen domain experts into a generalist vision-language model, via a router and token masking, lifts an 8B model to top scores on two math-reasoning benchmarks.

desk verdict Chimera's engineering is plausible and the GSCM mechanism is new, but the MathVista SOTA claim is compromised by an unreported train/test overlap with the testmini source datasets. read the letter →

arxiv 2412.05983 v3 pith:Z2P7JWTD submitted 2024-12-08 cs.CV

Tianshuo Peng , Mingsheng Li , Jiakang Yuan , Hongbin Zhou , Renqiu Xia , Renrui Zhang , Lei Bai , Song Mao
show 6 more authors
Bin Wang Aojun Zhou Botian Shi Tao Chen Bo Zhang Xiangyu Yue
This is my paper · ORCID
classification cs.CV
keywords largemultimodalmodelsdomain-specificexpertsexpertroutingvisualtokenmaskingmulti-modalmathematicalreasoningstructuralextractionGeneralist-SpecialistCollaboration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that a generalist vision-language model can acquire specialist skills—reading charts, tables, math diagrams, and document layouts—without retraining on private domain data or sacrificing its general abilities. The proposed pipeline, Chimera, keeps domain-expert encoders frozen, projects their features into the language model's input, and uses a small router to decide which expert, if any, should be invoked for a given image. The central empirical claim is that this reaches 64.9 on MathVista and 32.4 on MathVerse, state-of-the-art for models of comparable scale, while matching or approaching dedicated experts on chart, table, and document extraction. If true, this matters because it turns scarce specialist knowledge into a cheap modular upgrade that can be layered onto broadly capable models.

What carries the argument

The load-bearing mechanism is a trio: a lightweight linear router, frozen domain-expert encoders whose outputs are linearly projected into the language model's embedding space, and the Generalist-Specialist Collaboration Masking (GSCM) rule. The router takes the general encoder's classification token and outputs a distribution over 'no expert' plus one entry per specialist, deciding which expert token stream, if any, gets appended to the masked general visual tokens. GSCM is the sampling rule that, during instruction tuning, randomly selects 30% of the general visual tokens without replacement and blocks their attention, forcing the model to use expert features as a supplement. The paper's attention analysis shows that with masking the model attends substantially more to expert tokens, which is the mechanism that is claimed to balance the optimization between the well-aligned general encoder and the specialists.

What would settle it

Run the exact two-stage Chimera recipe on MathVista with the specialist token stream ablated at inference, forcing the router to choose no expert for every image, and compare accuracy with the full model: if the score does not fall well below 64.9, the expert features are not what carries the claimed gain.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that the real obstacle to combining generalists with specialists is not the representational gap itself but an optimization imbalance: a well-aligned general visual encoder lets the model ignore the newly added expert tokens. Chimera counters this with Generalist-Specialist Collaboration Masking (GSCM), which randomly hides 30% of the general visual tokens during visual instruction tuning so the language model must draw on the specialist stream. With a progressive two-stage recipe that first aligns projections on structure-oriented tasks and then instruction-tunes with masking, Chimera turns an existing generalist LMM into a multi-domain model that sets new accuracy marks on MathVista (64.9) and MathVerse (32.4) and nearly matches specialist systems on chart, table, and document structural extraction. The paper further reports that a standard preference-optimization pass raises MathVista accuracy to 68.3.

Load-bearing premise

The load-bearing premise is that frozen specialist encoders still deliver useful domain information after a simple linear projection, and that randomly masking 30% of the general image tokens is enough to force the model to use that information rather than rely on its already-aligned general encoder.

Editorial extensions

If this is right

  • Specialist capability becomes a plug-in: a new domain expert can be added to an existing generalist model by training only a router, projectors, and one instruction-tuning pass, with no need for private domain training data.
  • One model can hold several specialist skills at once: the router selects the right expert per image, so chart, table, math, and document abilities do not have to live in separate single-purpose systems.
  • The gains survive standard alignment training: a direct preference-optimization run on 60K pairs lifts MathVista accuracy from 64.9 to 68.3, so expert-augmented models remain compatible with preference tuning.
  • General capabilities do not collapse: on a general perception benchmark, Chimera stays comparable to its base model while gaining the specialist skills, suggesting the masking constraint does not destroy existing knowledge.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the same recipe should be testable on other frozen visual specialists—medical imaging, satellite imagery, code or diagram understanding—provided their encoders emit dense token features that a linear projector can align; the low training cost makes this an attractive experiment.
  • The paper's ablations suggest the optimal masking ratio and even the value of a given expert depend on the gap between that expert's pretraining task and the target task; a natural extension is to make the mask ratio or the router's decision task-aware rather than fixed at 0.3.
  • Because the router is trained with dataset-level labels and the paper notes that its 'general' category contains mixed-domain images, per-image expert annotation or confidence-based routing could reduce the reported 4.6% router error and further raise end-to-end accuracy.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Chimera, a training pipeline that augments a generalist large multi-modal model (InternVL2) with features from frozen domain-specific expert encoders (StructEqTable, ChartVLM, Math-CLIP, and GOT). A lightweight router selects which expert, if any, to invoke; trainable projectors map expert features into the LLM embedding space; and a proposed Generalist-Specialist Collaboration Masking (GSCM) mechanism randomly masks a fraction of general visual tokens during instruction tuning to encourage reliance on expert tokens. The authors report state-of-the-art results on MathVista (64.9 for Chimera-8B) and MathVerse (32.4), with a DPO post-training variant reaching 68.3 on MathVista, plus strong results on chart/table/document structural extraction benchmarks. The paper includes ablations on the mask ratio, router accuracy, and number of experts, and states that model weights and training/evaluation data will be released.

Significance. If the reported results are trustworthy, the paper would make a useful contribution: a low-cost, scalable recipe for injecting frozen specialist knowledge into generalist LMMs, with public data and weights promised and multiple standard benchmarks used. The GSCM mechanism is simple and the ablation of mask-ratio and expert-set size is informative. However, the headline claims rest on MathVista and MathVerse scores, and the manuscript does not currently establish that those scores are unaffected by overlap between training corpora and the evaluation sets. Until that is resolved, the central claim of state-of-the-art performance is not yet credible. The general idea of fusing frozen expert encoders with masking is plausible, but the empirical validation needs a decontamination analysis.

major comments (3)
  1. [Supp. Table 8 vs. Table 1] The Stage 2 visual instruction tuning data listed in Supplementary Table 8 includes AI2D, ChartQA, PlotQA, DVQA, FigureQA, Geometry3K, GeoQA+, InterGPS, TQA, CLEVR-Math, Super-CLEVR, ScienceQA, IconQA, and MapQA. These are the same source datasets that constitute MathVista's testmini. The paper reports the headline number 64.9 on MathVista testmini (Table 1) without any decontamination analysis, such as exact-duplicate or near-duplicate detection between training samples and evaluation samples. Since every Chimera variant in the ablations is trained on the same data, the internal comparisons cannot disentangle method gains from possible memorization of the test distribution. Please report overlap statistics (image-level and question-level) and, if any overlap exists, re-run the main comparisons on the disjoint subset or on a fully uncontaminated benchmark.
  2. [Supp. Table 11 / Sec. 4.4] The GSCM mask ratio is selected by ablating on the MathVista testmini subset (Supplementary Table 11), which is the same set used for the headline accuracy in Table 1. Tuning a hyperparameter on the evaluation set and then reporting that set's accuracy as the method's performance inflates the apparent gain and makes the specific choice of 0.3 and the resulting 64.9 score partially fitted. The mask-ratio ablation should be performed on a separate validation set, and the final mask ratio should be justified without reference to the testmini numbers.
  3. [Supp. Sec. 9 / Table 1] The DPO post-training constructs 60K preference pairs by sampling 10K problems from MathV360K and generating 16 responses per problem with Chimera. No decontamination is reported with respect to MathVista testmini or MathVerse, yet Chimera† is evaluated on exactly those benchmarks (Table 1 and Table 2). If MathV360K contains questions from MathVista or MathVerse, the DPO gain of 3.4 points could be due to preference optimization on the evaluation distribution rather than to the method itself. Please check for overlap and report results on a clean subset.
minor comments (6)
  1. [Sec. 2] The sentence 'of which the visual content differing differs significantly from natural images' contains a grammatical error; please revise to 'the visual content of which differs significantly from natural images'.
  2. [Table 1] The table header contains the typo 'IntenrVL2'; it should read 'InternVL2'.
  3. [Eq. (2)] The expression 'Sp_i (Se_i (Xv))' would be clearer with explicit function-application notation, e.g., 'Sp_i(Se_i(Xv))', and the index 'i' is used both as an argmax target and as a free index in the set definition; please disambiguate.
  4. [Supp. Fig. 13] The caption in the supplementary text references 'Fig. ??' when discussing chart structural extraction; this undefined reference should be fixed.
  5. [Sec. 3.1] The word 'privides' in 'Eg privides the general visual features' is a typo for 'provides'.
  6. [Sec. 4.4] The router error analysis in Table 7 uses dataset-based labels and the paper itself notes that the 'general' category contains mixed-domain images (last sentence of Sec. 4.4); this is an acknowledged limitation, but it would be useful to quantify how much the 95.4% router accuracy depends on the dataset identity of the source rather than on visual content.

Circularity Check

2 steps flagged · score 6.0 of 10

MathVista SOTA is partly fitted: GSCM mask ratio is tuned on testmini and the training set overlaps MathVista source datasets.

  1. fitted input called prediction [Supplementary Sec. 10, Table 11; main Sec. 4.2, Table 1]
    "Table 11. Ablation results on different visual content domain on the testmini subset of MathVista. ... Based on the above observations, we set the mask ratio to 0.3 in Chimera’s implementation."

    The GSCM mask ratio is selected by comparing accuracies on MathVista testmini (values 0.0, 0.3, 0.5, 1.0 are all evaluated on that split). The same testmini split is then used to report Chimera-8B's headline 64.9 SOTA in Table 1. The reported number is therefore not an independent prediction of the method; it is the outcome of a hyperparameter choice made using the evaluation set itself. The claimed benefit of GSCM is supported only by this same fitted comparison, so the MathVista gain is partly forced by construction.

  2. other [Supplementary Table 8 (Stage 2 training data); main Sec. 4.2, Table 1]
    "Stage 2: General: ... AI2D (GPT4V) [30], AI2D (InternVL [12]), AI2D (Original) [27], MathVision [76], IconQA [47], MapQA [8], ScienceQA [63], ArxivQA [33], TQA [28], CLEVR-Math [21], Super-CLEVR [36] ... Chart: PlotQA [57], ChartX [84], SimChart [81], Chart2Text [25], ChartQA [52], LRV Chart [43], ChartGemma [55], DVQA [23], FigureQA [24], VisText [68] ... Math: ... Geometry3K [46], GeoQA+ [9], InterGPS [46]"

    MathVista testmini is assembled from AI2D, ChartQA, PlotQA, DVQA, FigureQA, Geometry3K, GeoQA+, InterGPS, TQA, CLEVR-Math, Super-CLEVR, ScienceQA, IconQA, and MapQA. Those exact datasets appear in Chimera's Stage 2 visual instruction tuning list, and the paper reports no split-level decontamination. The 64.9 score on MathVista testmini is therefore not an independent evaluation of the expert-integration method: the model was trained on the same question/image sources it is then tested on, so the benchmark score is partly an artifact of training-data overlap rather than a clean prediction. This makes the SOTA claim statistically forced unless a decontamination analysis is supplied.

full rationale

Chimera is an empirical systems paper, so there is no formal derivation chain to be circular about; the relevant question is whether the reported benchmark numbers are independent of the inputs used to produce them. One genuine fitted-input issue exists: the GSCM mask ratio (0.3) is selected by running ablations directly on MathVista testmini (Supp. Table 11), and the same split is then used for the headline 64.9 SOTA in Table 1. That number is therefore not a clean prediction; it is a hyperparameter-selected result on the evaluation set. A second, related non-independence is that Chimera's Stage 2 training list (Supp. Table 8) includes AI2D, ChartQA, PlotQA, DVQA, FigureQA, Geometry3K, GeoQA+, InterGPS, TQA, CLEVR-Math, Super-CLEVR, ScienceQA, IconQA, and MapQA, which are the source datasets of MathVista testmini, and the paper reports no decontamination; without split-level filtering, the MathVista score reflects training-data overlap rather than only the method. The paper does provide some independent support: MathVerse (not in the Stage 2 list) also improves, and the extraction results cover newly constructed Table-SE/Doc-SE benchmarks, though those are author-collected. The many same-team citations (StructEqTable/DocGenome, ChartVLM/ChartX, MAVIS/Math-CLIP, MathVerse) are normal prior-work citations and are not themselves load-bearing: each expert is used as an off-the-shelf encoder and the gains are tested empirically. On balance, the central 'experts help' claim has independent content, but the specific MathVista SOTA is partially fitted, so the paper earns a 6 rather than a 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical entities. GSCM and the router are methodological components, not postulated entities, and they are evaluated inside the paper.

free parameters (3)
  • GSCM mask ratio = 0.3
    Selected via ablation on MathVista testmini (Supplementary Table 11); directly influences GSCM and all reported reasoning results.
  • Expert set size = 3 for reasoning (table, chart, math), 1 for extraction (document)
    Fixed by design; scaling experiments only vary chart expert, not a full search.
  • DPO preference pair count = 60k
    Constructed from 10k sampled MathV360K problems with 16 responses each; not swept, but affects Chimera-dagger scores.
assumptions (3)
  • domain assumption Expert encoder features are linearly projectable into the LMM's input space and retain domain information after projection.
    Sec 3.2 defines expert projectors and concatenates expert tokens with general tokens; if the projection destroyed information, the pipeline would have no benefit.
  • ad hoc to paper Masking 30% of general visual tokens during training forces the model to use expert tokens and improves fusion.
    The GSCM mechanism in Sec 3.3 is the paper's own hypothesis; it is supported only by the paper's ablation and attention visualizations.
  • domain assumption Router training labels based on dataset source are sufficient supervision for expert selection.
    Section 3.1 defines the router with classification loss; Table 7 shows 95.4% accuracy but also errors between general and math categories.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Chimera: Improving Generalist Model with Domain-Specific Experts." pith.science (2026). https://pith.science/paper/Z2P7JWTD

@misc{pith2026241205983,
  author       = {Pith},
  title        = {Pith review of: Chimera: Improving Generalist Model with Domain-Specific Experts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Z2P7JWTD}},
  note         = {Machine review of arXiv:2412.05983}
}
read the original abstract

Recent advancements in Large Multi-modal Models (LMMs) underscore the importance of scaling by increasing image-text paired data, achieving impressive performance on general tasks. Despite their effectiveness in broad applications, generalist models are primarily trained on web-scale datasets dominated by natural images, resulting in the sacrifice of specialized capabilities for domain-specific tasks that require extensive domain prior knowledge. Moreover, directly integrating expert models tailored for specific domains is challenging due to the representational gap and imbalanced optimization between the generalist model and experts. To address these challenges, we introduce Chimera, a scalable and low-cost multi-modal pipeline designed to boost the ability of existing LMMs with domain-specific experts. Specifically, we design a progressive training strategy to integrate features from expert models into the input of a generalist LMM. To address the imbalanced optimization caused by the well-aligned general visual encoder, we introduce a novel Generalist-Specialist Collaboration Masking (GSCM) mechanism. This results in a versatile model that excels across the chart, table, math, and document domains, achieving state-of-the-art performance on multi-modal reasoning and visual content extraction tasks, both of which are challenging tasks for assessing existing LMMs.

Figures

Figures reproduced from arXiv: 2412.05983 by the authors.

Figure 1
Figure 1. Performance comparison of different models on multi [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of our Chimera framework. Chimera uses Generalist-Specialist Collaboration Masking to facilitate the alignment with expert models. During inference, the Router R decides expert invocations based on the visual input, resulting in a versatile model that excels across multiple specialized domains and tasks. reduce generalizability, leading to trade-offs or suboptimal performance across subtasks. The latter app… view at source ↗
Figure 4
Figure 4. Attention distribution on Chimera with (Top) & without [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figures from the paper (14 more)
Figure 3
Figure 3. Figure 3: Comparison of Edit Distance ↓ across different docu￾ment categories on Document Structural Extraction (Doc-SE) task. by 0.092. As for Doc-SE, Chimera leads in most metrics for both English and Chinese documents, showing better over￾all generalization across document ca…
Figure 5
Figure 5. Figure 5: Pre-training tasks of expert models considered by [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Showcase of Table-SE [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]
Figure 7
Figure 7. Figure 7: Showcase of Table-SE. thumbnail images as input for Eg without employing the widely-used Dynamic High Resolution (DHR) tech￾nique [12, 30]. In the Visual Instruction Tuning stage, DHR is introduced, allowing up to six times more visual tokens. At this stage, we apply t…
Figure 8
Figure 8. Figure 8: Output of Chimera-8B on Table Format Transformation. [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: Output of Chimera-8B on Table Format Transformation. [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: Output of Chimera-8B on Table Format Transformation. [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Output of Chimera-8B on Chart Structural Extraction. [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: Output of Chimera-8B on Chart Structural Extraction. [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Output of Chimera-8B on Chart Structural Extraction. [PITH_FULL_IMAGE:figures/full_fig_p021_13.png]
Figure 14
Figure 14. Figure 14: Output of Chimera on Document Context Extraction. [PITH_FULL_IMAGE:figures/full_fig_p022_14.png]
Figure 15
Figure 15. Figure 15: Output of Chimera on Document Context Extraction [PITH_FULL_IMAGE:figures/full_fig_p023_15.png]
Figure 16
Figure 16. Figure 16: Output of Chimera on Document Context Extraction. [PITH_FULL_IMAGE:figures/full_fig_p024_16.png]
Figure 17
Figure 17. Figure 17: Output of Chimera on Document Context Extraction. [PITH_FULL_IMAGE:figures/full_fig_p025_17.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MME-Reasoning: A Comprehensive Benchmark for Logical Reasoning in MLLMs

    cs.AI 2025-05 conditional novelty 6.0 of 10

    A new 1,188-question multimodal benchmark covering deductive, inductive, and abductive reasoning shows that leading MLLMs score around 60% and are especially weak at abductive reasoning.

  2. GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training

    cs.CV 2024-12 conditional novelty 6.0 of 10

    GeoX achieves state-of-the-art accuracy on GeoQA, UniGeo, Geometry3K, and PGPS9K geometry benchmarks using formalized vision-language pre-training with a Generator-and-Sampler Transformer.

  3. Plane Geometry Problem Solving with Multi-modal Reasoning: A Survey

    cs.CV 2025-05 accept novelty 4.0 of 10

    A survey of plane geometry problem solving that classifies methods into an encoder-decoder framework and analyzes hallucination and data leakage in current benchmarks.

Reference graph

Works this paper leans on

92 extracted references · 13 canonical work pages · cited by 3 Pith papers

  1. [1]

    The claude 3 model family: Opus, sonnet, haiku. 6

  2. [2]

    Pixtral 12b

    Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Devendra Chaplot, Jessica Chudnovsky, Saurabh Garg, Theophile Gervet, Soham Ghosh, Am ´elie H ´eliou, Paul Ja- cob, et al. Pixtral 12b. arXiv preprint arXiv:2410.07073 ,

    arXiv

  3. [3]

    Mathqa: Towards interpretable math word problem solving with operation-based formalisms

    Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel- Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. arXiv preprint arXiv:1905.13319, 2019. 1

    arXiv 1905

  4. [4]

    The claude 3 model family: Opus, sonnet, haiku

    Anthropic. The claude 3 model family: Opus, sonnet, haiku. https://www.anthropic.com,, 2024. 1

    2024

  5. [5]

    Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for un- derstanding, localization, text reading, and beyond. arXiv preprint arXiv:2308.12966, 1(2):3, 2023. 7, 8

    arXiv 2023

  6. [6]

    Language models are few-shot learners

    Tom B Brown. Language models are few-shot learners. arXiv preprint ArXiv:2005.14165, 2020. 2

    arXiv 2005

  7. [7]

    Wiki-llava: Hierarchical retrieval-augmented gener- ation for multimodal llms

    Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, and Rita Cuc- chiara. Wiki-llava: Hierarchical retrieval-augmented gener- ation for multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1818–1826, 2024. 1

    2024

  8. [8]

    Mapqa: A dataset for question answering on choropleth maps

    Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler- Lussier, and Ningchuan Xiao. Mapqa: A dataset for question answering on choropleth maps. arXiv preprint arXiv:2211.08545, 2022. 1

    arXiv 2022

Show all 92 references
  1. [9]

    Geoqa: A geometric question answering benchmark towards multimodal numeri- cal reasoning

    Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numeri- cal reasoning. arXiv preprint arXiv:2105.14517, 2021. 1

    2021 arXiv

  2. [10]

    Sharegpt4v: Improving large multi-modal models with better captions

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023. 1, 2

    2023 arXiv

  3. [11]

    Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning

    Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. Ll3da: Visual interactive instruction tuning for omni-3d understand- ing reasoning and planning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  4. [12]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhang- wei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024. 1, 2, 5, 6, 7, 8

    2024 arXiv

  5. [13]

    Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation mod- els and aligning for generic visual-linguistic tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer...

    2024

  6. [14]

    Benchmarking and improv- ing detail image caption

    Hongyuan Dong, Jiawen Li, Bohong Wu, Jiacong Wang, Yuan Zhang, and Haoyuan Guo. Benchmarking and improv- ing detail image caption. arXiv preprint arXiv:2405.19092,

    arXiv

  7. [15]

    Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model

    Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, et al. Internlm-xcomposer2: Mastering free-form text-image composition and compre- hension in vision-language large model. arXiv preprint arXiv:2401.1...

    2024 arXiv

  8. [16]

    Mme: A comprehensive evaluation bench- mark for multimodal large language models

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, et al. Mme: A comprehensive evaluation bench- mark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023. 1

    2023 arXiv

  9. [17]

    G-llava: Solving geometric prob- lem with multi-modal large language model

    Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wan- jun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric prob- lem with multi-modal large language model. arXiv preprint arXiv:2312.11370, 2023. 2, 3, 1

    2023 arXiv

  10. [18]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceed- ings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016. 2

    2016

  11. [19]

    mplug-docowl2: High-resolution compressing for ocr- free multi-page document understanding

    Anwen Hu, Haiyang Xu, Liang Zhang, Jiabo Ye, Ming Yan, Ji Zhang, Qin Jin, Fei Huang, and Jingren Zhou. mplug-docowl2: High-resolution compressing for ocr- free multi-page document understanding. arXiv preprint arXiv:2409.03420, 2024. 1

    2024 arXiv

  12. [20]

    Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency

    Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang, Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.09621, 2025. 2

    2025 arXiv

  13. [21]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognitio...

    2017

  14. [22]

    Billion- scale similarity search with GPUs

    Jeff Johnson, Matthijs Douze, and Herv ´e J ´egou. Billion- scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547, 2019. 2

    2019

  15. [23]

    Dvqa: Understanding data visualizations via ques- tion answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via ques- tion answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656,

  16. [24]

    Fig- ureqa: An annotated figure dataset for visual reasoning

    Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkin- son, ´Akos K´ad´ar, Adam Trischler, and Yoshua Bengio. Fig- ureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017. 1

    2017 arXiv

  17. [25]

    Chart-to-text: A large-scale benchmark for chart sum- marization

    Shankar Kantharaj, Rixie Tiffany Ko Leong, Xiang Lin, Ahmed Masry, Megh Thakkar, Enamul Hoque, and Shafiq Joty. Chart-to-text: A large-scale benchmark for chart sum- marization. arXiv preprint arXiv:2203.06486, 2022. 1

    2022 arXiv

  18. [26]

    Geomverse: A systematic evalu- ation of large models for geometric reasoning.arXiv preprint arXiv:2312.12241, 2023

    Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evalu- ation of large models for geometric reasoning.arXiv preprint arXiv:2312.12241, 2023. 1

    2023 arXiv

  19. [27]

    A diagram is worth a dozen images

    Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part IV 14 , pages 235–

    2016

  20. [28]

    Are you smarter than a sixth grader? textbook question answer- ing for multimodal machine comprehension

    Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answer- ing for multimodal machine comprehension. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern...

    2017

  21. [29]

    Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024

    Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024. 7

    2024

  22. [30]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024. 2, 5, 6, 1

    2024 arXiv

  23. [31]

    Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2

    2022

  24. [32]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In In- ternational conference on machine learning , pages 19730– 19742. PMLR, 2023. 2

    2023

  25. [33]

    Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models

    Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.00231, 2024. 1

    2024 arXiv

  26. [34]

    M3dbench: Let’s instruct large models with multi-modal 3d prompts

    Mingsheng Li, Xin Chen, Chi Zhang, Sijin Chen, Hongyuan Zhu, Fukun Yin, Gang Yu, and Tao Chen. M3dbench: Let’s instruct large models with multi-modal 3d prompts. arXiv preprint arXiv:2312.10763, 2023. 2

    2023 arXiv

  27. [35]

    Omnicorpus: An unified mul- timodal corpus of 10 billion-level images interleaved with text

    Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shen- glong Ye, Zhenjiang Jin, Guanzhou Chen, Yinan He, Zhang- wei Gao, Erfei Cui, et al. Omnicorpus: An unified mul- timodal corpus of 10 billion-level images interleaved with text. arXiv preprint arXiv:2406.08418, 2024. 2

    2024 arXiv

  28. [36]

    Super-clevr: A virtual benchmark to diagnose do- main robustness in visual reasoning

    Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Ko- rtylewski, Wufei Ma, Benjamin Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose do- main robustness in visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2023

  29. [37]

    Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning

    Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Ser- ena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Advances in Neural Information Processing Sys- tems, 35:17612–17625, 2022. 2

    2022

  30. [38]

    Kag- gle - llm science exam

    Will Lifferth, Walter Reade, and Addison Howard. Kag- gle - llm science exam. https : / / kaggle . com / competitions / kaggle - llm - science - exam ,

  31. [39]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceeding...

    2014

  32. [40]

    Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models

    Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023. 1, 2, 6, 7

    2023 arXiv

  33. [41]

    Playground v3: Improving text-to- image alignment with deep-fusion large language models

    Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to- image alignment with deep-fusion large language models. arXiv preprint arXiv:2409.10695, 2024. 1

    2024 arXiv

  34. [42]

    Deplot: One-shot visual language reasoning by plot-to-table trans- lation, 2022

    Fangyu Liu, Julian Martin Eisenschlos, Francesco Piccinno, Syrine Krichene, Chenxi Pang, Kenton Lee, Mandar Joshi, Wenhu Chen, Nigel Collier, and Yasemin Altun. Deplot: One-shot visual language reasoning by plot-to-table trans- lation, 2022. 8

    2022

  35. [43]

    Aligning large multi-modal model with robust instruction tuning

    Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565, 2023. 1

    2023 arXiv

  36. [44]

    Improved baselines with visual instruction tuning

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296–26306, 2024. 3

    2024

  37. [45]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024. 3

    2024

  38. [46]

    Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning

    Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and sym- bolic reasoning. In The 59th Annual Meeting of the Associa- tion for Computational Linguistics (ACL), 2021. 1

    2021

  39. [47]

    Iconqa: A new benchmark for abstract diagram under- standing and visual language reasoning

    Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram under- standing and visual language reasoning. arXiv preprint arXiv:2110.13214, 2021. 1

    arXiv

  40. [48]

    Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathemat- ical reasoning of foundation models in visual contexts.arXiv preprint arXiv:2310.02255, 2023. 2, 5, 3

    arXiv

  41. [49]

    Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning

    Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023. 1

    2023

  42. [50]

    Ovis: Structural em- bedding alignment for multimodal large language model

    Shiyin Lu, Yang Li, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang, and Han-Jia Ye. Ovis: Structural em- bedding alignment for multimodal large language model. arXiv:2405.20797, 2024. 6

    2024 arXiv

  43. [51]

    Omnicaptioner: One captioner to rule them all

    Yiting Lu, Jiakang Yuan, Zhen Li, Shitian Zhao, Qi Qin, Xinyue Li, Le Zhuo, Licheng Wen, Dongyang Liu, Yuewen Cao, et al. Omnicaptioner: One captioner to rule them all. arXiv preprint arXiv:2504.07089, 2025. 2

    2025 arXiv

  44. [52]

    Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question an- swering about charts with visual and logical reasoning.arXiv preprint arXiv:2203.10244, 2022. 5, 1

    2022 arXiv

  45. [53]

    Unichart: A universal vision- language pretrained model for chart comprehension and rea- soning

    Ahmed Masry, Parsa Kavehzadeh, Xuan Long Do, Ena- mul Hoque, and Shafiq Joty. Unichart: A universal vision- language pretrained model for chart comprehension and rea- soning. arXiv preprint arXiv:2305.14761, 2023. 8

    2023 arXiv

  46. [54]

    Chartinstruct: Instruction tuning for chart comprehension and reasoning

    Ahmed Masry, Mehrad Shahmohammadi, Md Rizwan Parvez, Enamul Hoque, and Shafiq Joty. Chartinstruct: Instruction tuning for chart comprehension and reasoning. arXiv preprint arXiv:2403.09028, 2024. 3

    2024 arXiv

  47. [55]

    Chartgemma: Vi- sual instruction-tuning for chart reasoning in the wild

    Ahmed Masry, Megh Thakkar, Aayush Bajaj, Aaryaman Kartha, Enamul Hoque, and Shafiq Joty. Chartgemma: Vi- sual instruction-tuning for chart reasoning in the wild. arXiv preprint arXiv:2407.04172, 2024. 3, 1

    2024 arXiv

  48. [56]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021. 1

    2021

  49. [57]

    Plotqa: Reasoning over scientific plots

    Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 1527–1536, 2020. 5, 1

    2020

  50. [58]

    Orca-math: Unlocking the potential of slms in grade school math, 2024

    Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah. Orca-math: Unlocking the potential of slms in grade school math, 2024. 1

    2024

  51. [59]

    OpenAI. Gpt-4v. https://openai.com/index/ gpt-4v-system-card/ , 2023. 1, 2

    2023

  52. [60]

    Hello gpt-4o

    OpenAI. Hello gpt-4o. https : / / openai . com / index/hello-gpt-4o/, 2024. 1, 2

    2024

  53. [61]

    Im2text: Describing images using 1 million captioned pho- tographs

    Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned pho- tographs. Advances in neural information processing sys- tems, 24, 2011. 2

    2011

  54. [63]

    Scienceqa: A novel resource for question answering on scholarly articles

    Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3):289–301, 2022. 1

    2022

  55. [64]

    Laion-400m: Open dataset of clip-filtered 400 million image-text pairs

    Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. 2

    2021 arXiv

  56. [65]

    Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning

    Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, im- age alt-text dataset for automatic image captioning. In Pro- ceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Paper...

    2018

  57. [66]

    Mome: Mixture of multimodal experts for gen- eralist multimodal large language models

    Leyang Shen, Gongwei Chen, Rui Shao, Weili Guan, and Liqiang Nie. Mome: Mixture of multimodal experts for gen- eralist multimodal large language models. arXiv preprint arXiv:2407.12709, 2024. 2, 3

    2024 arXiv

  58. [67]

    Math- llava: Bootstrapping mathematical reasoning for multimodal large language models

    Wenhao Shi, Zhiqiang Hu, Yi Bin, Junhua Liu, Yang Yang, See-Kiong Ng, Lidong Bing, and Roy Ka-Wei Lee. Math- llava: Bootstrapping mathematical reasoning for multimodal large language models. arXiv preprint arXiv:2406.17294 ,

    arXiv

  59. [68]

    Vistext: A benchmark for semantically rich chart captioning

    Benny J Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semantically rich chart captioning. arXiv preprint arXiv:2307.05356, 2023. 1

    2023 arXiv

  60. [69]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023. 7

    2023 arXiv

  61. [70]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of con- text. arXiv preprint arXiv:2403.05530, 2024. 6

    2024 arXiv

  62. [71]

    Novelseek: When agent becomes the scientist–building closed-loop system from hy- pothesis to verification

    NovelSeek Team, Bo Zhang, Shiyang Feng, Xiangchao Yan, Jiakang Yuan, Zhiyin Yu, Xiaohan He, Songtao Huang, Shaowei Hou, Zheng Nie, et al. Novelseek: When agent becomes the scientist–building closed-loop system from hy- pothesis to verification. arXiv preprint arXiv:2505.16938 ,

    arXiv

  63. [72]

    Cambrian- 1: A fully open, vision-centric exploration of multimodal llms

    Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian- 1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2406.16860, 2024. 1

    2024 arXiv

  64. [73]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023. 2

    2023 arXiv

  65. [74]

    Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models.arXiv preprint arXiv:2307.09288, 2023. 2

    2023 arXiv

  66. [75]

    Mineru: An open-source solution for precise document content extraction

    Bin Wang, Chao Xu, Xiaomeng Zhao, Linke Ouyang, Fan Wu, Zhiyuan Zhao, Rui Xu, Kaiwen Liu, Yuan Qu, Fukai Shang, et al. Mineru: An open-source solution for precise document content extraction. arXiv preprint arXiv:2409.18839, 2024. 2, 3

    2024 arXiv

  67. [76]

    Measuring multimodal mathemat- ical reasoning with math-vision dataset, 2024

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathemat- ical reasoning with math-vision dataset, 2024. 1

    2024

  68. [78]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024. 1

    2024 arXiv

  69. [79]

    Enhancing the reason- ing ability of multimodal large language models via mixed preference optimization

    Weiyun Wang, Zhe Chen, Wenhai Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Jinguo Zhu, Xizhou Zhu, Lewei Lu, Yu Qiao, and Jifeng Dai. Enhancing the reason- ing ability of multimodal large language models via mixed preference optimization. arXiv preprint arXiv:2411.10442,

    arXiv

  70. [80]

    General ocr theory: Towards ocr-2.0 via a unified end-to-end model

    Haoran Wei, Chenglong Liu, Jinyue Chen, Jia Wang, Lingyu Kong, Yanming Xu, Zheng Ge, Liang Zhao, Jianjian Sun, Yuang Peng, et al. General ocr theory: Towards ocr-2.0 via a unified end-to-end model. arXiv preprint arXiv:2409.01704,

    arXiv

  71. [81]

    Structchart: Perception, structuring, reasoning for vi- sual chart understanding

    Renqiu Xia, Bo Zhang, Haoyang Peng, Hancheng Ye, Xi- angchao Yan, Peng Ye, Botian Shi, Yu Qiao, and Junchi Yan. Structchart: Perception, structuring, reasoning for vi- sual chart understanding. arXiv preprint arXiv:2309.11268,

    arXiv

  72. [82]

    Geox: Geometric prob- lem solving through unified formalized vision-language pre- training

    Renqiu Xia, Mingsheng Li, Hancheng Ye, Wenjie Wu, Hongbin Zhou, Jiakang Yuan, Tianshuo Peng, Xinyu Cai, Xiangchao Yan, Bin Wang, et al. Geox: Geometric prob- lem solving through unified formalized vision-language pre- training. arXiv preprint arXiv:2412.11863, 2024. 3, 6

    2024 arXiv

  73. [83]

    Docgenome: An open large- scale scientific document benchmark for training and test- ing multi-modal large language models

    Renqiu Xia, Song Mao, Xiangchao Yan, Hongbin Zhou, Bo Zhang, Haoyang Peng, Jiahao Pi, Daocheng Fu, Wen- jie Wu, Hancheng Ye, et al. Docgenome: An open large- scale scientific document benchmark for training and test- ing multi-modal large language models. arXiv preprint arXiv:...

    2024

  74. [84]

    Chartx & chartvlm: A versatile bench- mark and foundation model for complicated chart reasoning

    Renqiu Xia, Bo Zhang, Hancheng Ye, Xiangchao Yan, Qi Liu, Hongbin Zhou, Zijun Chen, Min Dou, Botian Shi, Junchi Yan, et al. Chartx & chartvlm: A versatile bench- mark and foundation model for complicated chart reasoning. arXiv preprint arXiv:2402.12185, 2024. 1, 2, 3, 5, 8

    2024 arXiv

  75. [85]

    mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration

    Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , ...

    2024

  76. [86]

    Mme-reasoning: A comprehensive benchmark for logical reasoning in mllms

    Jiakang Yuan, Tianshuo Peng, Yilei Jiang, Yiting Lu, Ren- rui Zhang, Kaituo Feng, Chaoyou Fu, Tao Chen, Lei Bai, Bo Zhang, et al. Mme-reasoning: A comprehensive benchmark for logical reasoning in mllms. arXiv preprint arXiv:2505.21327, 2025. 3

    2025 arXiv

  77. [87]

    Mammoth: Building math generalist models through hybrid instruction tuning

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. arXiv preprint arXiv:2309.05653, 2023. 1

    2023 arXiv

  78. [88]

    Sciglm: Training scientific language models with self- reflective instruction annotation and tuning

    Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. Sciglm: Training scientific language models with self- reflective instruction annotation and tuning. arXiv preprint arXiv:2401.07950, 2024. 1

    2024 arXiv

  79. [89]

    Mavis: Mathematical visual in- struction tuning

    Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual in- struction tuning. arXiv preprint arXiv:2407.08739, 2024. 2, 3, 5, 6, 7, 1

    2024 arXiv

  80. [90]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pages 169–186. Sprin...

    2025

  81. [91]

    Llavar: Enhanced visual instruction tuning for text-rich image understanding

    Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv preprint arXiv:2306.17107, 2023. 1

    2023 arXiv

  82. [92]

    Multimodal table understanding

    Mingyu Zheng, Xinwei Feng, Qingyi Si, Qiaoqiao She, Zheng Lin, Wenbin Jiang, and Weiping Wang. Multimodal table understanding. arXiv preprint arXiv:2406.08100 ,

    arXiv

  83. [93]

    Math-puma: Progressive upward multimodal align- ment to enhance mathematical reasoning

    Wenwen Zhuang, Xin Huang, Xiantao Zhang, and Jin Zeng. Math-puma: Progressive upward multimodal align- ment to enhance mathematical reasoning. arXiv preprint arXiv:2408.08640, 2024. 2, 6, 7

    2024 arXiv

  84. [94]

    Mova: Adapting mixture of vision experts to multimodal context

    Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li, and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context. arXiv preprint arXiv:2404.13046, 2024. 2, 3

    2024 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.