Pith. sign in

REVIEW 4 major objections 4 minor 42 references

Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation

T0 review · 4 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that fixing the visual input and varying the task across QA, captioning, question generation, and verification separates genuine MLLM understanding from memorized benchmark answers.

desk verdict Worth a serious look, but its own Table 2 undercuts the headline claim that cross-task sharpness detects contamination. read the letter →

arxiv 2506.07202 v1 pith:DSGNU7YX submitted 2025-06-08 cs.AI

Ming Liu , Wensheng Zhang This is my paper
classification cs.AI
keywords multimodallargelanguagemodelsdatacontaminationdynamicevaluationtaskperturbationcross-taskabilityvectortask-spacesharpnessvisualquestionansweringbenchmarkleakage
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that benchmark scores of multimodal large language models can hide data contamination, and it offers a way to see through the hiding: hold the image or video fixed and vary only the task. A model is evaluated on question answering, captioning, question generation, and answer verification for the same visual input, producing a cross-task ability vector. The paper claims that contaminated or overfit models show sharp, spiky profiles—high on the memorized task, low on the others—while genuinely generalizing models have flatter profiles. This matters because static single-task benchmarks cannot tell a memorized answer from real understanding, so high scores may be inflated. The authors demonstrate the idea by fine-tuning models on test-set QA data, which spikes QA accuracy while hurting or not helping other tasks, and they interpret the resulting performance range as task-space sharpness.

What carries the argument

The central object is the fixed-input multi-task evaluation pipeline: the same image or video is presented with four different task prompts (T0 visual QA, T1 captioning, T2 question generation, T3 answer verification), and the four resulting scores form a model's ability vector. The load-bearing identity is task-space sharpness, measured as the range and standard deviation of those scores, which the paper maps onto the flat-minima versus sharp-minima distinction from loss-landscape theory. A reasoning MLLM judge scores the open-ended captioning and question-generation outputs using rubrics, and a PEFT fine-tune on test QA data serves as the controlled contamination simulator.

What would settle it

Take a model trained on clean data but with an intentionally narrow skill profile—say, strong at captioning and weak at verification—and run the four-task battery on it. If its cross-task range and standard deviation are as large as a model that memorized QA test answers, then the sharpness proxy alone cannot separate inherent task imbalance from contamination.

Watch

Extended reading notes

Core claim

The central claim is that perturbing the task, rather than the input, reveals whether an MLLM's performance is genuine generalization or narrow memorization. For each visual input, the framework evaluates the model on four tasks—visual QA, captioning, question generation, and answer verification—and measures the spread of scores (range and standard deviation) as a proxy for sharpness of the model's loss minimum. A model that memorized a task sits in a sharp minimum: it scores high on that task and noticeably lower on the related tasks sharing the same image. A model with a robust internal representation occupies a flatter region and degrades gracefully. Applying this to 12 image and 11 video MLLMs across three benchmarks, the paper reports that reasoning-oriented models tend to show flatter profiles, and it uses a PEFT fine-tune on test QA data as a controlled contamination experiment to show task-specific sharpening.

Load-bearing premise

The load-bearing premise is that the spread of a model's scores across the four tasks (range and standard deviation) reflects contamination or overfitting rather than genuine differences in task difficulty or in the skills each model was built to have.

Editorial extensions

If this is right

  • If task-space sharpness tracks contamination, benchmark leaders with spiky ability vectors are less trustworthy than their single-task accuracy suggests.
  • Evaluating a model across related tasks for the same input can be added to existing benchmarks like MME, RealWorldQA, and CVRR-ES without new test data, only new prompts and scoring.
  • Reasoning-oriented post-training that generalizes across tasks would be expected to flatten the ability vector, making cross-task range a cheap monitoring signal during model development.
  • Worst-task risk and inter-task distance give developers a specific target: the weakest task for a given image indicates where understanding is missing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same fixed-input task battery could be applied to other modalities such as audio or 3D scenes, where contamination is equally hard to detect, since the method does not depend on image-specific augmentation.
  • Beyond the paper: combining task perturbation with input perturbation—for example, asking for a caption of an image with a deliberately added distractor—would test whether the two failure modes are independent or compound.
  • Beyond the paper: the authors' own 7B PEFT result shows the range can shrink after contamination, so a user of this framework would want to look at per-task drops rather than relying on aggregate sharpness alone.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes a 'dynamic task perturbation' evaluation framework for multimodal large language models (MLLMs). Instead of perturbing the visual or textual input, the framework holds the input fixed and evaluates each model on four tasks derived from the same visual content: QA (T0), captioning (T1), question generation (T2), and answer verification (T3). A reasoning MLLM judge is used to score the open-ended outputs of T1 and T2. The authors define cross-task 'ability vectors' and use the range and standard deviation of per-task scores as 'task-space sharpness' metrics, arguing that contaminated or overfit models show high sharpness (spiky profiles) while genuinely generalizing models show flat profiles. The framework is instantiated on MME, RealWorldQA, and CVRR-ES across many open and closed MLLMs. The central empirical validation is a controlled contamination experiment (Table 2) in which Qwen2.5-VL models are PEFT fine-tuned on test QA data; the authors claim this sharpens task-specific performance while harming overall generalization.

Significance. The core idea of perturbing the task rather than the input is a natural and potentially valuable complement to existing dynamic evaluation methods. The paper also makes a useful concrete proposal for scoring open-ended generative outputs with a judge model, and it evaluates a broad suite of models across image and video benchmarks, which is a substantial empirical effort. If the central claim were established, the framework could serve as a practical tool for flagging suspicious benchmark scores. However, the paper's own controlled experiment does not support the central claim: the sharpness metrics do not reliably increase under simulated extreme contamination, and the judge-based scoring introduces a circularity problem for at least one of the headline models. Because these issues bear directly on the paper's main contribution, the current version does not yet demonstrate that task-space sharpness distinguishes contamination from genuine generalization.

major comments (4)
  1. [Section 3.2, Table 2] The controlled contamination experiment contradicts the central claim. After PEFT fine-tuning on test QA, Qwen2.5-VL (7B) shows T0 rising from 63.27 to 96.21, average performance rising from 77.23 to 86.31, but Rng decreasing from 28.31 to 27.12 and SD decreasing from 13.31 to 10.28. Thus for the larger model, the 'extreme contamination' condition actually reduces task-space sharpness as measured by both reported metrics. This directly undercuts the abstract's statement that such fine-tuning 'harms overall generalization' and the hypothesis that contaminated models exhibit spiky cross-task profiles. The 3B model does show an increase in Rng and SD, so the two models move in opposite directions; no error bars or significance tests are provided, so it is unclear whether any of these changes are meaningful. At minimum, the paper does not demonstrate that sharpness isolates contamination.
  2. [Section 2.1, Eqs. (2.1)-(2.4)] The theoretical motivation is not formally connected to task perturbation. Equations (2.1)-(2.4) give a Taylor expansion of the loss under an input-space perturbation δ and argue that contaminated models have higher input-space curvature. The text then asserts, without a derivation, that the same flat-versus-sharp logic applies when 'we fix inputs and stress-test across different tasks.' No formal object is defined for a Hessian or curvature in task space, and no argument shows that per-task score variance (Rng, SD) corresponds to the curvature of an underlying loss surface. The analogy may be heuristic, but it is presented as theoretical grounding for the core contamination-detection claim, so this gap is load-bearing.
  3. [Section 2.4 and Table 1] The judge-based scoring for the open-ended tasks T1 and T2 is not validated against human judgments, and for at least one model it is circular. The text says a reasoning MLLM 'e.g., VL-Rethinker' is used as the judge, and VL-Rethinker itself appears in Table 1; consequently VL-Rethinker's T1 and T2 scores are in part self-assessments. Even if a different judge were used for the other models, the paper provides no calibration evidence that judge scores agree with human ratings, and Appendix A acknowledges that such discrepancies are an 'empirical risk.' Because T1 and T2 are two of the four tasks that define the ability vector and the sharpness metrics, an unvalidated and partially circular judge undermines the quantitative claims built on those scores.
  4. [Section 3.2, Figure 3 and Table 1] The paper assumes, without adequate support, that cross-task performance variance is a valid proxy for data contamination or overfitting rather than for intrinsic task difficulty or model capability differences. The four tasks are not shown to be comparable in difficulty, and no task-difficulty normalization is attempted. The controlled PEFT experiment was supposed to validate this assumption, but as noted above it does not: for the 7B model, contamination reduces variance. Consequently, the interpretive claims such as 'models known to likely have seen the benchmark during training show markedly uneven performance' (Section 1) are not established by the data presented.
minor comments (4)
  1. [Figure 4 and Section 3.2] The text describing correlations swaps the task labels: it says 'Question Verification (T2) and Question Generation (T3)', but per Section 2.2, T2 is question generation and T3 is answer verification. This makes the correlation discussion hard to follow.
  2. [Table 1 and Section 2.3] The notation is confusing: Section 2.3 defines both Sdist (maximum pairwise distance) and Sdist (average pairwise distance) with the same symbol, while Table 1 reports 'SD = Sdist'. Use distinct symbols for the two metrics and clarify which one is reported in each table.
  3. [Abstract and Section 3.2] The abstract states that fine-tuning on simulated test data 'harms overall generalization', but Table 2 shows the average score increases substantially for both models (74.63 to 81.55 for 3B; 77.23 to 86.31 for 7B). The wording should be qualified to refer to performance on non-contaminated tasks, not overall average performance.
  4. [References] Reference [38] is listed as a survey on multimodal large language models but is cited in the MME benchmark paragraph in a way that may confuse it with the MME paper (Reference [10]). Please verify that all benchmark citations point to the correct sources.

Circularity Check

2 steps flagged · score 6.0 of 10

Self-evaluated judge for VL-Rethinker and contamination labels inferred from the metric itself make the core validation partially circular; the controlled experiment does not independently confirm the sharpness-contamination link.

  1. self definitional [Section 2.4 ('Evaluating Subjective Tasks with a Reasoning MLLM as Judge'); Table 1; Section 3.2 ('Interpreting Model Ability Vectors')]
    "We use an reasoning judge model to assess output quality. Reasoning MLLM as judge provides not only rate but also a reasoning. In evaluation, the judge scores each generated caption/question based on a detailed rubrics. This score informs task loss ℓt (e.g., 1 − judge_score). The judge is fixed for all MLLM evaluations. ... For example, reasoning-focused models(RL trained) like VL-Rethinker and GPT-o4 mini generally show strong, balanced performance."

    The paper names VL-Rethinker as the judge example and then includes VL-Rethinker in Table 1 as one of the evaluated models. Its T1 (caption) and T2 (question generation) scores are therefore produced by VL-Rethinker judging its own outputs. The later claim that VL-Rethinker shows 'strong, balanced performance' and occupies the desirable low-sharpness region is based on self-assessment, not independent measurement. Since no human calibration or separate judge is reported for these scores, the model's generative-task scores reduce to its own preferences, which is a self-referential evaluation step in the central comparison.

  2. other [Section 1, 'Contributions / Our findings validate...' versus Section 3.2, Table 2]
    "Our findings validate that dynamic task perturbation is highly revealing: models known to likely have seen the benchmark during training show markedly uneven performance, whereas models with true generalisation handle task shifts better."

    The framework identifies 'likely contaminated' models by their uneven cross-task performance (large Rng/SD), yet the validation claim treats the same uneven performance as evidence that those models are contaminated. No independent contamination labels for the pretrained models are provided, so the conclusion reduces to the metric's own interpretation. The one controlled check, PEFT on test QA in Table 2, does not break the circle: for Qwen2.5-VL 7B, simulated contamination decreases Rng (28.31 to 27.12) and SD (13.31 to 10.28) while raising average performance, contradicting the 'sharpness indicates contamination' premise instead of independently confirming it.

full rationale

The paper's core idea—task perturbation as a contamination detector—is not fully circular by construction, because the four task scores are measured and aggregated rather than fitted from a single contamination label. However, the central validation contains two self-referential links. First, the judge is itself an evaluated model: VL-Rethinker's caption and question-generation scores are self-judged, so its low task-space sharpness claim is not independent of the evaluator. Second, the paper's key finding that 'models known to likely have seen the benchmark' show uneven performance is circular, since those models are identified as likely contaminated by the very unevenness the framework measures; the controlled PEFT experiment is meant to supply external grounds, but for the 7B model the range and SD move in the opposite direction, so the experiment does not validate the assumed link. The Appendix A limitation statement also concedes that the MLLM judge 'may reflect inherent biases' and lacks human assessment, which reinforces the judge circularity concern. These issues affect the central claim's evidentiary support, but the framework itself does contribute new measurement content, so a score of 6 (partial circularity) is appropriate rather than 8 or 10.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on four unproven premises: that task-space variance measures contamination, that an MLLM judge gives valid scores for open-ended tasks, that the four tasks are comparable in difficulty, and that flat-minima theory transfers from input space to task space. No free numeric parameters are fitted to data in a quantitative model, but the judge choice and task prompts are uncontrolled qualitative free choices that materially affect every reported score.

free parameters (3)
  • Judge model selection and scoring rubric = not specified
    T1 (captioning) and T2 (question generation) scores are produced by a reasoning MLLM judge (VL-Rethinker in the example) with unreported rubric thresholds; the choice of judge is a free choice that determines the reported scores and is not independently calibrated against human labels in this paper.
  • PEFT hyperparameters = not reported
    The controlled contamination experiment uses LoRA fine-tuning on RealWorldQA test data; learning rate, rank, epochs, and data fraction are not reported, so the magnitude of the T0 increase and cross-task transfer is not reproducible or comparable.
  • Task prompts = not specified
    The prompts used for captioning, question generation, and verification are not given; different prompts likely change scores.
assumptions (4)
  • ad hoc to paper Cross-task performance variance (range and SD) is a valid indicator of data contamination or overfitting
    Used throughout Section 3.2 to classify models as contaminated or generalizing; not validated against known contamination labels except in the single PEFT simulation, which shows mixed results.
  • ad hoc to paper The MLLM judge's scores for open-ended outputs are valid and unbiased ground truth
    Section 2.4 and Appendix A; no human agreement study is presented, and the judge is itself one of the evaluated models.
  • ad hoc to paper The four tasks T0-T3 are comparable in difficulty so that performance differences reflect model capability rather than task difficulty
    The sharpness metric compares raw scores across distinct tasks with different formats and evaluation methods; no normalization or calibration is performed.
  • ad hoc to paper Flat vs sharp minima theory from input-space optimization transfers to task-space performance profiles
    Section 2.1 derives loss curvature under input perturbation delta, then states as an analogy that task perturbation reveals the same effect; the mathematical link is not established.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation." pith.science (2026). https://pith.science/paper/DSGNU7YX

@misc{pith2026250607202,
  author       = {Pith},
  title        = {Pith review of: Reasoning Multimodal Large Language Model: Data Contamination and Dynamic Evaluation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DSGNU7YX}},
  note         = {Machine review of arXiv:2506.07202}
}
read the original abstract

Multimodal Large Language Models (MLLMs) show impressive vision-language benchmark performance, yet growing concerns about data contamination (test set exposure during training) risk masking true generalization. This concern extends to reasoning MLLMs, often fine-tuned via reinforcement learning from potentially contaminated base models. We propose a novel dynamic evaluation framework to rigorously assess MLLM generalization, moving beyond static benchmarks. Instead of perturbing inputs, we perturb the task itself. Using the same visual input, models are evaluated across a family of tasks (e.g., QA, captioning, question posing, verification) to probe diverse capabilities. This task perturbation reveals whether model performance is robust or reliant on superficial task-specific cues. Our approach is analogous to loss landscape sharpness: models overfit or contaminated for a single task (sharp minima) falter under task shifts, unlike models with generalizable solutions (flatter minima). We developed an automated pipeline with a calibrated judge scoring open-ended generations (captions, questions) using paraphrase and corruption sampling. Applying this framework to leading image/video MLLMs on benchmarks including MME, RealWorldQA, and CVRR-ES, we analyze each model's cross-task "ability vector." We demonstrate that fine-tuning on simulated test data (extreme contamination) drastically sharpens task-specific performance but harms overall generalization. Our dynamic task perturbation offers deeper insights into MLLM generalization, distinguishing genuine understanding from spurious leakage or overfitting.

Figures

Figures reproduced from arXiv: 2506.07202 by the authors.

Figure 1
Figure 1. To mitigate contamination, we augment task and report scoring/sharpness metrics. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. How to convert a QA only benchmark into multi-tasks evaluation benchmark? [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Model average performance versus task-space sharpness (performance range) on the [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Correlation matrix of task perfor￾mances (T0: QA, T1: Captioning, T2: Ques￾tion Gen, T3: Verification) across models on the MME dataset. 1. Measuring Task-Space Sharpness: Based on each model’s performance vector [PQA, PCap, PQGen, PV erif ], we primarily measure task-…
Figure 5
Figure 5. Figure 5: How performance align with sharpness for RealworldQA dataset? [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: How related are each task for realworldQA dataset? [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: How does performance align with sharpness on the CVRR-ES dataset? [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

42 extracted references · 16 canonical work pages

  1. [1]

    Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs

    Abdelrahman Abouelenin, Nitya Absar, Saurabh Agarwal, Christopher Akiki, Mohanad Al- Ghossein, Saad Alkhereyf, Ahmed Almahallawi, Ahmed H Awadallah, Shally Batra, Anuj Bhaskar, et al. Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs. Technical report, Microsoft, 2025

    work page 2025

  2. [2]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024

    arXiv 2024

  3. [3]

    Microsoft coco captions: Data collection and evaluation server

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015

    arXiv 2015

  4. [4]

    Closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. Closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

    arXiv 2024

  5. [5]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V . Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025

    2025

  6. [6]

    Documenting large webtext corpora: A case study on the colossal clean crawled corpus

    Jesse Dodge, Maarten Sap, Ana Marasovi´c, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. arXiv preprint arXiv:2104.08758, 2021

    arXiv 2021

  7. [7]

    Complex Video Reasoning and Robustness Evaluation Suite (CVRR-ES)

    EvalAI. Complex Video Reasoning and Robustness Evaluation Suite (CVRR-ES). https: //eval.ai/web/challenges/challenge-page/2480/overview, 2024. Accessed:. 10

    work page 2024

  8. [8]

    Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes

    Lizhou Fan, Wenyue Hua, Lingyao Li, Haoyang Ling, Yongfeng Zhang, and Libby Hemphill. Nphardeval: Dynamic benchmark on reasoning ability of large language models via complexity classes. arXiv preprint arXiv:2312.14890, 2023

    arXiv 2023

Show all 42 references
  1. [9]

    Video-R1: Reinforcing video reasoning in mllms

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-R1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776, 2025. Preprint. Under review

    2025 arXiv

  2. [10]

    MME: A comprehensive evaluation benchmark for multimodal large language models

    Chao Fu, Xiaochi Wu, Shuzhen Geng, An Zhou, Jinheng Zhang, Haotian Liu, Chunyuan Li, Ziwei Liu, Peng Gao, and Lin Wang. MME: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.13394, 2023

    2023 arXiv

  3. [11]

    Sphinx-x: Scaling data and parameters for a family of multi-modal large language models

    Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. arXiv preprint arXiv:2402.05935, 2024

    2024 arXiv

  4. [12]

    Time travel in LLMs: Tracing data contamination in large language models

    Shahriar Golchin and Mihai Surdeanu. Time travel in LLMs: Tracing data contamination in large language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  5. [13]

    Goodfellow, Jonathon Shlens, and Christian Szegedy

    Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples, 2015

    2015

  6. [14]

    Flat minima

    Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Technical Report 3/97, Institut für Informatik, Technische Universität München, 1997

    1997

  7. [15]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021

    2021

  8. [16]

    On large-batch training for deep learning: Generalization gap and sharp minima

    Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836, 2016

    2016 arXiv

  9. [17]

    LLaV A-OneVision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. LLaV A-OneVision: Easy visual task transfer. Transactions on Machine Learning Research (TMLR), 2025. Accepted by TMLR

    2025

  10. [18]

    Seed- bench: Benchmarking multimodal llms with generative comprehension

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023

    2023 arXiv

  11. [19]

    VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning

    Xinhao Li, Ziang Yan, Desen Meng, Lu Dong, Xiangyu Zeng, Yinan He, Yali Wang, Yu Qiao, Yi Wang, and Limin Wang. VideoChat-R1: Enhancing spatio-temporal perception via reinforce- ment fine-tuning. arXiv preprint arXiv:2504.06958, 2025

    2025 arXiv

  12. [20]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023

    2023 arXiv

  13. [21]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    2024

  14. [22]

    On the robustness of multimodal language model towards distractions, 2025

    Ming Liu, Hao Chen, Jindong Wang, and Wensheng Zhang. On the robustness of multimodal language model towards distractions, 2025

    2025

  15. [23]

    Is your video language model a reliable judge? In The Thirteenth International Conference on Learning Representations, 2025

    Ming Liu and Wensheng Zhang. Is your video language model a reliable judge? In The Thirteenth International Conference on Learning Representations, 2025

    2025

  16. [24]

    Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307.06281, 2023

    2023 arXiv

  17. [25]

    The Llama 3 herd of models

    Llama Team, Meta AI. The Llama 3 herd of models. Technical report, Meta AI, July 2024. Accessed:. 11

    2024

  18. [26]

    Ok-vqa: A visual question answering benchmark requiring external knowledge

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195–3204, 2019

    2019

  19. [27]

    MM-EUREKA: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Tiancheng Han, Daocheng Fu, Botian Shi, Wenhai Wang, Junjun He, Kaipeng Zhang, Ping Luo, Yu Qiao, Qiaosheng Zhang, and Wenqi Shao. MM-EUREKA: Exploring the frontiers of multimodal reasoning with rule-based rei...

    2025 arXiv

  20. [28]

    Introducing GPT-4.1

    OpenAI. Introducing GPT-4.1. Blog post, November 2024. Accessed:

    2024

  21. [29]

    Introducing o3 and o4-mini: Our smartest models yet

    OpenAI. Introducing o3 and o4-mini: Our smartest models yet. Blog post, December 2024. Accessed:

    2024

  22. [30]

    Chatterji, Faisal Ladhak, and Tatsunori Hashimoto

    Yonatan Oren, Nicole Meister, Niladri S. Chatterji, Faisal Ladhak, and Tatsunori Hashimoto. Proving test set contamination in black-box language models. In The Twelfth International Conference on Learning Representations, 2024

    2024

  23. [31]

    Qwen2.5-VL Technical Report

    Qwen Team, Alibaba Group. Qwen2.5-VL Technical Report. Technical report, Alibaba Group, 2025

    2025

  24. [32]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  25. [33]

    VL- Rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning

    Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. VL- Rethinker: Incentivizing self-reflection of vision-language models with reinforcement learning. arXiv preprint arXiv:2504.08837, 2025. Preprint. Under review

    2025 arXiv

  26. [34]

    Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Jinze Bai, Zhe Chen, Wenhai Wang, Yi Wang, Yu Qiao, Wenqi Shao, and Qwen Team. Qwen2-VL: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  27. [35]

    InternVideo2.5: Empowering video mllms with long and rich context modeling

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Yu Qiao, Yali Wang, and Limin Wang. InternVideo2.5: Empowering video mllms with long and rich context modeling. arXiv prepr...

    2025 arXiv

  28. [36]

    Realworldqa

    X.AI. Realworldqa. Blog post, November 2024. Accessed: 2025-05-12

    2024

  29. [37]

    Dynamic multimodal evaluation with flexible complexity by vision-language bootstrapping

    Yue Yang, Shuibo Zhang, Kaipeng Zhang, Yi Bin, Yu Wang, Ping Luo, and Wenqi Shao. Dynamic multimodal evaluation with flexible complexity by vision-language bootstrapping. In The Thirteenth International Conference on Learning Representations, 2025

    2025

  30. [38]

    A survey on multimodal large language models

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306.13549, 2023

    2023 arXiv

  31. [39]

    Mm-vet: Evaluating large multimodal models for integrated capabilities

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490, 2023

    2023 arXiv

  32. [40]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on C...

    2024

  33. [41]

    Dyval: Graph-informed dynamic evaluation of large language models.arXiv preprint arXiv:2309.17167, 2023

    Kaijie Zhu, Jiaao Chen, Jindong Wang, Neil Zhenqiang Gong, Diyi Yang, and Xing Xie. Dyval: Graph-informed dynamic evaluation of large language models.arXiv preprint arXiv:2309.17167, 2023

    2023 arXiv

  34. [42]

    reasoning MLLMs,

    Kaijie Zhu, Jindong Wang, Qinlin Zhao, Ruochen Xu, and Xing Xie. Dynamic evaluation of large language models by meta probing agents. In Forty-first International Conference on Machine Learning, 2024. 12 A Limitations While our dynamic task perturbation framework offers valuabl...

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.