Pith. sign in

REVIEW 3 major objections 6 minor 14 cited by

Kwai Keye-VL Technical Report

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Kwai Keye-VL, an 8B multimodal model, claims the lead in short-video understanding with 68.03% on KC-MMBench.

desk verdict A detailed, useful video-MLLM technical report whose public-benchmark results are plausible but whose self-built short-video benchmark is likely measuring RL overfitting to the same task family it samples. read the letter →

arxiv 2507.01949 v1 pith:HD53LQPH submitted 2025-07-02 cs.CV

Kwai Keye Team , Biao Yang , Bin Wen , Changyi Liu , Chenglong Chu , Chengru Song , Chongling Rao , Chuan Yi
show 52 more authors
Da Li Dunju Zang Fan Yang Guorui Zhou Hao Peng Haojie Ding Jiaming Huang Jiangxia Cao Jiankang Chen Jingyun Hua Jin Ouyang Kaibing Chen Kaiyu Jiang Kaiyu Tang Kun Gai Shengnan Zhang Siyang Mao Sui Huang Tianke Zhang Tingting Gao Wei Chen Wei Yuan Xiangyu Wu Xiao Hu Xingyu Lu Yang Zhou Yi-Fan Zhang Yiping Yang Yulong Chen Zhenhua Wu Zhenyu Li Zhixin Ling Ziming Li Dehua Ma Di Xu Haixuan Gao Hang Li Jiawei Guo Jing Wang Lejian Ren Muhao Wei Qianqian Wang Qigen Hu Shiyao Wang Tao Yu Xinchen Luo Yan Li Yiming Liang Yuhang Hu Zeyi Lu Zhuoran Yang Zixing Zhang
This is my paper · ORCID
classification cs.CV
keywords KwaiKeye-VLmultimodallargelanguagemodelshort-videounderstandingchain-of-thoughtreasoningauto-thinkmodereinforcementlearningpreferenceoptimizationKC-MMBench
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces Kwai Keye-VL, an 8-billion-parameter multimodal model built for short-form video understanding, and argues that a video-heavy training corpus paired with a staged reasoning recipe is what lets a model of this size lead on video benchmarks. The authors report state-of-the-art results on public video benchmarks and a 68.03% average accuracy on their released KC-MMBench, against 57.62% for the next-best model at the same scale. The central claim is that the model learns when to reason and when to answer directly, taught through a five-mode cold-start data mixture followed by reinforcement learning and iterative preference alignment. If correct, this means an open 8B model can handle content moderation, product attribute prediction, event ordering, and similar short-video tasks without sacrificing general image and reasoning performance.

What carries the argument

The load-bearing mechanism is the five-mode cold-start data mixture used in the second post-training phase: 330,000 non-reasoning samples, 230,000 thinking samples, 20,000 auto-think samples, 100,000 agentic think-with-image samples, and curated high-quality video data. This mixture teaches the model to decide when to emit a long chain of thought and when to answer directly, which the paper credits for simultaneous gains in Thinking and Non-Thinking modes. Reinforcement learning then applies GRPO with outcome and consistency rewards, including a dedicated short-video RL stage using ground-truth labels, and the final iterative alignment step uses rule-based and model-based scores to build preference pairs that correct repetitive or illogical outputs.

What would settle it

Take KC-MMBench and re-annotate it on a freshly collected set of short videos with reworded questions by annotators who did not build the benchmark; if the 10-point gap over MiMo-VL narrows materially, the advantage is partly benchmark-specific. A cheaper check is to vary the text cosine decontamination threshold around 0.50 and count how many excluded training samples would then overlap with KC-MMBench.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that short-video understanding is primarily a data and reasoning-scheduling problem rather than a vision-encoder problem. Keye-VL starts from the Qwen3-8B language decoder and a native-resolution SigLIP vision encoder, pre-trains on over 600 billion tokens with a strong video component, and then runs a two-phase post-training recipe: supervised fine-tuning plus mixed preference optimization for foundational skills, followed by a five-mode cold-start mixture (thinking, non-thinking, auto-think, think-with-image, and high-quality video), GRPO reinforcement learning, and iterative alignment. Reported results put Keye-VL ahead of similar-scale open models on Video-MMMU, TempCompass, LongVideoBench, and MMVU, and give it a 10-point edge on KC-MMBench, a new short-video benchmark the authors built from in-house platform data and released.

Load-bearing premise

The headline short-video advantage rests on the assumption that KC-MMBench, built and decontaminated by the authors with image similarity above 0.98 and text cosine similarity above 0.50 used as exclusion thresholds, measures real short-video skill rather than matching the model's own training data and prompt wording.

Editorial extensions

If this is right

  • An 8B model with video-tailored data can outperform similar-scale open models on Video-MMMU, TempCompass, LongVideoBench, and MMVU while staying competitive on general image benchmarks.
  • The five-mode cold-start plus RL recipe provides a template for letting a model choose reasoning depth by task difficulty, as shown by Auto-Think thinking ratios of 0.35 on MathVista, 0.34 on MMStar, 0.08 on HallusionBench, and 0.00 on OCRBench.
  • KC-MMBench opens a way to evaluate skills that public English benchmarks miss, including pornographic comment detection, collection order, hot-video aggregation, high-like prediction, and e-commerce product matching and attribute prediction.
  • The decontamination analysis flags MM-Eureka and MMPR as datasets with substantial overlap against common benchmarks, warning practitioners that training on them can inflate reported scores.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the Auto-Think mechanism is as general as reported, the same five-mode scheduling could be applied to other mixed-difficulty modalities, such as long-form video or live audio, where the model routes simple inputs to direct answers and complex ones to chain-of-thought.
  • The 10-point KC-MMBench gap comes from a benchmark the authors built from their own platform; it would be more convincing if it survived independent re-annotation or a third-party short-video suite, since question wording and data decontamination choices can favor the model that trained on similar data.
  • The reported mutual enhancement between reasoning and non-reasoning data suggests long chain-of-thought data may act as a general regularizer, a hypothesis worth testing on smaller models where per-task ablations are cheaper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper introduces Kwai Keye-VL, an 8-billion-parameter multimodal large language model built on a SigLIP-400M vision encoder and Qwen3-8B language decoder, with a native-resolution vision transformer, a 600B-token four-stage pre-training pipeline, and a two-phase post-training recipe. The post-training combines no-reasoning SFT and preference optimization with a five-mode CoT cold-start mixture and mix-mode GRPO reinforcement learning, followed by iterative alignment. The authors report state-of-the-art results on several public video benchmarks, competitive performance on general image and reasoning benchmarks, a large advantage on their newly released KC-MMBench for short-video tasks, and a small-scale human evaluation of user experience.

Significance. If the claims hold, the paper shows that an 8B open-source model can lead short-video understanding while remaining competitive on general vision-language tasks, which is practically significant for deployment on video-centric platforms. The paper's strengths include a detailed and transparent description of the data pipeline and training recipe, explicit reporting of decontamination procedures, the release of the KC-MMBench benchmark, and an honest limitations section. The central short-video claim, however, rests on a self-constructed benchmark whose task family overlaps with the RL training data, and the quantitative evidence base is weakened by single-run benchmark numbers and a small human evaluation.

major comments (3)
  1. [§4.2.2, §6.3.3, Appendix B] The KC-MMBench advantage is not independent evidence for general short-video capability. The RL stage in §4.2.2 is trained on ground-truth or annotated labels from 'various short video content understanding tasks' and explicitly aligns outputs with 'desired value orientations,' while the six KC-MMBench tasks in Appendix B and Table 4 (CPV, SPU, Collection Order, Hot Videos Aggregation, High Like, Pornographic Comment) are drawn from the same Kuaishou platform data and annotation pipelines. The decontamination in Appendix A.2 covers 29 benchmarks but does not state that KC-MMBench is among them, and the image/text similarity thresholds (0.98 and 0.50) are author-chosen with no sensitivity analysis. Under these conditions, the 68.03 vs. 57.62 gap in Table 3 may measure task-specific optimization rather than a general short-video understanding capability. I recommend either releasing the benchmark and reporting an ablation with the short-video RL data held out, or demonstrating transfer to independently constructed short-video tasks.
  2. [§6.2, Table 3] The claim of 'significantly outperforms' on public video benchmarks is stronger than the table supports. On LongVideoBench in thinking mode Keye-VL scores 62.8, below InternVL3's 63.9; on Video-MMME the margin over MiMo-VL is 0.3 percentage points (67.7 vs. 67.4), and GPT-4o scores 71.9. All numbers are single runs with no error bars or significance tests, so margins this small cannot support the word 'significantly.' The abstract should also carry the qualification 'among models of similar scale' that appears in Figure 1.
  3. [§6.3.2, Tables 5-6] The human evaluation is too small and weakly reported to substantiate 'superior user experience.' It uses only 150 question-answer pairs per modality, gives no inter-annotator agreement statistics, no confidence intervals, and no significance tests; the overall video composite difference is 0.02 (3.33 vs. 3.31). This claim should be softened or the evaluation should be expanded with appropriate statistical analysis.
minor comments (6)
  1. [§2.1] 'existing a large gap' should be 'there exists a large gap.'
  2. [§6.2] The benchmark name 'MathVistaMINI' should be written as 'MathVista-Mini' for consistency with the original benchmark naming.
  3. [Appendix C.1.1] Figure 6 contains emojis that do not render in the text, making the example difficult to follow.
  4. [Appendix B, Table 4] The task is called 'High-Like Video Classification' in Appendix B but 'High Like' in Table 4; the naming should be unified.
  5. [References] The dataset 'MMPR' is cited several times but is not clearly defined in the reference list; the MPO paper by Wang et al. (2024b) does not appear to be the MMPR dataset reference.
  6. [§7.1] The text says 'MathVista' but Table 7 and the public benchmark table use 'MathVistaMINI' or 'MathVista-Mini'; please use consistent terminology.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the headline short-video and general-capability claims are grounded in independent public benchmarks; KC-MMBench is self-assessed but not a by-construction reduction.

full rationale

The paper's central capability claims rest primarily on external public benchmarks. Table 3 reports Keye-VL-8B at 57.6 on Video-MMMU versus 47.4 for Qwen2.5-VL-7B and 48.9 for InternVL3-8B, 66.1 on MMVU versus 45.5, 71.5 on TempCompass, and strong MMMU and AI2D results; these benchmarks were constructed outside the authors' pipeline, so the central claim has independent evidentiary content. The training recipe uses openly described components (SigLIP encoder, Qwen3 decoder, GRPO, MPO) and the paper reports measurable ablations, such as Section 7.2's RL gains of 1.44%/2.17% across ten benchmarks, and Section 7.1's cold-start gains. The acknowledged limitations in Sections 6.3.4 and 8 further indicate that the report is not engineered so that every result equals its inputs. The only self-referential elements are the author-constructed KC-MMBench, the internal human evaluation, and citations to prior Kuaishou work such as TaskGalaxy and MM-RLHF. Section 4.2.2 trains the model with RL on 'various short video content understanding tasks' using 'ground-truth or annotated labels,' while Appendix B describes KC-MMBench tasks drawn from the same Kuaishou business domains (CPV, SPU, Collection Order, Hot Videos Aggregation, High Like, Pornographic Comment); Appendix A.2's decontamination protocol is stated for 29 benchmarks and does not explicitly address KC-MMBench task-family overlap. This makes the 68.03% versus 57.62% KC-MMBench gap a weaker, partly self-assessed piece of evidence. However, the paper never states that KC-MMBench itself or its exact instances were used as RL training data, and no equation or fitted parameter makes the benchmark score equal to a training target by construction. The TaskGalaxy and MM-RLHF citations involve overlapping authors but serve only as data-construction tools, not as uniqueness theorems that force the reported outcomes. Under the requirement that circularity be exhibited by a concrete reduction, the derivation chain is not circular; the appropriate finding is low circularity due to minor self-assessment and self-citation that is not load-bearing.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's claims rest on several unstated domain assumptions about the reliability of synthetic data and reward models, plus hand-set training and decontamination parameters. No invented physical entities are introduced. The free parameters and assumptions do not make the results circular, but they limit the generality of the performance claims.

free parameters (3)
  • CoT cold-start data mixture proportions = 330k non-reasoning, 230k thinking, 20k auto-think, 100k agentic, 24k thinking + 80k non-thinking video
    Chosen by hand in Section 4.2.1; no ablation isolates each proportion, yet the recipe is the paper's core innovation.
  • Decontamination similarity thresholds = 0.98 image / 0.50 text CLIP cosine
    Set in Appendix A.2; if thresholds are too lax, benchmark gains may reflect leakage; no sensitivity analysis is provided.
  • Vision token budgets = 16384 tokens per image, 24576 max per video, 128/768 min/max per frame
    Hand-set in Section 2.2; affects how much visual detail the model sees and therefore all benchmark scores.
assumptions (4)
  • domain assumption Qwen3-8B and SigLIP-400M provide sufficient base language and vision capabilities for the target tasks.
    Model is built on these open checkpoints (Section 2), so the final result inherits their strengths and biases.
  • domain assumption Captions and CoT paths synthesized by MLLMs such as Qwen2.5-VL, GPT-4o, and Gemini are accurate enough to serve as training targets.
    Used for re-captioning, thinking data, agentic code generation, and preference data (Sections 3.1, 4.2.1); no verification rate is reported except sandbox checks for code.
  • domain assumption Reward signals from large multimodal models are valid for RL optimization.
    Mix-Mode RL uses MLLM judges for outcome and consistency rewards (Section 4.2.2); the paper concedes reliability and cost limitations in Section 8.
  • domain assumption Internal benchmark and human evaluation are representative of real short-video user experience.
    KC-MMBench and the 300-item internal set are built in-house (Sections 6.2, 6.3); generalization to all short-video scenarios is assumed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Kwai Keye-VL Technical Report." pith.science (2026). https://pith.science/paper/HD53LQPH

@misc{pith2026250701949,
  author       = {Pith},
  title        = {Pith review of: Kwai Keye-VL Technical Report},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HD53LQPH}},
  note         = {Machine review of arXiv:2507.01949}
}
read the original abstract

While Multimodal Large Language Models (MLLMs) demonstrate remarkable capabilities on static images, they often fall short in comprehending dynamic, information-dense short-form videos, a dominant medium in today's digital landscape. To bridge this gap, we introduce \textbf{Kwai Keye-VL}, an 8-billion-parameter multimodal foundation model engineered for leading-edge performance in short-video understanding while maintaining robust general-purpose vision-language abilities. The development of Keye-VL rests on two core pillars: a massive, high-quality dataset exceeding 600 billion tokens with a strong emphasis on video, and an innovative training recipe. This recipe features a four-stage pre-training process for solid vision-language alignment, followed by a meticulous two-phase post-training process. The first post-training stage enhances foundational capabilities like instruction following, while the second phase focuses on stimulating advanced reasoning. In this second phase, a key innovation is our five-mode ``cold-start'' data mixture, which includes ``thinking'', ``non-thinking'', ``auto-think'', ``think with image'', and high-quality video data. This mixture teaches the model to decide when and how to reason. Subsequent reinforcement learning (RL) and alignment steps further enhance these reasoning capabilities and correct abnormal model behaviors, such as repetitive outputs. To validate our approach, we conduct extensive evaluations, showing that Keye-VL achieves state-of-the-art results on public video benchmarks and remains highly competitive on general image-based tasks (Figure 1). Furthermore, we develop and release the \textbf{KC-MMBench}, a new benchmark tailored for real-world short-video scenarios, where Keye-VL shows a significant advantage.

Figures

Figures reproduced from arXiv: 2507.01949 by the authors.

Figure 1
Figure 1. Benchmark performance of Kwai Keye-VL: Keye-VL-8B establishes a new state-of-the-art among models of a similar scale, showing a clear lead in video-centric benchmarks (left) while maintaining competitive results on general perception and reasoning tasks (right), validating our training approach. 1 arXiv:2507.01949v1 [cs.CV] 2 Jul 2025 [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The Kwai Keye-VL model architecture is based on the Qwen3-8B language model and incor￾porates a vision encoder initialized from the open-source SigLIP. It supports native dynamic resolution, preserving the original aspect ratio of images by dividing each into a 14x14 patch sequence. A simple MLP layer then maps and merges the visual tokens. The model uses 3D RoPE for unified processing of text, image, and video info… view at source ↗
Figure 3
Figure 3. The Kwai Keye pre-training pipeline, featuring a four-stage progressive strategy: Image-Text Matching, ViT-LLM Alignment, Multi-task Pre-training, and Annealing with model merging. 3.2 Training Recipe We employ a four-stage progressive training strategy to build a powerful multi-modal foundation model with strong vision-language alignment capabilities. The training pipeline, illustrated in [PITH_FULL_IMAGE:figures/… view at source ↗
Figures from the paper (10 more)
Figure 4
Figure 4. Figure 4: No-Reasoning Training Pipeline: The process begins with a Base model, proceeds through Supervised Fine-Tuning (utilizing 70k task galaxy, 200k filtered QA pairs, and human-annotated im￾age/video captions), and culminates in Mixed Preference Optimization with various pr…
Figure 5
Figure 5. Figure 5: Reasoning Training Pipeline: The process consists of three key steps: CoT Cold Start (involving sampling, quality checks, and high-frequency think path selection from a data pool to create a CoT Cold￾Start Dataset), Mix-Mode RL (featuring Thinking Mode, Non-Thinking Mo…
Figure 6
Figure 6. Figure 6: An example of Kwai Keye-VL successfully deducing the corresponding movie name based on [PITH_FULL_IMAGE:figures/full_fig_p033_6.png]
Figure 7
Figure 7. Figure 7: An example demonstrating Kwai Keye-VL’s reflective ability learned from the RL process. Kwai [PITH_FULL_IMAGE:figures/full_fig_p034_7.png]
Figure 8
Figure 8. Figure 8: An example demonstrating Kwai Keye-VL’s application in an e-commerce scenario: Kwai [PITH_FULL_IMAGE:figures/full_fig_p035_8.png]
Figure 9
Figure 9. Figure 9: An example demonstrating Kwai Keye-VL’s understanding of short video comments: Kwai [PITH_FULL_IMAGE:figures/full_fig_p036_9.png]
Figure 10
Figure 10. Figure 10: An example demonstrating how Kwai Keye-VL solves complex physics problems based on [PITH_FULL_IMAGE:figures/full_fig_p037_10.png]
Figure 11
Figure 11. Figure 11: An example of Kwai Keye-VL writing code to extract the relevant parts from high-resolution [PITH_FULL_IMAGE:figures/full_fig_p038_11.png]
Figure 12
Figure 12. Figure 12: An example demonstrating Kwai Keye-VL analyzing the difficulty of a problem and choosing [PITH_FULL_IMAGE:figures/full_fig_p039_12.png]
Figure 13
Figure 13. Figure 13: An example demonstrating Kwai Keye-VL choosing the Thinking mode to solve a hard [PITH_FULL_IMAGE:figures/full_fig_p040_13.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 14 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MonkeyOCRv2: A Visual-Text Foundation Model for Document AI

    cs.CV 2026-07 conditional novelty 6.5 of 10

    A document-oriented ViT family pretrained with text generation plus pixel reconstruction on 113M images transfers across recognition, detection, parsing, and understanding, setting open-source SOTA on MDPBench with a ...

  2. CodePercept: Code-Grounded Visual STEM Perception for MLLMs

    cs.CV 2026-03 conditional novelty 6.5 of 10

    Perception, not reasoning, is the main bottleneck for MLLM STEM visual reasoning, and training on executable reconstruction code measurably fixes it.

  3. InSight-doc: Agentic Visual Perception for Long-Document Understanding

    cs.CV 2026-08 conditional novelty 6.0 of 10

    InSight-doc trains an 8B vision-language model to zoom into document sub-regions on demand, improving long-document VQA accuracy by up to 16.4 points while cutting latency by 41-68%.

  4. FaithEyes: Towards Faithful Tool Use via Multi-Agent Process-Image Verification

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A self-judging multi-agent VLM injects process-image helpfulness verdicts into tool observations and scales tool rewards by the helpful-call ratio, improving accuracy and tool faithfulness.

  5. LENS: Adaptive Spatio-Temporal Zooming for Keyframe Sampling in Long-Form Videos

    cs.CV 2026-07 accept novelty 6.0 of 10

    Adaptive allocation of a limited frame budget between CLIP-prompted spatial zoom-ins and graph-based temporal hyperframes beats prior keyframe samplers on long-video MLLM benchmarks.

  6. Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Attention maps from a small MLLM can serve as a training-free, query-conditioned frame selector, improving long-video QA accuracy under fixed frame budgets.

  7. HoloCount: A Holistic Visual Counting Benchmark for MLLMs

    cs.CV 2026-07 conditional novelty 6.0 of 10

    HoloCount is a three-tier visual counting benchmark showing that MLLMs fail systematically on analytical reasoning, high-density scenes, and linguistic prior conflicts, with even the best models dropping below 50% acc...

  8. BVS: Bayesian Visual Search with Multimodal Large Language Model for Fine-grained Perception

    cs.CV 2026-07 conditional novelty 6.0 of 10

    BVS combines early-stop attention rollout priors with a scale-aware non-stationary kernel and GP-UCB to locate tiny objects in UHR images more accurately and with fewer MLLM queries than prior visual-search methods.

  9. R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    R-4B uses bi-mode annealing plus a GRPO variant with forced thinking and non-thinking rollouts so a 4B multimodal model adaptively picks when to reason, reportedly matching 16B thinking models on several reasoning benchmarks.

  10. Grounding Multilingual Multimodal LLMs With Cultural Knowledge

    cs.CL 2025-08 conditional novelty 6.0 of 10

    A Wikidata-derived multilingual multimodal dataset improves cultural understanding of a vision-language model, yielding state-of-the-art results on cultural benchmarks among open models.

  11. VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VL-Cogito, trained with progressive curriculum RL, online difficulty weighting, and dynamic length rewards, matches or beats prior reasoning MLLMs on ten multimodal benchmarks.

  12. VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videos

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VRBench is a benchmark of 960 long narrative videos with 8,243 human-written multi-step questions, plus a two-level evaluation of answer accuracy and reasoning quality for 31 large models.

  13. MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A new benchmark of 2,000 K-12 math questions photographed in real scenes finds that multimodal LLMs perform significantly worse on authentic handheld images than on clean benchmarks.

  14. ARC-Hunyuan-Video-7B: Structured Video Comprehension of Real-World Shorts

    cs.CV 2025-07 conditional novelty 5.0 of 10

    A 7B multimodal model that fuses audio and visual signals with explicit timestamps achieves strong measured comprehension of real-world short videos on the authors' new ShortVid-Bench benchmark.

Reference graph

Works this paper leans on

48 extracted references · 2 canonical work pages · cited by 14 Pith papers

  1. [1]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219,

    arXiv

  2. [6]

    Silent data corruptions at scale

    Harish Dattatraya Dixit, Sneha Pendharkar, Matt Beadon, Chris Mason, Tejasvi Chakravarthy, Bharath Muthiah, and Sriram Sankar. Silent data corruptions at scale. arXiv preprint arXiv:2102.11245,

    arXiv

  3. [7]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929,

    arXiv 2010

  4. [8]

    Align-kd: Distilling cross-modal alignment knowledge for mobile vision-language model

    Qianhan Feng, Wenshuo Li, Tong Lin, and Xinghao Chen. Align-kd: Distilling cross-modal alignment knowledge for mobile vision-language model. arXiv preprint arXiv:2412.01282,

    arXiv

  5. [9]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24108–24118, 2025a. Chaoyou Fu, Haojia Lin, Xi...

    arXiv

  6. [10]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    arXiv

  7. [11]

    Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data

    Shuhao Gu, Jialing Zhang, Siyuan Zhou, Kevin Yu, Zhaohu Xing, Liangdong Wang, Zhou Cao, Jintao Jia, Zhuoyi Zhang, Yixuan Wang, et al. Infinity-mm: Scaling multimodal performance with large-scale and high-quality instruction data. arXiv preprint arXiv:2410.18558,

    arXiv

  8. [12]

    Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025a. Dong Guo, Faming Wu, Feida Zhu, Fuxing Leng, Guang Shi, Haobin Chen, Haoqi Fan, Jian Wang, Jianyu Jiang, Ji...

    arXiv

Show all 48 references
  1. [13]

    Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems

    Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008,

    arXiv

  2. [14]

    Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model

    23 Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24290, 2025a. Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yua...

    arXiv

  3. [15]

    ReferItGame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Proce...

    2014

  4. [18]

    Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models

    Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895,

    arXiv

  5. [19]

    Denseworld-1m: Towards detailed dense grounded caption in the real world, 2025a

    Xiangtai Li, Tao Zhang, Yanwei Li, Haobo Yuan, Shihao Chen, Yikang Zhou, Jiahao Meng, Yueyi Sun, Shilin Xu, Lu Qi, Tianheng Cheng, Yi Lin, Zilong Huang, Wenhao Huang, Jiashi Feng, and Guang Shi. Denseworld-1m: Towards detailed dense grounded caption in the real world, 2025a. U...

    arXiv

  6. [20]

    Model merging in pre-training of large language models

    Yunshui Li, Yiyuan Ma, Shen Yan, Chaoyi Zhang, Jing Liu, Jianqiao Lu, Ziwen Xu, Mengzhao Chen, Minrui Wang, Shiyi Zhan, et al. Model merging in pre-training of large language models. arXiv preprint arXiv:2505.12082, 2025b. Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng...

    arXiv

  7. [21]

    Microsoft coco: Common objects in context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision–ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings...

    2014

  8. [22]

    Deepseek-v3 technical report

    Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024a. Yangzhou Liu, Yue Cao, Zhangwei Gao, Weiyun Wang, Zhe Chen, Wenhai Wang, Ha...

    arXiv

  9. [23]

    Vlm as policy: Common-law content moderation framework for short video platform

    Xingyu Lu, Tianke Zhang, Chang Meng, Xiaobei Wang, Jinpeng Wang, YiFan Zhang, Shisong Tang, Changyi Liu, Haojie Ding, Kaiyu Jiang, et al. Vlm as policy: Common-law content moderation framework for short video platform. arXiv preprint arXiv:2504.14904,

    arXiv

  10. [24]

    Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models

    Gen Luo, Yiyi Zhou, Yuxin Zhang, Xiawu Zheng, Xiaoshuai Sun, and Rongrong Ji. Feast your eyes: Mixture-of-resolution adaptation for multimodal large language models. arXiv preprint arXiv:2403.03003, 2024a. Run Luo, Haonan Zhang, Longze Chen, Ting-En Lin, Xiong Liu, Yuchuan Wu,...

    arXiv

  11. [25]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning

    Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203.10244,

    arXiv

  12. [27]

    Public domain 12m: A highly aesthetic image-text dataset with novel governance mechanisms

    Jordan Meyer, Nick Padgett, Cullen Miller, and Laura Exline. Public domain 12m: A highly aesthetic image-text dataset with novel governance mechanisms. arXiv preprint arXiv:2410.23144,

    arXiv

  13. [28]

    We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284,

    Runqi Qiao, Qiuna Tan, Guanting Dong, Minhui Wu, Chong Sun, Xiaoshuai Song, Zhuoma GongQue, Shanglin Lei, Zhe Wei, Miaoxuan Zhang, et al. We-math: Does your large multimodal model achieve human-like mathematical reasoning? arXiv preprint arXiv:2407.01284,

    arXiv

  14. [30]

    Zerobench: An impossible visual benchmark for contemporary large multimodal models

    Jonathan Roberts, Mohammad Reza Taesiri, Ansh Sharma, Akash Gupta, Samuel Roberts, Ioana Croitoru, Simion-Vlad Bogolin, Jialu Tang, Florian Langer, Vyas Raina, et al. Zerobench: An impossible visual benchmark for contemporary large multimodal models. arXiv preprint arXiv:2502.09696,

    arXiv

  15. [31]

    ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914,

  16. [32]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300,

    arXiv

  17. [33]

    Long-vita: Scaling large multi-modal models to 1 million tokens with leading short-context accuray

    Yunhang Shen, Chaoyou Fu, Shaoqi Dong, Xiong Wang, Peixian Chen, Mengdan Zhang, Haoyu Cao, Ke Li, Xiawu Zheng, Yan Zhang, et al. Long-vita: Scaling large multi-modal models to 1 million tokens with leading short-context accuray. arXiv preprint arXiv:2502.05177,

  18. [34]

    Openthinkimg: Learning to think with images via visual tool reinforcement learning

    Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, et al. Openthinkimg: Learning to think with images via visual tool reinforcement learning. arXiv preprint arXiv:2505.08617,

    arXiv

  19. [35]

    Robobrain 2.0 technical report

    BAAI RoboBrain Team. Robobrain 2.0 technical report. arXiv preprint arXiv:TODO, 2025a. Baidu ERNIE Team. Ernie 4.5 technical report, 2025b. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, K...

    arXiv

  20. [36]

    Gemini robotics: Bringing ai into the physical world

    Gemini Robotics Team, Saminda Abeyruwan, Joshua Ainslie, Jean-Baptiste Alayrac, Montserrat Gonzalez Arenas, Travis Armstrong, Ashwin Balakrishna, Robert Baruch, Maria Bauza, Michiel Blokzijl, et al. Gemini robotics: Bringing ai into the physical world. arXiv preprint arXiv:250...

    arXiv

  21. [37]

    Toloka visual question answering benchmark

    Dmitry Ustalov, Nikita Pavlichenko, Sergey Koshelev, Daniil Likhobaba, and Alisa Smirnova. Toloka visual question answering benchmark. arXiv preprint arXiv:2309.16511,

    arXiv

  22. [38]

    Measuring multimodal mathematical reasoning with math-vision dataset

    Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Houxing Ren, Aojun Zhou, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. Advances in Neural Information Processing Systems, 37:95095–95169, 2024a. Weiyun Wang, Zhe Chen, Wenhai Wa...

    arXiv

  23. [39]

    Logicvista: Multimodal llm logical reasoning benchmark in visual contexts

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts. arXiv preprint arXiv:2407.04973,

    arXiv

  24. [40]

    Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al

    URL https://arxiv.org/abs/2506.03569. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215, 2025a. Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wen...

    arXiv

  25. [42]

    Chun-Hsiao Yeh, Chenyu Wang, Shengbang Tong, Ta-Ying Cheng, Ruoyu Wang, Tianzhe Chu, Yuexiang Zhai, Yubei Chen, Shenghua Gao, and Yi Ma

    URL https: //arxiv.org/abs/2408.01800. Chun-Hsiao Yeh, Chenyu Wang, Shengbang Tong, Ta-Ying Cheng, Ruoyu Wang, Tianzhe Chu, Yuexiang Zhai, Yubei Chen, Shenghua Gao, and Yi Ma. Seeing from another perspective: Evaluating multi-view understanding in mllms. arXiv preprint arXiv:2...

    arXiv

  26. [43]

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al

    URL https://arxiv.org/abs/2501.07888. Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proce...

    arXiv

  27. [44]

    Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp

    Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Yu Qiao, et al. Mathverse: Does your multi-modal llm truly see the diagrams in visual math problems? In European Conference on Computer Vision, pp. 169–186. Springe...

    arXiv

  28. [45]

    thinking with images

    Ziwei Zheng, Michael Yang, Jack Hong, Chenxiao Zhao, Guohai Xu, Le Yang, Chao Shen, and Xing Yu. Deepeyes: Incentivizing" thinking with images" via reinforcement learning. arXiv preprint arXiv:2505.14362,

    arXiv

  29. [46]

    Onerec technical report

    Guorui Zhou, Jiaxin Deng, Jinghao Zhang, Kuo Cai, Lejian Ren, Qiang Luo, Qianqian Wang, Qigen Hu, Rui Huang, Shiyao Wang, et al. Onerec technical report. arXiv preprint arXiv:2506.13695,

  30. [47]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models

    27 Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504.10479,

    arXiv

  31. [48]

    Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models

    Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836,

    arXiv

  32. [49]

    likes" a video receives within a specific timeframe after being uploaded. Using a predetermined threshold, we classify videos into two categories:

    28 A Strategies for Data Decontamination A.1 Pre-training To prevent benchmark leakage (i.e., the model memorizes evaluation examples), we conduct rigorous data decontamination strategies to avoid the data leakage problem. In pre-training, considering the large-scale data volu...

    2024

  33. [2014]

    doi: 10.3115/v1/D14-1086

    Association for Computational Linguistics. doi: 10.3115/v1/D14-1086. URL https://aclanthology.org/D14-1086. Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision–ECCV 2016: 14th Eu...

    doi

  34. [2017]

    URL https://doi.org/10.1007/ s11263-016-0981-7

    doi: 10.1007/s11263-016-0981-7. URL https://doi.org/10.1007/ s11263-016-0981-7 . Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pag...

    doi

  35. [2020]

    Eve: Efficient multimodal vision language models with elastic visual experts

    Miao Rang, Zhenni Bi, Chuanjian Liu, Yehui Tang, Kai Han, and Yunhe Wang. Eve: Efficient multimodal vision language models with elastic visual experts. arXiv preprint arXiv:2501.04322,

    arXiv

  36. [2021]

    Taskgalaxy: Scaling multi-modal instruction fine-tuning with tens of thousands vision task types

    Jiankang Chen, Tianke Zhang, Changyi Liu, Haojie Ding, Yaya Shi, Feng Cheng, Huihui Xiao, Bin Wen, Fan Yang, Tingting Gao, et al. Taskgalaxy: Scaling multi-modal instruction fine-tuning with tens of thousands vision task types. arXiv preprint arXiv:2502.09925,

    arXiv

  37. [2022]

    Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning

    Fanqing Meng, Lingxiao Du, Zongkai Liu, Zhixiang Zhou, Quanfeng Lu, Daocheng Fu, Tiancheng Han, Botian Shi, Wenhai Wang, Junjun He, et al. Mm-eureka: Exploring the frontiers of multimodal reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2503.07365,

    arXiv

  38. [2023]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. arXiv preprint arXiv:2409.17146,

    arXiv

  39. [2024]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923,

    arXiv

  40. [2025]

    Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024a

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision-language models? arXiv preprint arXiv:2403.20330, 2024a. Tao Chen, Enwei Zhang, Yuting Gao, Ke Li, Xi...

    arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.