Pith. sign in

REVIEW 2 major objections 2 minor 68 cited by

LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts

T0 review · 2 major / 2 minor · reviewed 2026-05-16 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LogicVista provides a benchmark of 448 visual questions to evaluate logical reasoning in multimodal LLMs across five tasks and nine capabilities.

desk verdict LogicVista adds a targeted benchmark for visual logical reasoning in MLLMs with 448 annotated questions, but lacks enough detail on question construction to confirm broad coverage. read the letter →

arxiv 2407.04973 v1 pith:LC2EMAER submitted 2024-07-06 cs.AI cs.CLcs.CVcs.LG

Yijia Xiao , Edward Sun , Tianyu Liu , Wei Wang This is my paper · ORCID
classification cs.AIcs.CLcs.CVcs.LG
keywords LogicVistamultimodalLLMslogicalreasoningvisualcontextsbenchmarkevaluationcapabilitiesMLLM
checked against Cost.FunctionalEquationFoundation.DimensionForcing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces LogicVista to address the lack of systematic tests for logical reasoning in multimodal large language models when images are present. It covers five tasks that together test nine capabilities through 448 multiple-choice questions, each paired with a correct answer and human-written reasoning steps. This setup allows evaluation in both multiple-choice and open-ended formats. A sympathetic reader would care because logical reasoning combined with visuals underpins activities such as navigation and puzzle-solving, skills that current model assessments largely ignore. The authors test eight existing MLLMs on the benchmark and release the questions and annotations.

What carries the argument

The LogicVista benchmark, a set of 448 image-based multiple-choice questions with human reasoning annotations that test logical cognition in visual settings.

What would settle it

A demonstration that high-scoring models on LogicVista fail at comparable logical tasks with new images or real-world visual scenarios would show the benchmark does not measure general visual-logic abilities.

Watch

Extended reading notes

Core claim

LogicVista assesses the integrated logical reasoning capabilities of MLLMs in visual contexts across 5 logical reasoning tasks encompassing 9 different capabilities using a sample of 448 multiple-choice questions. Each question is annotated with the correct answer and the human-written reasoning behind the selection, enabling both open-ended and multiple-choice evaluation.

Load-bearing premise

The 448 questions and their human-written reasoning annotations accurately and comprehensively capture general logical cognition abilities in visual contexts without significant selection bias or coverage gaps.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 2 minor

Summary. The paper introduces LogicVista, a new benchmark for assessing the integrated logical reasoning capabilities of multimodal large language models (MLLMs) in visual contexts. It consists of 448 multiple-choice questions spanning 5 logical reasoning tasks that together cover 9 distinct capabilities. Each question includes the correct answer and human-written reasoning annotations to support both multiple-choice and open-ended evaluation. The authors evaluate 8 MLLMs on the benchmark and release the code and data.

Significance. If the questions are shown to be representative and free of major selection bias, LogicVista would fill a clear gap by providing a systematic visual-context benchmark for logical reasoning, an area where current MLLM evaluations remain limited. The release of code, data, and human reasoning annotations is a clear strength that supports reproducibility and further research.

major comments (2)
  1. [Benchmark Construction] The central claim that the 448 questions comprehensively cover the 9 capabilities across 5 tasks without significant selection bias or gaps is not supported by sufficient methodological detail. The manuscript provides no quantitative breakdown (e.g., number of questions per capability or task), sampling strategy, visual diversity metrics, or validation against external logical-reasoning taxonomies in the benchmark-construction section.
  2. [Annotation Process] The human-written reasoning annotations are presented as enabling reliable open-ended evaluation, yet no inter-annotator agreement statistics or validation procedure for these annotations are reported, which is load-bearing for claims about the benchmark's utility beyond multiple-choice accuracy.
minor comments (2)
  1. [Abstract and §3] The abstract states 'a sample of 448 multiple-choice questions' but does not clarify whether this is the full benchmark size or a subset; this should be stated explicitly in the main text.
  2. [Figures and Tables] Figure captions and table headers should explicitly list the 5 tasks and 9 capabilities to improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on LogicVista. We agree that additional methodological details are needed to support the claims of comprehensive coverage and annotation reliability. We address each major comment below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Benchmark Construction] The central claim that the 448 questions comprehensively cover the 9 capabilities across 5 tasks without significant selection bias or gaps is not supported by sufficient methodological detail. The manuscript provides no quantitative breakdown (e.g., number of questions per capability or task), sampling strategy, visual diversity metrics, or validation against external logical-reasoning taxonomies in the benchmark-construction section.

    Authors: We acknowledge that the current manuscript lacks these details in the benchmark-construction section. In the revised version, we will expand this section to include: a table reporting the exact number of questions per task and per capability (totaling 448), a description of the sampling strategy (stratified selection to ensure balanced coverage of the 9 capabilities without over-representation), quantitative visual diversity metrics (e.g., distribution across image sources, types, and complexity levels), and explicit mapping/validation against established logical-reasoning taxonomies from cognitive science to demonstrate coverage and minimize gaps or bias. revision: yes

  2. Referee: [Annotation Process] The human-written reasoning annotations are presented as enabling reliable open-ended evaluation, yet no inter-annotator agreement statistics or validation procedure for these annotations are reported, which is load-bearing for claims about the benchmark's utility beyond multiple-choice accuracy.

    Authors: We agree that inter-annotator agreement statistics and validation details are necessary to substantiate the reliability of the human-written reasoning annotations for open-ended evaluation. In the revision, we will add a dedicated subsection describing the annotation process (including annotator qualifications and guidelines), report inter-annotator agreement metrics (e.g., Fleiss' kappa across reasoning steps), and outline the validation procedure (e.g., review rounds for consistency and accuracy). This will directly support the benchmark's utility claims. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: direct benchmark construction and evaluation with no derivations or self-referential steps

full rationale

The paper introduces LogicVista as a new benchmark dataset of 448 multiple-choice questions with human annotations, then directly evaluates 8 MLLMs on it across 5 tasks and 9 capabilities. No equations, parameter fitting, predictions derived from inputs, or load-bearing self-citations appear in the provided text. The central claim rests on the explicit creation and application of the dataset rather than any reduction of results to prior fitted values or self-defined constructs. This is a standard empirical benchmark paper with no mathematical derivation chain to inspect for circularity.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The central claim rests on the assumption that the chosen tasks and questions validly measure logical reasoning abilities; no free parameters or invented entities are introduced.

assumptions (1)
  • domain assumption The five logical reasoning tasks and nine capabilities adequately represent general logical cognition in visual contexts.
    Invoked in the abstract when defining the benchmark scope without further justification or validation details.

how reviews work

0 comments
Cite this review

Pith. "Pith review of LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts." pith.science (2026). https://pith.science/paper/LC2EMAER

@misc{pith2026240704973,
  author       = {Pith},
  title        = {Pith review of: LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LC2EMAER}},
  note         = {Machine review of arXiv:2407.04973}
}
read the original abstract

We propose LogicVista, an evaluation benchmark that assesses the integrated logical reasoning capabilities of multimodal large language models (MLLMs) in Visual contexts. Recent advancements in MLLMs have demonstrated various fascinating abilities, from crafting poetry based on an image to performing mathematical reasoning. However, there is still a lack of systematic evaluation of MLLMs' proficiency in logical reasoning tasks, which are essential for activities like navigation and puzzle-solving. Thus we evaluate general logical cognition abilities across 5 logical reasoning tasks encompassing 9 different capabilities, using a sample of 448 multiple-choice questions. Each question is annotated with the correct answer and the human-written reasoning behind the selection, enabling both open-ended and multiple-choice evaluation. A total of 8 MLLMs are comprehensively evaluated using LogicVista. Code and Data Available at https://github.com/Yijia-Xiao/LogicVista.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 68 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 68 Pith citations

  1. PRISM: Priority-aware Rubric Internalization via Structured Multimodal Data Synthesis

    cs.LG 2026-08 conditional novelty 7.0 of 10

    PRISM trains MLLMs to act as rubric executors by synthesizing typed, prioritized rules and verification traces, lifting Qwen3-VL-4B from 9.5% to 30.1% Strict accuracy on the authors' PRISM-Eval benchmark.

  2. Reason Twice: Segmentation via Candidate Discovery and Comparative Reasoning

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Rea2Seg turns image segmentation into candidate mask discovery from MLLM attention followed by MLLM-based comparative scoring and selection, plus a new multi-dimensional reasoning benchmark ReasonSeg-SGDR.

  3. DeepLatent: Think with Images via Parallel Latent Visual Reasoning

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    DeepLatent introduces a parallel latent visual reasoning framework with learnable 2D tokens and continuous RL, trained via distillation then RL, plus a new 180K dataset, claiming SOTA benchmark results.

  4. Are Tools Always Beneficial? Learning to Invoke Tools Adaptively for Dual-Mode Multimodal LLM Reasoning

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    AutoTool uses reinforcement learning with dual-mode rewards to train multimodal LLMs to adaptively choose between tool-assisted and text-centric reasoning, yielding accuracy and efficiency gains on V* and POPE benchmarks.

  5. CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization

    cs.LG 2026-05 conditional novelty 7.0 of 10

    CEPO sharpens token credit in RLVR by requiring tokens to be favored by the correct answer and disfavored by wrong answers drawn from rejected rollouts, delivering accuracy gains on five multimodal math benchmarks.

  6. Beyond the Eye: Efficient Multimodal Reasoning via Self-Regulated Implicit Visual Tools

    cs.CV 2026-07 accept novelty 6.5 of 10

    Self-regulated implicit visual tools let MLLMs match or beat external-tool and latent-reasoning methods on high-resolution perception while cutting inference latency by up to 86%.

  7. CodePercept: Code-Grounded Visual STEM Perception for MLLMs

    cs.CV 2026-03 conditional novelty 6.5 of 10

    Perception, not reasoning, is the main bottleneck for MLLM STEM visual reasoning, and training on executable reconstruction code measurably fixes it.

  8. Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

    cs.CV 2026-08 conditional novelty 6.0 of 10

    A training framework that adds a removable image-generation branch to multimodal LLMs improves visual understanding benchmarks with zero inference-time cost.

  9. Remember-R1: Mitigating Long-Context Visual Forgetting through Reinforcement Learning

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Remember-R1 adds three process-level rewards to GRPO, encouraging keyword coverage, sustained visual attention, and focus on question-relevant regions, improving MLLM reasoning across seven benchmarks.

  10. Correcting What You Cannot See: Credit Assignment for Perception Distillation in Multimodal Reasoners

    cs.AI 2026-07 conditional novelty 6.0 of 10

    Perception-Correction Distillation up-weights teacher supervision on perception spans only when low downstream success and high teacher–student disagreement co-occur, lifting 8B→2B and 32B→8B multimodal averages over ...

  11. PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Frontier MLLMs remain far from mastering atomic visual perception: none reach 60% on a failure-derived, perception-only benchmark of ten capabilities.

  12. Scaling Native Multimodal Pre-Training From Scratch

    cs.CL 2026-07 conditional novelty 6.0 of 10

    In models trained from scratch on text plus images, the text-objective scaling law is data-mix-invariant while the image-conditioned objective shifts toward many more tokens relative to parameters as the multimodal sh...

  13. Trace: A Taxonomy-Guided Environment for Multidomain Visual Reasoning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    RLVR training on 64,000 procedurally generated Trace instances improves Qwen2.5-VL macro-average on 24 external visual reasoning benchmarks by 3.51 points at 3B and 4.06 points at 7B.

  14. Contrastive On-Policy Distillation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Contrasting a teacher's token scores under light- and heavy-thinking prompts yields a training signal that cuts reasoning length roughly in half without hurting accuracy.

  15. SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

    cs.CV 2026-07 conditional novelty 6.0 of 10

    SIVA-RL uses the observed reward drop between clean and locally edited images to route training toward sensitivity or invariance, improving GRPO/DAPO-based multimodal RL across nine benchmarks.

  16. H-OPD: Confidence Aware Heterogeneous Multi-Teacher Multimodal On-policy Distillation

    cs.CV 2026-07 conditional novelty 6.0 of 10

    Token-level confidence arbitration between VL and text teachers during on-policy distillation improves multimodal student reasoning over single-teacher OPD and GRPO.

  17. TACO: Tool-Augmented Credit Optimization for Agentic Tool Use

    cs.MA 2026-06 unverdicted novelty 6.0 of 10

    TACO combines Differential Answer-Probe Reward (DAPR) and Outcome-Gated Advantage Routing (OGAR) to assign credit to tool calls in agentic visual reasoning, producing accuracy gains on multimodal benchmarks.

  18. CFPO: Counterfactual Policy Optimization for Multimodal Reasoning

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    CFPO is a counterfactual policy optimization method that regularizes RL policies in LVLMs by maximizing prediction discrepancy under suppressed visual cues, reporting 3-6% gains over baselines.

  19. MMGist: A Comprehensive Multimodal Benchmark for 2027

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    MMGist filters 23,250 items from 18 benchmarks down to 7,262 using three-stage pipeline, preserving model rankings (Spearman ρ=0.98) while cutting items 69% and raising discrimination 78%.

  20. LOCUS: Local Visual Cue Search for Enhancing Fine-Grained Perception in Multimodal Large Language Models

    cs.CV 2026-06 conditional novelty 6.0 of 10

    Training MLLMs to locate a local crop in the full image with an IoU-based reward internalizes visual evidence search and improves fine-grained VQA at inference.

  21. Teaching the Way, Not the Answer: Privileged Tutoring Distillation for Multimodal Policy Optimization

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    PTD-PO supplies step-wise token-distribution supervision to student policies via in-context privileged hints derived from spatial attention and intermediate reasoning, while keeping the student in an answer-free conte...

  22. Do Multimodal Agents Really Benefit from Tool Use? A Systematic Study of Capability Gains

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    Tool use adds minimal new problem-solving ability in the tested multimodal agents, as 93-96% of tool-solved cases are also solved without tools.

  23. TRON: Targeted Rule-Verifiable Online Environments for Visual Reasoning RL

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    TRON supplies 520 rule-verifiable online visual reasoning environments across five ability buckets that generate unlimited training instances for RL post-training, yielding consistent gains on ten external multimodal ...

  24. Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    EASE augments multimodal RLVR with evidence-anchored spatial attention supervision using privileged annotations, improving average benchmark scores by 2.5-3.1 points over DAPO on Qwen VL models.

  25. Guidance Contrastive Token Credit Assignment for Discrete Policy Optimization

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    GCPO performs per-token credit assignment in discrete policy optimization by setting token advantages proportional to the difference in model predictions under positive versus negative prompts, outperforming GRPO and ...

  26. AnE: Pushing the Reasoning Frontier of Multimodal LLMs via Anchor Evolution

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    AnE combines Truth Anchor Expansion and Scaffold-Stripping to deliver 10.3% gains on eight multimodal reasoning benchmarks for MLLMs.

  27. Beyond Mode Collapse: Distribution Matching for Diverse Reasoning

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    DMPO approximates forward KL minimization in on-policy RL by aligning the policy to a group-level reward-proportional target distribution, yielding 9-12% relative gains over GRPO on NP-Bench and smaller gains on math ...

  28. Visually-Guided Policy Optimization for Multimodal Reasoning

    cs.CV 2026-04 unverdicted novelty 6.0 of 10

    VGPO introduces visual attention compensation and dual-grained advantage re-weighting to reinforce visual focus in VLMs, yielding better activation and performance on multimodal reasoning tasks.

  29. MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphs

    cs.LG 2026-02 unverdicted novelty 6.0 of 10

    MapTab is a new multimodal benchmark with 328 images and nearly 200k queries that shows current MLLMs have substantial difficulty with multi-criteria route planning when visual and tabular information must be combined.

  30. Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models

    cs.CV 2026-02 unverdicted novelty 6.0 of 10

    ReAlign corrects the modality gap in unpaired data to let MLLMs learn visual distributions from text alone before instruction tuning, reducing dependence on expensive paired corpora.

  31. ChartVerse: Scaling Chart Reasoning via Reliable Programmatic Synthesis from Scratch

    cs.CV 2026-01 conditional novelty 6.0 of 10

    ChartVerse uses Rollout Posterior Entropy and truth-anchored inverse QA synthesis to produce 640K high-quality chart reasoning samples, training an 8B model that surpasses its 30B teacher.

  32. SPHINX: A Synthetic Environment for Visual Perception and Reasoning

    cs.CV 2025-11 unverdicted novelty 6.0 of 10

    SPHINX generates synthetic visual puzzles for benchmarking LVLMs, where GPT-5 scores 51.1% and RLVR training improves both in-domain and external visual reasoning performance.

  33. Boosting Reasoning in Large Multimodal Models via Activation Replay

    cs.CV 2025-11 unverdicted novelty 6.0 of 10

    Activation Replay boosts multimodal reasoning in post-trained LMMs by replaying low-entropy activations from base models to RLVR counterparts at test time via visual token manipulation.

  34. DeepEyesV2: Toward Agentic Multimodal Model

    cs.CV 2025-11 unverdicted novelty 6.0 of 10

    DeepEyesV2 uses a two-stage cold-start plus reinforcement learning pipeline to produce an agentic multimodal model that adaptively invokes tools and outperforms direct RL on real-world reasoning benchmarks.

  35. VOLD: Reasoning Transfer from LLMs to Vision-Language Models via On-Policy Distillation

    cs.CV 2025-10 conditional novelty 6.0 of 10

    VOLD transfers text-only LLM reasoning to a 3B VLM via teacher-trace SFT followed by GRPO plus on-policy distillation, improving visual-math accuracy over GRPO-alone and prior text-only baselines (e.g., MathVision 28....

  36. Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Framework

    cs.CV 2025-09 unverdicted novelty 6.0 of 10

    DRP decouples reasoning from perception in LMMs by using an LLM reasoner to query an LMM observer for visual details as needed, reducing visual grounding loss.

  37. Dual Knowledge-Enhanced Two-Stage Reasoner for Multimodal Dialog Systems

    cs.CL 2025-09 conditional novelty 6.0 of 10

    A two-stage LLM system that retrieves structured attributes and user reviews, filters the useful knowledge type via LLM probes, and separates intent reasoning from response generation outperforms prior methods on MMConv.

  38. Kwai Keye-VL 1.5 Technical Report

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Keye-VL-1.5 combines similarity-based Slow-Fast video token allocation with progressive context extension and iterative RL, reporting leading video-understanding results among 8B-scale multimodal models.

  39. R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    R-4B uses bi-mode annealing plus a GRPO variant with forced thinking and non-thinking rollouts so a 4B multimodal model adaptively picks when to reason, reportedly matching 16B thinking models on several reasoning benchmarks.

  40. Vision-G1: Towards General Vision Language Reasoning with Multi-Domain Data Curation

    cs.CV 2025-08 reject novelty 6.0 of 10

    Vision-G1 trains a 7B VLM with multi-round GRPO on an influence-filtered, difficulty-curated 46-source dataset spanning eight visual reasoning domains.

  41. MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models

    cs.CL 2025-07 conditional novelty 6.0 of 10

    A new 2,700-task benchmark with budget, time, and distance constraints shows that even the best multimodal LLMs produce feasible plans less than 22% of the time.

  42. VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoning

    cs.CV 2025-07 conditional novelty 6.0 of 10

    VL-Cogito, trained with progressive curriculum RL, online difficulty weighting, and dynamic length rewards, matches or beats prior reasoning MLLMs on ten multimodal benchmarks.

  43. Position: Reasoning After Perception Means Reasoning Without Vision

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Reasoning in text space cannot recover visual information that was collapsed during perception, so multimodal models need architectures that reason within the visual representation.

  44. Perception-Aware Policy Optimization for Multimodal Reasoning

    cs.CL 2025-07 unverdicted novelty 6.0 of 10

    PAPO integrates perception-aware supervision via a KL-based loss into RLVR methods like GRPO, yielding 4.4-17.5% gains on multimodal benchmarks and 30.5% fewer perception errors, with larger gains on vision-heavy tasks.

  45. Kwai Keye-VL Technical Report

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.

  46. OIBench: Benchmarking Strong Reasoning Models with Olympiad in Informatics

    cs.AI 2025-06 conditional novelty 6.0 of 10

    A private, contamination-resistant benchmark of 250 olympiad-level programming problems shows top reasoning models reaching about 36% solve rates, far above conventional models.

  47. ZeroBench: An Impossible Visual Benchmark for Contemporary Large Multimodal Models

    cs.CV 2025-02 conditional novelty 6.0 of 10

    ZeroBench is a hand-built 100-question visual reasoning benchmark, adversarially filtered so every evaluated frontier LMM scored 0% at release.

  48. Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimization

    cs.CL 2024-11 conditional novelty 6.0 of 10

    Mixed Preference Optimization with the MMPR dataset boosts multimodal CoT reasoning, lifting InternVL2-8B to 67.0 accuracy on MathVista (+8.7 points) and matching the 76B model.

  49. DecoupleMix: Decoupled Ratio Search and Convex Allocation for Scalable VLM Data Recipes

    cs.CV 2026-07 conditional novelty 5.0 of 10

    Decoupling inter-class ratio search from intra-class convex allocation yields VLM data recipes that beat stacking and transfer from small proxies to larger scales.

  50. LASER: A Corrective Lens for LVLMs via Visual Attention Preservation and Sink Suppression

    cs.CV 2026-07 unverdicted novelty 5.0 of 10

    LASER uses Visual Grounding Reward and Sink Suppression Reward to preserve visual attention trajectories and suppress sink tokens, reducing visual forgetting in LVLMs.

  51. DOPD: Dual On-policy Distillation

    cs.AI 2026-06 unverdicted novelty 5.0 of 10

    DOPD is an advantage-aware dual distillation method that dynamically assigns token supervision from either privileged teacher or student to transfer capability while mitigating non-replicable information asymmetry in ...

  52. TuringViT: Making SOTA Vision Transformers Accessible to All

    cs.CV 2026-06 unverdicted novelty 5.0 of 10

    TuringViT uses Turing Linear Attention, VISTA-Curation, and dynamic-resolution pretraining to outperform open ViT baselines with 10% data while improving VLM performance and high-resolution latency.

  53. Stabilizing On-Policy Distillation for MLLM Reasoning with Global Normalization

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    GNDPO applies batch-level normalization to KL divergences in on-policy distillation to reduce gradient instability and improve robustness on multimodal reasoning tasks.

  54. Reversing the Flow: Generation-to-Understanding Synergy in Large Multimodal Models

    cs.CV 2026-05 unverdicted novelty 5.0 of 10

    Generation-to-Understanding synergy lets multimodal models create self-generated visual edits as intermediate steps, improving performance on twelve benchmarks while revealing limits in task-aligned self-reflection.

  55. LanteRn: Latent Visual Structured Reasoning

    cs.CV 2026-03 conditional novelty 5.0 of 10

    A 3B vision-language model trained to emit latent visual thought tokens interleaved with text, then refined by reinforcement learning, outperforms a matched text-only baseline on several visual reasoning benchmarks.

  56. Limits of Spatial Imagery Reasoning in Frontier LLM Models

    cs.CV 2026-03 unverdicted novelty 5.0 of 10

    An external 3D imagery tool does not fix frontier MLLM mental-rotation performance (≤62.5%), revealing missing visual-spatial primitives rather than a pure working-memory limit.

  57. MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Models

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    A new benchmark of 2,000 K-12 math questions photographed in real scenes finds that multimodal LLMs perform significantly worse on authentic handheld images than on clean benchmarks.

  58. VRPRM: Process Reward Modeling via Visual Reasoning

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    VRPRM combines visual reasoning with a two-stage SFT-plus-RL strategy to deliver higher-quality process reward modeling using far less annotated data than prior non-thinking PRMs.

  59. Skywork-R1V3 Technical Report

    cs.CL 2025-07 conditional novelty 5.0 of 10

    A 38B open-source VLM reaches 76.0% on MMMU using RL post-training and connector-only tuning, with a critical-token entropy metric for checkpoint selection.

  60. WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning

    cs.CV 2025-06 conditional novelty 5.0 of 10

    WeThink, a 120K-image QA dataset with AI-generated reasoning paths, combined with hybrid-reward reinforcement learning, improves a 7B vision-language model across 14 benchmarks.

See all 68 Pith citations

Reference graph

Works this paper leans on

58 extracted references · 58 canonical work pages · cited by 68 Pith papers (see all)

  1. [1]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner...

    work page 2024

  2. [2]

    Flamingo: a visual language model for few-shot learning, 2022

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski,...

    work page 2022

  3. [4]

    Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023

    work page 2023

  4. [5]

    A survey on multimodal large language models, 2023

    Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models, 2023

    work page 2023

  5. [6]

    Mme: A comprehensive evaluation benchmark for multimodal large language models, 2023

    Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2023

    work page 2023

  6. [7]

    Pmc-vqa: Visual instruction tuning for medical visual question answering, 2023

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weixiong Lin, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-vqa: Visual instruction tuning for medical visual question answering, 2023

    work page 2023

  7. [8]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015

    work page 2015

  8. [10]

    Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2023

    work page 2023

Show all 58 references
  1. [11]

    Wavering

    Michael J. Wavering. Logical reasoning necessary to make line graphs. Journal of Research in Science Teaching, 26(5):373–379, May 1989

    1989

  2. [12]

    Somerville

    Catherine Sophian and Susan C. Somerville. Early developments in logical reasoning: Considering alternative possibilities. Cognitive Development, 3(2):183–222, 1988

    1988

  3. [13]

    Logical reasoning in formal and everyday reasoning tasks - international journal of science and mathematics education, Dec 2019

    Hugo Bronkhorst, Gerrit Roorda, Cor Suhre, and Martin Goedhart. Logical reasoning in formal and everyday reasoning tasks - international journal of science and mathematics education, Dec 2019

    2019

  4. [14]

    Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts, 2024

    2024

  5. [15]

    Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2017

    2017

  6. [16]

    Lawrence Zitnick

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context , page 740–755. Springer International Publishing, 2014. 13

    2014

  7. [17]

    Textcaps: a dataset for image captioning with reading comprehension, 2020

    Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020

    2020

  8. [18]

    Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models, 2024

    Rohan Wadhawan, Hritik Bansal, Kai-Wei Chang, and Nanyun Peng. Contextual: Evaluating context-sensitive text-rich visual reasoning in large multimodal models, 2024

    2024

  9. [19]

    Visit-bench: A benchmark for vision-language instruction following inspired by real-world use, 2023

    Yonatan Bitton, Hritik Bansal, Jack Hessel, Rulin Shao, Wanrong Zhu, Anas Awadalla, Josh Gardner, Rohan Taori, and Ludwig Schmidt. Visit-bench: A benchmark for vision-language instruction following inspired by real-world use, 2023

    2023

  10. [20]

    Lawrence Zitnick

    Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server, 2015

    2015

  11. [21]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering, 2017

    2017

  12. [22]

    Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and-language tasks, 2019

    Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguis- tic representations for vision-and-language tasks, 2019

    2019

  13. [23]

    Uniter: Universal image-text representation learning, 2020

    Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning, 2020

    2020

  14. [24]

    Oscar: Object-semantics aligned pre-training for vision-language tasks, 2020

    Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision-language tasks, 2020

    2020

  15. [25]

    Vilt: Vision-and-language transformer without convolution or region supervision, 2021

    Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-language transformer without convolution or region supervision, 2021

    2021

  16. [26]

    Simvlm: Simple visual language model pretraining with weak supervision, 2022

    Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision, 2022

    2022

  17. [27]

    Git: A generative image-to-text transformer for vision and language, 2022

    Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language, 2022

    2022

  18. [28]

    Unitab: Unifying text and box outputs for grounded vision-language modeling, 2022

    Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling, 2022

    2022

  19. [29]

    Vision-language pre-training: Basics, recent advances, and future trends, 2022

    Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, and Jianfeng Gao. Vision-language pre-training: Basics, recent advances, and future trends, 2022

    2022

  20. [30]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff...

    2020

  21. [31]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vino...

    2022

  22. [32]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language...

    2023

  23. [33]

    Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models, 2021

    2021

  24. [34]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint...

    2023

  25. [35]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. O...

    2022

  26. [36]

    Instruction tuning with gpt-4, 2023

    Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, 2023

    2023

  27. [37]

    Openflamingo: An open-source framework for training large autoregressive vision-language models, 2023

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. Openflamingo: An open-source framew...

    2023

  28. [38]

    Visual instruction tuning, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023

    2023

  29. [39]

    Otter: A multi-modal model with in-context instruction tuning, 2023

    Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning, 2023

    2023

  30. [40]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    2023

  31. [41]

    Multimodal-gpt: A vision and language model for dialogue with humans, 2023

    Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multimodal-gpt: A vision and language model for dialogue with humans, 2023

    2023

  32. [42]

    mplug-owl: Modularization empowers large language models with multimodality, 2023

    Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qian Qi, Ji Zhang, and Fei Huang. mplug-owl: Modularization empowers large language models with multimodal...

    2023

  33. [43]

    Mm-react: Prompting chatgpt for multimodal reasoning and action, 2023

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompting chatgpt for multimodal reasoning and action, 2023

    2023

  34. [44]

    Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. 15

    2023

  35. [45]

    Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn, 2023

    Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. Assistgpt: A general multi-modal assistant that can plan, execute, inspect, and learn, 2023

    2023

  36. [46]

    nocaps: novel object captioning at scale

    Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV) . IEEE, October 2019

    2019

  37. [47]

    Towards vqa models that can read, 2019

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read, 2019

    2019

  38. [48]

    Tap: Text-aware pre-training for text-vqa and text-caption, 2020

    Zhengyuan Yang, Yijuan Lu, Jianfeng Wang, Xi Yin, Dinei Florencio, Lijuan Wang, Cha Zhang, Lei Zhang, and Jiebo Luo. Tap: Text-aware pre-training for text-vqa and text-caption, 2020

    2020

  39. [49]

    From recognition to cognition: Visual commonsense reasoning, 2019

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning, 2019

    2019

  40. [50]

    Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge, 2019

    2019

  41. [51]

    Mmbench: Is your multi-modal model an all-around player?, 2023

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2023

    2023

  42. [52]

    Can large language models be an alternative to human evaluations?, 2023

    Cheng-Han Chiang and Hung yi Lee. Can large language models be an alternative to human evaluations?, 2023

    2023

  43. [53]

    G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023

    2023

  44. [54]

    Gptscore: Evaluate as you desire, 2023

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. Gptscore: Evaluate as you desire, 2023

    2023

  45. [55]

    Mm-soc: Benchmarking multimodal large language models in social media platforms

    Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, and Srijan Kumar. Mm-soc: Benchmarking multimodal large language models in social media platforms. In ACL, 2024

    2024

  46. [56]

    Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities

    Mina Lee, Percy Liang, and Qian Yang. Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities. In CHI Conference on Human Factors in Computing Systems, CHI ’22. ACM, April 2022

    2022

  47. [57]

    Zhang, Mark Harman, and Meng Wang

    Shuyin Ouyang, Jie M. Zhang, Mark Harman, and Meng Wang. Llm is like a box of chocolates: the non-determinism of chatgpt in code generation, 2023

    2023

  48. [58]

    Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024

    2024

  49. [59]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023

    2023

  50. [60]

    spending on IT hardware will decline

    Kenton Lee, Mandar Joshi, Iulia Turc, Hexiang Hu, Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding, 2023. 16 Appendix: LogicVista: Multimodal ...

    2023

Pith tools

Reviewed May 16, 2026 · model on record in the stance chip above.