Pith. sign in

REVIEW 3 major objections 4 minor 45 references

MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MPCC benchmark shows state-of-the-art multimodal models rarely produce feasible plans under budget, time, and distance constraints.

desk verdict A useful benchmark for constrained multimodal planning, but the article's headline claim about constraint-complexity sensitivity is not established because difficulty levels confound constraint complexity with search-space size. read the letter →

arxiv 2507.23382 v1 pith:DHSE2TE5 submitted 2025-07-31 cs.CL cs.AIcs.CV

Yiyan Ji , Haoran Chen , Qiguang Chen , Chengyue Wu , Libo Qin , Wanxiang Che This is my paper
classification cs.CLcs.AIcs.CV
keywords multimodallargelanguagemodelsplanningbenchmarkconstraintsconstraintsatisfactionflightcalendarmeetingevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

MPCC is a benchmark built to answer one question: can multimodal large language models produce plans that actually satisfy the constraints a real itinerary or meeting requires? It contains 2,700 tasks in flight, calendar, and meeting planning, each requiring the model to read budget limits, time windows, and distances from a mix of images and text, with difficulty graded from EASY to HARD. The headline result is that they mostly cannot: the best closed-source model reaches 21.3% feasible plans on average, open-source models average under 11%, and on HARD Meeting Planning the best model scores 2.0%. The paper also reports that Chain-of-Thought and Plan-and-Solve prompting help only on easy low-constraint tasks, while text-only in-context learning actively hurts. The paper's contribution is to formalize multimodal constraints in planning and to provide a benchmark where constraint satisfaction, not just perception or next-step prediction, is the thing being measured.

What carries the argument

The central object is the MPCC benchmark itself, built from three task generators and a formal constraint vocabulary. A plan must satisfy a budget inequality ($\sum_j \mathrm{cost}(r_j\mid p)\le B_{\max}$), a temporal condition either sequential ($\sum_i\Delta t_i\le T_{\max}$ with $\Delta t_i\ge t_{\min}$) or concurrent ($\bigcap_i A_i\neq\emptyset$), and a spatial bound ($d(l(p_i),l_{\text{meet}})\le D_i$). Difficulty levels are created by varying the number of transit cities, planning days, venue availability, and schedule resolution, which grows the average search space from 27 to 617; brute-force search over that space provides ground truth. Two metrics are defined: feasible plan rate (all constraints satisfied) and optimal plan rate (feasible and budget-minimizing). The machinery works by making the constraint satisfaction itself the measured quantity, with sparse feasible regions (average 38 feasible plans per instance) that force genuine constraint reasoning.

What would settle it

Hold the size of the search space fixed--for example, the same set of flight combinations or meeting rooms--and vary only the number and type of constraints; if feasible-plan rates do not change across such matched pairs, the paper's central claim that constraint complexity drives MLLM failure would be undercut, since the observed EASY-to-HARD drops would then track search-space expansion instead.

Watch

Extended reading notes

Core claim

The paper claims that MPCC is the first benchmark to systematically test whether MLLMs can handle composite multimodal constraints in planning, and that the answer so far is largely no. The benchmark formalizes three constraint types--budget, temporal, and spatial--and combines them into three real-world tasks: Flight Planning (sequential temporal plus budget), Calendar Planning (concurrent temporal plus budget), and Meeting Planning (temporal plus spatial plus budget). Every instance is generated with a brute-force-verified optimal solution, and human rechecking ensures constraints are aligned across modalities and at least one feasible plan exists. Evaluated on 13 models, GPT-4o, Gemini-2.0-Flash, and Claude-3.5V-Sonnet average 16.7%, 14.6%, and 21.3% feasible plans respectively, while open-source models average below 11%; on HARD Meeting Planning, Claude-3.5V-Sonnet falls to 2.0% feasible plans, at or below random expectations. The paper further claims MLLMs are highly sensitive to constraint complexity and that standard prompting strategies fail in multi-constraint scenarios.

Load-bearing premise

The claim that MLLMs are highly sensitive to constraint complexity rests on the benchmark's difficulty levels isolating constraint complexity from search-space expansion, but the construction changes transit cities, planning days, venue availability, and schedule resolution together, so the levels may also differ in combinatorial size.

Editorial extensions

If this is right

  • Planning benchmarks that measure unconstrained or loosely constrained tasks overstate real MLLM planning ability; adding hard budget, temporal, and spatial constraints is what separates current models from usable planners.
  • Scaling model size improves the ratio of optimal to feasible plans, so larger MLLMs may be part of the path, but the best model still lands near zero on HARD meeting planning, so scale alone does not solve constraint-aware planning.
  • Chain-of-Thought and Plan-and-Solve prompting help on EASY planning, but their benefit shrinks or reverses as constraints grow, so prompt engineering should not be assumed to transfer to constrained multimodal planning.
  • Deploying MLLMs as itinerary or scheduling assistants would require a separate constraint-checking or verification step, because raw model outputs frequently violate budget, time, or distance limits.
  • Text-only in-context learning hurts performance, suggesting that effective few-shot adaptation for these tasks would need multimodal demonstrations.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cleaner experimental design would hold search-space size constant while varying only the number and type of constraints, since the current EASY-to-HARD comparison changes multiple parameters at once and does not fully isolate constraint complexity.
  • The stable bias of smaller models toward particular infeasible schedules suggests a learned prior about calendar conventions; fine-tuning on explicit constraint-violation examples would be a direct test of whether that bias is correctable.
  • If these results generalize, a hybrid planner that uses an MLLM for perception and a symbolic solver for constraint satisfaction would likely beat the pure-model approach, a route the paper does not explore.
  • The benchmark's feasible-plan metric treats a plan as acceptable if it satisfies constraints and optimal if it minimizes budget; a natural extension would add user-preference weighting or multi-objective optimization, which real planning also requires.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The manuscript introduces MPCC, a benchmark of 2,700 multimodal planning tasks spanning flight, calendar, and meeting planning, with budget, temporal, and spatial constraints and three graded difficulty levels. The authors evaluate 13 MLLMs and report that closed-source models attain only 21.3% feasible plans on average and open-source models below 11%, with performance dropping sharply from EASY to HARD. They also analyze prompting strategies (CoT, Plan-and-Solve, ICL) and a reasoning model. The central contributions claimed are the benchmark itself, the 'multimodal constraint' formalization, and the observation that MLLMs are sensitive to constraint complexity.

Significance. The benchmark addresses a real gap: existing multimodal planning benchmarks largely ignore explicit cross-modal constraints. If the measurements are reliable, the finding that state-of-the-art MLLMs produce mostly infeasible plans in constrained settings is an important result for the field. The paper also ships a reproducible evaluation pipeline via VLMEvalKit and provides a human-checked dataset with inter-annotator agreement (Kappa = 0.83), which are strengths. However, the key interpretive claim about constraint complexity is not yet supported by the experimental design.

major comments (3)
  1. [Abstract and §3.2] The abstract and §3.2 claim that the EASY/MEDIUM/HARD levels 'separate constraint complexity from search space expansion.' However, §3.2 states that complexity is controlled by adjusting the number of transit cities, planning days, venue availability, and schedule resolution, which are precisely the parameters that determine the size of the plan search space (Table 2 reports average search spaces of 27, 184, and 617). Within each task, the constraint types (T_seq ⊗ B, T_con ⊗ B, T_con ⊗ S ⊗ B) are fixed across levels; only instance-level bounds vary. Consequently, the systematic performance drops from EASY to HARD in Table 3 are attributable to search-space expansion, not to constraint complexity. The central claim that MLLMs are 'highly sensitive to constraint complexity' requires a matched condition that holds search-space size fixed while varying the number or strictness of constraints, or the paper must be reframed to avoid the separation claim.
  2. [Table 3/4 and §5.2] The 'Empirical Max' baseline is defined as 'the maximum accuracy attainable by exhaustively evaluating all options within the search space, where each option is uniformly applied to all tasks for a given difficulty level.' As written, an exhaustive evaluation would achieve 100%, but the reported values (7.0–12.0) appear to be the expected feasibility/optimality probability of a uniformly random plan. If the intended baseline is uniform random selection, it should be renamed (e.g., 'Random Baseline') and described as such; the current terminology is contradictory and makes the claim that models 'fail to outperform random selection' ambiguous.
  3. [§5.1 and Table 3] All results are obtained through exact matching (VLMEvalKit). Since the paper itself identifies 'Output Format Error' as a failure category (Figure 10), exact string matching likely undercounts feasible plans that are semantically correct but formatted differently. The feasible-plan rate should be computed by parsing the model output and verifying the plan against the constraints, or by reporting a format-insensitive metric as a sensitivity check. Without this, the headline 21.3% feasible rate may partially reflect evaluation strictness rather than planning failure.
minor comments (4)
  1. [§2 and Eqs. 5–7] The formal constraint definitions (Eqs. 1–4) are closer to descriptions than formalizations; e.g., the composite operator ⊗ is used in Eqs. 5–7 without a definition. Since the paper claims to 'formalize multimodal constraints,' this should be tightened.
  2. [§5.4.2 and Figure 6] Figure 6 uses Simpson's concentration and diversity indices, but the paper does not define how model responses are tokenized or grouped for these indices; a short methodological description would improve reproducibility.
  3. [§5.4.3 and Figure 7] The ICL experiment is limited to text-only demonstrations on Calendar Planning EASY; the conclusion that 'ICL significantly interferes with multimodal planning processes' should be scoped to this setting, since the abstract states a stronger, more general claim.
  4. [Table 3] The Empirical Max row does not include Flight Planning values, while it includes values for Calendar and Meeting Planning; clarify why this baseline is omitted for Flight Planning.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MPCC's benchmark claims are not derived from their evaluation targets; self-citations are contextual, not load-bearing.

full rationale

MPCC is an empirical benchmark paper, not a paper that derives predictions from fitted parameters. The constraint definitions (Eqs. 1-4) are explicit satisfaction conditions; ground-truth feasible and optimal plans are obtained by brute-force enumeration of each task instance (Section 3.2), independent of any MLLM output. No model performance number feeds back into the benchmark construction or into the difficulty labels. The difficulty levels are constructed by adjusting task parameters (transit cities, planning days, venue availability, schedule resolution), and Table 2 reports that average search space grows from 27 to 617; this means that EASY/MEDIUM/HARD do not isolate constraint complexity from search-space size, and the headline 'highly sensitive to constraint complexity' is therefore under-supported as a controlled claim. That is a validity/confound concern, not a circularity: the conclusion is not equivalent to the inputs by construction. Self-citations are present (e.g., [5], [8], [9], [25], [34] share authors), but they appear in related-work context or as methodological inspiration for text-only ICL experiments (Section 5.4.3, 'following Qin et al. [25]'); none carries the central benchmark-construction argument. The 'Empirical Max' baseline in Tables 3-4 is ambiguously defined: an exhaustive per-instance maximum should approach 100% because every instance has a feasible solution, whereas the reported 7-12% values suggest a fixed-option/uniform-policy baseline; this affects the 'fail to outperform random selection' claims but is an evaluation-baseline clarity issue, not circularity. No fitted parameter is renamed as a prediction, and no uniqueness theorem or preregistered result is imported from the authors' prior work. Hence the central claims are self-contained empirical measurements; the paper receives 0 on the circularity scale.

Assumptions & free parameters 2 free parameters · 5 assumptions · 2 invented entities

The central results do not rest on fitted parameters in the usual sense, but they do rest on hand-chosen design choices: the difficulty calibration, the constraint bounds, and the evaluation protocol. The main risk is that difficulty and search-space size are varied together, so the benchmark's key interpretation is an assumption rather than a controlled finding. The ⊗ operator and the multimodal-constraint label add terminology without adding measurable empirical content.

free parameters (2)
  • Per-difficulty search-space calibration = 27 / 184 / 617 average feasible configurations (EASY/MEDIUM/HARD)
    Hand-chosen calibration that defines the difficulty gradient. Because these values co-vary with the number and type of constraints, they confound the paper's constraint-complexity conclusion.
  • Instance-level constraint bounds = Randomly assigned per instance (e.g., B_max, t_min, D_i)
    Chosen so each task has at least one feasible solution and to shape feasibility density. They are not fitted to model outputs, but they determine task difficulty and are not published as parameter files.
assumptions (5)
  • domain assumption Synthetic Google Flights, Calendar, and Maps-style images are a valid proxy for real-world multimodal planning inputs.
    The benchmark's realism claim rests on interface mimicry, not real user data. This assumption enters in Section 3.3.
  • domain assumption Exact-string matching is a valid correctness measure for planning answers.
    Section 5.1 says results were obtained through exact matching; this penalizes semantically correct but differently formatted plans and may overstate model failure.
  • standard math Brute-force enumeration over the generated state space finds all feasible and optimal plans.
    Used to label ground truth in Section 3.2. It requires the synthetic generator to truthfully model all constraints and the state space to be fully enumerable.
  • domain assumption The human recheck procedure guarantees that every instance requires both modalities and has a defined solution.
    Section 3.4 reports Kappa=0.83 and less than 10% discarded instances, but this is a subjective quality judgment applied before evaluation.
  • ad hoc to paper Increasing search-space size and constraint counts together yields comparable difficulty across task types.
    Section 3.2 asserts comparable search-space sizes and task-specific semantics without a formal equivalence proof. This assumption underwrites the difficulty-level analysis.
invented entities (2)
  • Composite constraint operator ⊗ (e.g., C = T_seq ⊗ B)
    purpose: Notational device to represent joint enforcement of multiple constraint types in Equations 5 through 7.
    The ⊗ symbol is never given algebraic semantics; it reduces to logical conjunction of feasibility inequalities, so it is a label rather than a new formal object.
  • Multimodal constraint as a formal category
    purpose: The paper claims to formalize a new concept for planning benchmarks.
    The three categories are standard budget, temporal, and spatial constraints encoded as inequalities. The term is new, but the underlying content has no independent falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/DHSE2TE5

@misc{pith2026250723382,
  author       = {Pith},
  title        = {Pith review of: MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHSE2TE5}},
  note         = {Machine review of arXiv:2507.23382}
}
read the original abstract

Multimodal planning capabilities refer to the ability to predict, reason, and design steps for task execution with multimodal context, which is essential for complex reasoning and decision-making across multiple steps. However, current benchmarks face two key challenges: (1) they cannot directly assess multimodal real-world planning capabilities, and (2) they lack constraints or implicit constraints across modalities. To address these issues, we introduce Multimodal Planning with Complex Constraints (MPCC), the first benchmark to systematically evaluate MLLMs' ability to handle multimodal constraints in planning. To address the first challenge, MPCC focuses on three real-world tasks: Flight Planning, Calendar Planning, and Meeting Planning. To solve the second challenge, we introduce complex constraints (e.g. budget, temporal, and spatial) in these tasks, with graded difficulty levels (EASY, MEDIUM, HARD) to separate constraint complexity from search space expansion. Experiments on 13 advanced MLLMs reveal significant challenges: closed-source models achieve only 21.3% feasible plans, while open-source models average below 11%. Additionally, we observe that MLLMs are highly sensitive to constraint complexity and that traditional multimodal prompting strategies fail in multi-constraint scenarios. Our work formalizes multimodal constraints in planning, provides a rigorous evaluation framework, and highlights the need for advancements in constraint-aware reasoning for real-world MLLM applications.

Figures

Figures reproduced from arXiv: 2507.23382 by the authors.

Figure 1
Figure 1. The example of multimodal planning tasks without [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of the categories of multimodal constraints featured in MPCC. These constraints are instantiated through [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Performance of different models in feasible plan. shown in [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figures from the paper (5 more)
Figure 3
Figure 3. Figure 3: Distribution of the number of feasible plans across different tasks. EASY MEDIUM HARD EASY MEDIUM HARD EASY MEDIUM HARD GPT-4o Gemini-2.0-Flash Claude-3.5V-Sonnet InternVL-78B Qwen2-VL-72B Multimodal Large Language Models [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 6
Figure 6. Figure 6: Simpson’s Concentration and Diversity Index of [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: In-Context-Learning analysis on feasible plan per￾formance with text-only demonstrations. complex problems. Enhancing their ability to plan under intricate multimodal constraints is therefore critical. 5.4.3 The text-only in-context learning approach does not work effe…
Figure 9
Figure 9. Figure 9: Performance of MLLMs on Flight Planning tasks [PITH_FULL_IMAGE:figures/full_fig_p008_9.png]
Figure 10
Figure 10. Figure 10: Manual analysis of incorrect responses. Over 40% [PITH_FULL_IMAGE:figures/full_fig_p008_10.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

45 extracted references · 25 canonical work pages

  1. [1]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. 2023

    work page 2023

  2. [2]

    Towards reasoning era: A survey of long chain-of-thought for reasoning large language models

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wangxiang Che. Towards reasoning era: A survey of long chain-of-thought for reasoning large language models. arXiv preprint arXiv:2503.09567, 2025

    arXiv 2025

  3. [3]

    Ecm: A unified electronic circuit model for explaining the emergence of in-context learning and chain-of-thought in large language model.arXiv preprint arXiv:2502.03325, 2025

    Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiaqi Wang, Mengkang Hu, Zhi Chen, Wanxiang Che, and Ting Liu. Ecm: A unified electronic circuit model for explaining the emergence of in-context learning and chain-of-thought in large language model.arXiv preprint arXiv:2502.03325, 2025

    arXiv 2025

  4. [4]

    Un- locking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought

    Qiguang Chen, Libo Qin, Jiaqi Wang, Jingxuan Zhou, and Wanxiang Che. Un- locking the capabilities of thought: A reasoning boundary framework to quantify and optimize chain-of-thought. InProc. of NeurIPS, 2024

    work page 2024

  5. [5]

    M 3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of- thought

    Qiguang Chen, Libo Qin, Jin Zhang, Zhi Chen, Xiao Xu, and Wanxiang Che. M 3 cot: A novel benchmark for multi-domain multi-step multi-modal chain-of- thought. InProc. of ACL, 2024. MPCC: A Novel Benchmark for Multimodal Planning with Complex Constraints in Multimodal Large Language Models

    work page 2024

  6. [6]

    Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models.CoRR, 2023

    Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models.CoRR, 2023

    work page 2023

  7. [7]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProc. of CVPR, pages 24185–24198, 2024

    work page 2024

  8. [8]

    Visual thoughts: A unified perspective of understanding multimodal chain-of-thought.arXiv preprint arXiv:2505.15510, 2025

    Zihui Cheng, Qiguang Chen, Xiao Xu, Jiaqi Wang, Weiyun Wang, Hao Fei, Yidong Wang, Alex Jinpeng Wang, Zhi Chen, Wanxiang Che, et al. Visual thoughts: A unified perspective of understanding multimodal chain-of-thought.arXiv preprint arXiv:2505.15510, 2025

    arXiv 2025

Show all 45 references
  1. [9]

    Comt: A novel benchmark for chain of multi-modal thought on large vision-language models

    Zihui Cheng, Qiguang Chen, Jin Zhang, Hao Fei, Xiaocheng Feng, Wanxiang Che, Min Li, and Libo Qin. Comt: A novel benchmark for chain of multi-modal thought on large vision-language models. InProc. of AAAI, 2025

    2025

  2. [10]

    A survey on in-context learning

    Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Jingyuan Ma, Rui Li, Heming Xia, Jingjing Xu, Zhiyong Wu, Tianyu Liu, et al. A survey on in-context learning. arXiv preprint arXiv:2301.00234, 2022

    2022 arXiv

  3. [11]

    Vlmevalkit: An open- source toolkit for evaluating large multi-modality models

    Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open- source toolkit for evaluating large multi-modality models. InProceedings of the 32nd ACM International Conference on Multimedia...

    2024

  4. [12]

    Deepseek-r1: Incen- tivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incen- tivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948, 2025

    2025 arXiv

  5. [13]

    Mllm-compbench: A compara- tive reasoning benchmark for multimodal llms

    Jihyung Kil, Zheda Mai, Justin Lee, Arpita Chowdhury, Zihe Wang, Kerrie Cheng, Lemeng Wang, Ye Liu, and Wei-Lun Harry Chao. Mllm-compbench: A compara- tive reasoning benchmark for multimodal llms. InProc. of NeurIPS, 2024

    2024

  6. [14]

    Visualwebarena: Evaluating multimodal agents on realistic visual web tasks

    Jing Yu Koh, Robert Lo, Lawrence Jang, Vikram Duvvur, Ming Chong Lim, Po-Yu Huang, Graham Neubig, Shuyan Zhou, Ruslan Salakhutdinov, and Daniel Fried. Visualwebarena: Evaluating multimodal agents on realistic visual web tasks. arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  7. [15]

    Tree search for language model agents, 2024

    Jing Yu Koh, Stephen McAleer, Daniel Fried, and Ruslan Salakhutdinov. Tree search for language model agents, 2024

    2024

  8. [16]

    Large language models are zero-shot reasoners, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners, 2022

    2022

  9. [17]

    Llava-onevision: Easy visual task transfer, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024

    2024

  10. [18]

    Seed-bench: Benchmarking multimodal large language models

    Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In Proc. of CVPR, 2024

    2024

  11. [19]

    Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation

    Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al. Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation. InConference on R...

    2023

  12. [20]

    Ferret-ui 2: Mastering universal user interface understanding across platforms, 2024

    Zhangheng Li, Keen You, Haotian Zhang, Di Feng, Harsh Agrawal, Xiujun Li, Mohana Prasad Sathya Moorthy, Jeff Nichols, Yinfei Yang, and Zhe Gan. Ferret-ui 2: Mastering universal user interface understanding across platforms, 2024

    2024

  13. [21]

    Robomp 2: A robotic multimodal perception-planning framework with multimodal large language models.arXiv preprint arXiv:2404.04929, 2024

    Qi Lv, Hao Li, Xiang Deng, Rui Shao, Michael Yu Wang, and Liqiang Nie. Robomp 2: A robotic multimodal perception-planning framework with multimodal large language models.arXiv preprint arXiv:2404.04929, 2024

    2024 arXiv

  14. [22]

    Janusflow: Harmo- nizing autoregression and rectified flow for unified multimodal understanding and generation.arXiv preprint arXiv:2411.07975, 2024

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmo- nizing autoregression and rectified flow for unified multimodal understanding and generation.arXiv preprint arXiv:2411.07975, 2024

    2024 arXiv

  15. [23]

    m & m’s: A benchmark to evaluate tool-use for m ulti-step m ulti-modal tasks

    Zixian Ma, Weikai Huang, Jieyu Zhang, Tanmay Gupta, and Ranjay Krishna. m & m’s: A benchmark to evaluate tool-use for m ulti-step m ulti-modal tasks. In Proc. of ECCV, pages 18–34. Springer, 2024

    2024

  16. [24]

    Perception test: A diagnostic benchmark for multimodal video models

    Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Mar- keeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. InProc. of NeurIPS, volume 36, pages 42748–4...

    2023

  17. [25]

    What factors affect multi-modal in-context learning? an in-depth exploration.arXiv preprint arXiv:2410.20482, 2024

    Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi-modal in-context learning? an in-depth exploration.arXiv preprint arXiv:2410.20482, 2024

    2024 arXiv

  18. [26]

    Mementos: System support for long-running computation on rfid-scale devices

    Benjamin Ransford, Jacob Sorber, and Kevin Fu. Mementos: System support for long-running computation on rfid-scale devices. InProceedings of the sixteenth international conference on Architectural support for programming languages and operating systems, pages 159–170, 2011

    2011

  19. [27]

    Alfred: A benchmark for interpreting grounded instructions for everyday tasks

    Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: A benchmark for interpreting grounded instructions for everyday tasks. InProc. of CVPR, 2020

    2020

  20. [28]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  21. [29]

    Qvq: To see the world with wisdom, 2024

    Qwen Team. Qvq: To see the world with wisdom, 2024

    2024

  22. [30]

    Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change, 2023

    Karthik Valmeekam, Matthew Marquez, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change, 2023

    2023

  23. [31]

    Xlrs-bench: Could your multimodal llms understand extremely large ultra-high-resolution remote sensing imagery?, 2025

    Fengxiang Wang, Hongzhen Wang, Mingshuo Chen, Di Wang, Yulin Wang, Zonghao Guo, Qiang Ma, Long Lan, Wenjing Yang, Jing Zhang, Zhiyuan Liu, and Maosong Sun. Xlrs-bench: Could your multimodal llms understand extremely large ultra-high-resolution remote sensing imagery?, 2025

    2025

  24. [32]

    Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.arXiv preprint arXiv:2305.04091, 2023

    Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models.arXiv preprint arXiv:2305.04091, 2023

    2023 arXiv

  25. [33]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution.arXiv preprint arXiv:2409.12191, 2024

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model’s pe...

    2024 arXiv

  26. [34]

    S3 agent: Unlocking the power of vllm for zero-shot multi-modal sarcasm detection.ACM Transactions on Multimedia Computing, Communications and Applications, 2024

    Peng Wang, Yongheng Zhang, Hao Fei, Qiguang Chen, Yukai Wang, Jiasheng Si, Wenpeng Lu, Min Li, and Libo Qin. S3 agent: Unlocking the power of vllm for zero-shot multi-modal sarcasm detection.ACM Transactions on Multimedia Computing, Communications and Applications, 2024

    2024

  27. [35]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. InProc. of NeurIPS, volume 35, pages 24824–24837, 2022

    2022

  28. [36]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    arXiv

  29. [37]

    Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal under- standing.arXiv preprint arXiv:2412.10302, 2024

    Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, et al. Deepseek-vl2: Mixture-of-experts vision-language models for advanced multimodal under- standing.arXiv preprint arXiv:2412.10302, 2024

    2024 arXiv

  30. [38]

    Logicvista: Multimodal llm logical reasoning benchmark in visual contexts.arXiv preprint arXiv:2407.04973, 2024

    Yijia Xiao, Edward Sun, Tianyu Liu, and Wei Wang. Logicvista: Multimodal llm logical reasoning benchmark in visual contexts.arXiv preprint arXiv:2407.04973, 2024

    2024 arXiv

  31. [39]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer envi- ronments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Jing Hua Toh, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer envi- ronments. InProc. of NeurIPS, volume 37, pages 5...

    2024

  32. [40]

    Mm-react: Prompt- ing chatgpt for multimodal reasoning and action, 2023

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. Mm-react: Prompt- ing chatgpt for multimodal reasoning and action, 2023

    2023

  33. [41]

    Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi.arXiv preprint arXiv:2404.16006, 2024

    Kaining Ying, Fanqing Meng, Jin Wang, Zhiqian Li, Han Lin, Yue Yang, Hao Zhang, Wenbo Zhang, Yuqi Lin, Shuo Liu, et al. Mmt-bench: A comprehensive multimodal benchmark for evaluating large vision-language models towards multitask agi.arXiv preprint arXiv:2404.16006, 2024

    2024 arXiv

  34. [42]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi

    Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. InProc. of CVPR, 2024

    2024

  35. [43]

    Open3dvqa: A benchmark for comprehensive spatial reasoning with multimodal large language model in open space, 2025

    Weichen Zhan, Zile Zhou, Zhiheng Zheng, Chen Gao, Jinqiang Cui, Yong Li, Xinlei Chen, and Xiao-Ping Zhang. Open3dvqa: A benchmark for comprehensive spatial reasoning with multimodal large language model in open space, 2025

    2025

  36. [44]

    Le, Ed H

    Huaixiu Steven Zheng, Swaroop Mishra, Hugh Zhang, Xinyun Chen, Minmin Chen, Azade Nova, Le Hou, Heng-Tze Cheng, Quoc V. Le, Ed H. Chi, and Denny Zhou. Natural plan: Benchmarking llms on natural language planning, 2024

    2024

  37. [45]

    Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Srid- har, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents.arXiv preprint arXiv:2307.13854, 2023

    2023 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.