Pith. sign in

REVIEW 4 major objections 6 minor 28 references

The paper introduces a lightweight benchmark with 400+ procedurally generated 2D-physics problems and reports that VLM physics reasoning scales with model size while spatial reasoning remains a weakness.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-04 20:53 UTC pith:IBECZ25D

load-bearing objection The benchmark generator is a genuinely useful idea, but the paper's central scaling claim is contradicted by its own Table 1, and the undefined composite metric and missing code make the results unverifiable. the 4 major comments →

arxiv 2509.08270 v1 pith:IBECZ25D submitted 2025-09-10 cs.LG cs.AI

Interpretable Physics Reasoning and Performance Taxonomy in Vision-Language Models

Pranav Pawar , Kavish Shah , Akshat Bhalani , Komal Kasat , Dev Mittal , Hadi Gala , Deepali Patil , Nikita Raichada
show 1 more author
Monali Deshmukh
This is my paper
classification cs.LG cs.AI
keywords vision-language modelsphysics reasoningbenchmarkingprocedural scenario generationmultimodal evaluationspatial reasoningchain-of-thought promptingscaling laws
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper proposes a lightweight, procedurally generated benchmark for evaluating vision-language models (VLMs) on 2D physics, with more than 400 problems across projectile motion, collision dynamics, mechanics, and fluid dynamics. It uses this testbed to compare four open-weight VLMs and reports that physics reasoning improves with model scale, with Qwen2.5-VL-7B scoring highest on an overall composite (0.815). The paper's more interesting claim is behavioral: models handle formula-driven problems well but struggle with abstract spatial reasoning, and most errors are conceptual rather than perceptual or arithmetic. A sympathetic reader would care because this gives a cheap, replicable way to separate pattern-matching from physical understanding in multimodal systems.

Core claim

On its own terms, the paper establishes a new evaluation method and a first taxonomy of VLM physics reasoning. The key findings are: (1) a composite score combining accuracy, reasoning quality, and adaptability ranks Qwen2.5-VL-7B first at 0.815, above LLaMA-3.2-Vision-11B (0.765), Gemma2-27B-Vision (0.75), and DeepSeek-VL-1.3B (0.70); (2) rank order roughly follows model scale, but Gemma2-27B has higher raw physics accuracy (0.88) than the top-ranked model (0.85), so the composite ranking is not simply accuracy; (3) across domains, formula-driven fluid dynamics and projectile problems are handled best, while spatial and mechanics reasoning is weakest; and (4) 52–67% of failures are conceptu

What carries the argument

The benchmark pipeline: a procedural scenario generator samples physical parameters from specified distributions (e.g., projectile velocities 10–50 m/s, angles 15–75°) to create Easy, Medium, and Hard items; a computational ground-truth engine computes reference answers from analytic formulas and numerical methods; and an evaluation protocol scores four dimensions—physics accuracy, reasoning quality, computational efficiency, and domain adaptability—using chain-of-thought and few-shot prompts. The novelty the paper claims is the lightweight reproducibility of this generation-evaluation loop, without expensive physics simulators.

Load-bearing premise

The central ranking rests on an undisclosed weighting of accuracy, reasoning quality, and adaptability; changing those weights could reorder the models, so the scale-correlation conclusion is only as strong as that unpublished formula.

What would settle it

Recompute the Overall Score from the scores in Tables 1 and 2 once the composite formula and answer-tolerance margin are known: if no nonnegative weighting makes Qwen2.5-VL-7B beat Gemma2-27B-Vision, or if equal weighting puts Gemma2-27B first, the claimed ranking and the scale-correlation conclusion fail. Alternatively, evaluate a much larger VLM on the same 400+ problems; if it does not outscore Qwen2.5-VL-7B, scale is not the controlling factor.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • If scale correlates with physics reasoning, continued scaling of VLMs remains a viable route to better performance on formulaic physics problems.
  • If the error taxonomy holds, future work should target conceptual understanding and spatial reasoning rather than arithmetic or perception; the paper's own proposed directions (3D environments, thermodynamics, electromagnetism) follow naturally.
  • The efficiency results imply that practical deployments can use 8-bit quantization with under 3% degradation, and smaller models give better performance per unit of inference time for resource-constrained use.
  • The benchmark itself provides a reusable, simulator-free testbed for comparing future VLMs on physical reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper does not run: recompute the composite from the published per-component numbers under different plausible weights. Because Gemma2-27B has higher accuracy and reasoning quality than Qwen2.5-VL-7B, the top ranking can only survive if the undisclosed adaptability term is weighted heavily; until the formula is released, the scale-correlation claim should be read cautious
  • The formulaic-versus-spatial distinction could be sharpened by generating paired problems that differ only in the diagram's geometry while keeping equations identical; such controlled pairs would isolate visual-spatial failure from equation-application failure.
  • The procedural generator could be extended to cross-domain transfer tests, such as solving a projectile problem after evaluating on fluid dynamics, to test whether conceptual physics knowledge transfers or stays compartmentalized by domain.
  • Releasing per-item scores and rubrics would let the community audit the qualitative taxonomy and reuse the items for diagnostic probes or instruction-tuning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces a benchmark framework for evaluating vision-language models (VLMs) on 2D physics reasoning. Problems are procedurally generated across four domains (Projectile Motion, Collision Dynamics, Mechanics, Fluid Dynamics), and four open VLMs of different sizes are evaluated. The authors report a composite Overall Score, domain-specific scores, reasoning-quality rubrics, an error taxonomy, and efficiency measurements. The central claims are that model scale correlates strongly with physics reasoning ability and that the 7B Qwen2.5-VL model is the best performer, while models excel at formulaic problems but struggle with abstract spatial reasoning.

Significance. If the evaluation were sound, the framework would be a lightweight, reproducible diagnostic tool for scientific reasoning in VLMs, complementing heavier simulators and existing multimodal benchmarks. The procedural generator, multi-domain coverage, and explicit attempt to separate physics accuracy from reasoning quality are constructive contributions. The error taxonomy and efficiency analysis also address practically relevant dimensions. However, the headline scaling result is directly contradicted by the paper's own reported scores, and the composite metric is not defined, so the central empirical contribution is not currently credible. The framework may be salvageable with corrected claims and full metric transparency, but as presented the evaluation does not support its conclusions.

major comments (4)
  1. [§4.1 / Table 1] The claim of a 'strong positive correlation between model parameter count and overall performance' is contradicted by the table it cites. The Overall Scores are: Qwen2.5-VL-7B 0.815, LLaMA-3.2-Vision-11B 0.765, Gemma2-27B-Vision 0.75, DeepSeek-VL-1.3B 0.70. The rank order is 7B > 11B > 27B > 1.3B, which is not monotonic; the 7B model outperforms the 27B model. With only four points, this ordering alone refutes a monotonic scaling claim, and the Spearman correlation is negative (rho ≈ -0.2). Section 4.2 further notes that Gemma2-27B is the most accurate performer in projectile motion, collision dynamics, and mechanics, yet ranks third overall. The abstract and conclusion repeat the scaling claim, so this is a load-bearing inconsistency, not a local wording issue.
  2. [§3.3 / Table 1] The 'Overall Score' composite is never defined. Section 3.3 lists four criteria (Physics Accuracy, Reasoning Quality, Computational Efficiency, Domain Adaptability) but no formula, weighting, or aggregation procedure is given. Table 1 reports Overall Score, Physics Accuracy, and Reasoning Quality only, and the relationship between these columns is unexplained. For example, Gemma2-27B has higher Physics Accuracy (0.88) and higher Reasoning Quality (0.85) than Qwen2.5-VL-7B (0.85 and 0.78), yet Qwen ranks above it. Without the composite definition, the ranking is unauditable; different reasonable weightings could easily reorder the models. The 'appropriate margin for floating-point precision' used in accuracy scoring is also unspecified, further compromising reproducibility.
  3. [§3.3 / §4.4] The rubric-based scoring of reasoning quality and the error taxonomy are described only in broad terms. Section 3.3 states that explanation text is evaluated for 'logical adherence, correct physics terminology usage, and solution completeness,' but the actual rubric, scoring scale, and annotation procedure are not provided. Section 4.4 reports percentages of Conceptual, mathematical, and Perception errors, but the coding protocol and inter-annotator reliability are absent. Since the paper emphasizes 'interpretable' reasoning assessment, these missing details are essential to the central evaluation methodology.
  4. [§4.6 / Table 1] The statistical significance claim is overstated. Section 4.6 says bootstrap-resampled confidence intervals show that 'performance differences between all model pairs are statistically significant for overall scores.' Yet the 95% CIs in Table 1 overlap substantially for LLaMA-3.2-Vision-11B ([0.750, 0.780]) and Gemma2-27B-Vision ([0.735, 0.765]). With n=4 models and no details on the bootstrap procedure or the number of evaluation instances per model, the assertion of universal pairwise significance is not supported by the reported data.
minor comments (6)
  1. [§4.2] The narrative descriptions are inconsistent with the numerical averages. Fluid Dynamics and Mechanics both have an average of 0.79, yet the text labels them 'Fluid Dynamics Excellence' and 'Mechanics Challenges,' respectively. The domain-level differences are small and the qualitative interpretation should be tempered.
  2. [Table 3] The 'Perf./Efficiency' column is not defined. If it is Overall Score divided by inference time, the reported values do not match direct division (e.g., 0.70/2.3 ≈ 0.304 vs. 0.332 for DeepSeek-VL-1.3B; 0.75/11.7 ≈ 0.064 vs. 0.076 for Gemma2-27B). Clarify the formula.
  3. [§4.4] The sentence 'Computational errors were more common in smaller models (23% vs. 12%)' does not specify which models are being compared. It also appears to conflate 'computational' with 'mathematical' error categories; please clarify.
  4. [References] Several references appear misattributed or incomplete. For example, reference [23] lists 'Maarten Wang, Disha Gupta, and Percy Liang' for Chain-of-Thought prompting, which is not the well-known Wei et al. paper, and reference [22] appears to duplicate PIQA under different authorship. The reference list needs careful verification.
  5. [Throughout] There are numerous typos and grammatical errors: 'uptil', 'chose models', 'expland', 'later' for 'latter', 'futher', 'ofscientific', 'light & replicable design', and inconsistent spacing. These do not affect the technical content but detract from presentation.
  6. [Figure 6 / Figure 7] The captions mention metrics such as 'success rate' and 'response quality distribution' that are not defined in Section 3.3. Please define all plotted quantities or remove them from the figures.

Circularity Check

0 steps flagged

No circularity: the paper is an empirical benchmark evaluation; no prediction reduces by construction to its inputs.

full rationale

The paper's central activity is measurement, not derivation. Section 3.2 describes procedural generation of physics problems from parameter distributions and a ground-truth computation engine; Section 3.3 defines the evaluation metrics (physics accuracy, reasoning quality, computational efficiency, domain adaptability). The abstract's claim of a 'strong correlation between model scale and reasoning ability' is an empirical interpretation of Table 1, not a quantity fitted to that table or defined in terms of model scale: parameter count and the reported scores are measured separately. The single most suspicious item, the 'overall score' composite, is not defined (weights/rubric omitted), but omitting the formula makes the ranking unauditable rather than circular; no equation in the paper reduces the score to parameter count. The references cited for problem generation [13], numerical integration [20], and CoT prompting [23] are external; none of the authors cite their own prior work as the basis of a load-bearing theorem. The rubric-based reasoning-quality metric is authored for this study, but that is a methodological validity concern, not a circular step. The paper's own data also undermine the scaling conclusion (Gemma2-27B has higher accuracy and reasoning quality than Qwen2.5-VL-7B yet ranks third), but contradiction is the opposite of circularity: the claim is not forced by the inputs. Therefore no circularity is present; the score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

The paper depends on unstated evaluation choices (answer margin, difficulty thresholds, composite weights) and domain assumptions about what the generated testbed measures. Since no code or data are released, these remain unverified.

free parameters (3)
  • Answer tolerance margin
    Physics accuracy is scored 'with appropriate margin for floating-point precision' (Section 3.3), but the margin value is not given; different margins can change pass/fail labels.
  • Difficulty thresholds (Easy/Medium/Hard)
    Problems are categorized into three difficulty levels (Section 3.2) without specifying the threshold rules, which affect per-difficulty results and error analyses.
  • Overall score weighting
    Table 1 reports a composite 'overall score' combining accuracy, reasoning quality, and adaptability, but the weights are not defined; the headline ranking depends on arbitrary weighting choices.
axioms (3)
  • domain assumption The ground-truth engine's analytical and numerical solutions are correct for all generated problems.
    The entire accuracy metric rests on these unshown computations (Section 3.2); no validation or test cases are provided.
  • domain assumption The rubric-based criteria (detection of principles, correct equations, valid progression, interpretation) yield objective, reproducible reasoning-quality scores.
    Section 3.3 describes the rubric at a high level but provides no scoring sheet, annotator instructions, or inter-annotator agreement.
  • domain assumption Performance on this procedurally generated 2D physics testbed measures general physics reasoning in VLMs.
    The paper assumes transfer from the synthetic benchmark to broader physics competence without a comparison to established benchmarks such as ScienceQA, MathVista, or PIQA.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Interpretable Physics Reasoning and Performance Taxonomy in Vision-Language Models." pith.science (2026). https://pith.science/paper/IBECZ25D

@misc{pith2026250908270,
  author       = {Pith},
  title        = {Pith review of: Interpretable Physics Reasoning and Performance Taxonomy in Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IBECZ25D}},
  note         = {Machine review of arXiv:2509.08270}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

As Vision-Language Models (VLMs) grow in sophistication, their ability to perform reasoning is coming under increasing supervision. While they excel at many tasks, their grasp of fundamental scientific principles, such as physics, remains an underexplored frontier. To reflect the advancements in these capabilities, we introduce a novel and accessible framework designed to rigorously evaluate VLMs on their understanding of 2D physics. Our framework features a pragmatic scenario generator that creates a diverse testbed of over 400 problems across four core domains: Projectile Motion, Collision Dynamics, Mechanics, and Fluid Dynamics. Through comprehensive evaluation of four state-of-the-art VLMs, we demonstrate a strong correlation between model scale and reasoning ability, with our top-performing model, Qwen2.5-VL-7B, achieving an overall score of 0.815. We find that while models excel at formulaic problems, they struggle significantly with domains requiring abstract spatial reasoning. By designing this framework, we aim to democratize the study of scientific reasoning in VLMs and foster deeper insights into their capabilities and limitations.

Figures

Figures reproduced from arXiv: 2509.08270 by Akshat Bhalani, Deepali Patil, Dev Mittal, Hadi Gala, Kavish Shah, Komal Kasat, Monali Deshmukh, Nikita Raichada, Pranav Pawar.

Figure 1
Figure 1. Figure 1: Qualitative Analysis: Fluid Dynamics. 1 Introduction In recent years, VLMs have captured the imagination of the Artificial Intelligence(AI) community, demonstrating an impressive ability to interpret, reason about, and generate content that covers both text and image handling. From answering questions about visual scenes to engaging in multi-modal dialogue, models such as Flamingo [1], PaLI [25], and BLIP-… view at source ↗
Figure 2
Figure 2. Figure 2: Qualitative Analysis: Mechanics. 2 [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Qualitative Analysis: Collision Dynamics. [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative Analysis: Projectile Motion. [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Framework Architecture This flowchart illustrates the end-to-end pipeline, starting from the foundational setup (Phase 1), through environment and scenario generation (Phase 2), to the multi-faceted evaluation framework (Phase 3), rigor￾ous experimental design (Phase 4), and culminating in comprehensive results analysis and visualization (Phase 5). 3.2 Physics Environment Design and Problem Generation The … view at source ↗
Figure 6
Figure 6. Figure 6: Model Performance Analysis. A six-panel evaluation of physics-task performance, reporting physics accuracy, reasoning quality, and overall metrics (overall score and success rate) for each model. Bottom panels compare performance across environments (projectile motion, collision dynamics, mechanics, fluid dynamics), provide an overall ranking, and show cross-environment consistency. Bars are normalized on … view at source ↗
Figure 7
Figure 7. Figure 7: Model Response & Correctness Analysis. A six-panel overview comparing four models, showing confidence scores, correctness rate versus response length, and error distribution by type across models. Lower panels summarize perfor￾mance by problem difficulty, response quality distribution, and an inter-model agreement matrix indicating pairwise agreement strength. Panels are read left-to-right, top-to-bottom; … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 24 canonical work pages · 2 internal anchors

  1. [1]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Yazhe Li, Aida Nematzadeh, Douwe Kiela, Hector Base, Jessica Yu, Aitor Torralba, and Iain Reid. Flamingo: a visual language model for few-shot learning. In Advances in Neural Information Processing Systems, 2022

    work page 2022

  2. [2]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ioannis Reid, Stephen Gould, and Anton van den Hengel. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019

    work page 2019

  3. [3]

    Lawrence Zitnick, and Devi Parikh

    Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. InIEEE International Conference on Computer Vision, 2015

    work page 2015

  4. [4]

    Phyre: A new benchmark for physical reasoning

    Anton Bakhtin, Anton Raichuk, Ruben Vieira, Anirudh Goyal, and Oriol Vinyals. Phyre: A new benchmark for physical reasoning. InAdvances in Neural Information Processing Systems, 2019

    work page 2019

  5. [5]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, and Arvind Neelakantan. Language models are few-shot learners. Technical report, OpenAI, 2020

    work page 2020

  6. [6]

    Patil, Peter Clark, and Wen tau Yih

    Yejin Choi, Daniel Khashabi, Vishal S. Patil, Peter Clark, and Wen tau Yih. Piqa: Reasoning about physical interactions from text. InInternational Conference on Learning Representations, 2020

    work page 2020

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, and Ashish Sabharwal. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Pith/arXiv arXiv 2018

  8. [8]

    Training verifiers to solve math word problems

    Karl Cobbe, John Schulman, Karan Goel, Jacob Hilton, and Pieter Abbeel. Training verifiers to solve math word problems. InAdvances in Neural Information Processing Systems, 2021

    work page 2021

  9. [9]

    Construction of a Surrogate Model: Multivariate Time Series Prediction with a Hybrid Model

    Tim Dettmers, Mike Lewis, Miguel Gonzalez, and Dominik Matusiak. Llm.int8(): 8-bit matrix multiplica- tion for transformers at scale.arXiv preprint arXiv:2212.07918, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  10. [10]

    Yen-Chun Fang, Chao-Han Huck Yang, and Christopher D. Manning. Benchmarking language–vision mod- els on multi-domain reasoning tasks. InProceedings of the AAAI Conference on Artificial Intelligence, 2019

    work page 2019

  11. [11]

    Measuring coding challenge competence of language models

    Dan Hendrycks, Colin Burns, Sunny Kadavath, Akhil Arora, Sergio Basart, Clement Berner, Xuezhi Cai, Michael Denkowski, Giulio Finocchiaro, and Deep Ganguli. Measuring coding challenge competence of language models. InProceedings of the International Conference on Artificial Intelligence and Statistics, 2021. 8

    work page 2021

  12. [12]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Pith/arXiv arXiv 2001

  13. [13]

    Procedural generation of physics problems for visual reasoning.IEEE Transactions on Visualization and Computer Graphics, 26(1):234–244, 2020

    Yuhang Lei, James Tompkin, and David Forsyth. Procedural generation of physics problems for visual reasoning.IEEE Transactions on Visualization and Computer Graphics, 26(1):234–244, 2020

    work page 2020

  14. [14]

    BLIP-2: Bootstrapped language–image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP-2: Bootstrapped language–image pre-training with frozen image encoders and large language models. InInternational Conference on Machine Learning, 2023

    work page 2023

  15. [15]

    Lee, and Tamara L

    Junnan Li, Pan Lu, Peter Anderson, Dahyun Kim, Stefanie T. Lee, and Tamara L. Berg. Nlvr2: A natural language for visual reasoning challenge. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, 2019

    work page 2019

  16. [16]

    Few-shot prompting for physics word problems with large language models

    Shang-Wen Li, Xin Wang, and Jiawei Chen. Few-shot prompting for physics word problems with large language models. InProceedings of the Annual Meeting of the ACL, 2021

    work page 2021

  17. [17]

    Scienceqa: A large-scale multimodal dataset for scientific question answering

    Pan Lu, Zisheng Lin, Zhiguo Wang, Jiawei Han, and Michael Zeng. Scienceqa: A large-scale multimodal dataset for scientific question answering. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 2021

    work page 2021

  18. [18]

    Mathvista: A benchmark for visualizing and reasoning about mathematical expressions

    Pan Lu, Zisheng Lin, Zhiguo Wang, Jiawei Han, Michael Zeng, and Jingjing Liu. Mathvista: A benchmark for visualizing and reasoning about mathematical expressions. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023

    work page 2023

  19. [19]

    Analyzing common failure modes in large language models.Transactions of the Association for Computational Lin- guistics, 11:612–633, 2023

    Aman Madaan, Ashwin Pant, Aditya Kalyanpur, Urvashi Khandelwal, and Mrinmaya Sachan. Analyzing common failure modes in large language models.Transactions of the Association for Computational Lin- guistics, 11:612–633, 2023

    work page 2023

  20. [20]

    Adaptive numerical integration for physics-based simu- lations.Journal of Computational Physics, 435:110260, 2021

    Stefan Müller, Thomas Neukam, and Sabine Lang. Adaptive numerical integration for physics-based simu- lations.Journal of Computational Physics, 435:110260, 2021

    work page 2021

  21. [21]

    Physion: Learning to predict physical interactions through video simulation

    Nikos Sarafianos, Armen Khachatrian, Amy Zhang, and Stephan Zheng. Physion: Learning to predict physical interactions through video simulation. InInternational Conference on Robotics and Automation, 2021

    work page 2021

  22. [22]

    Physical interaction question answering (piqa): A test of physical commonsense modeling

    Ankush Sharma, Yash Goyal, Mikhail Pavlov, and Kyunghyun Cho. Physical interaction question answering (piqa): A test of physical commonsense modeling. InConference on Empirical Methods in Natural Language Processing, 2021

    work page 2021

  23. [23]

    Chain-of-thought prompting elicits reasoning in large lan- guage models

    Maarten Wang, Disha Gupta, and Percy Liang. Chain-of-thought prompting elicits reasoning in large lan- guage models. InAdvances in Neural Information Processing Systems, 2022

    work page 2022

  24. [24]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Jacob Metcalf, and David Hernandez. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    Pith/arXiv arXiv 2022

  25. [25]

    A Comprehensive Survey on Multi-hop Machine Reading Comprehension Datasets and Metrics

    Haoyue Yu, Chandra Bhagavatula, Ammar A. Hassan, Qihang Yu, Ashish Sabharwal, Pedro A. Ortega, D. Sculley, Aravind Srinivas, Stephen Ibaraki, and Mingda Chen. Pali: A jointly-scaled multilingual lan- guage–image model.arXiv preprint arXiv:2212.04070, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  26. [26]

    Mmmu: A massive multi-discipline multi- modal understanding benchmark

    Xiang Yue, Ganesh Mani, Ashish Sabharwal, and Yelong Shen. Mmmu: A massive multi-discipline multi- modal understanding benchmark. InarXiv preprint arXiv:2311.16502, 2023

    Pith/arXiv arXiv 2023

  27. [27]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention

    Renrui Zhang, Xiang Liu, Pengyun Li, and Fei Xia. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. InInternational Conference on Learning Representations, 2023

    work page 2023

  28. [28]

    The cater dataset: A diagnostic dataset for compositional actions and temporal reasoning

    Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, and Li Fei-Fei. The cater dataset: A diagnostic dataset for compositional actions and temporal reasoning. InProceedings of the European Conference on Computer Vision, 2020. 9

    work page 2020

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.