TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models

Chengpei Tang; Keze Wang; Shiyuan Zhang; Zeqing Wang

REVIEW 4 major objections 5 minor 58 references

TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Current vision-language models cannot reliably judge which of two images came first and why, even when they handle standard visual questions well.

desk verdict A genuinely new benchmark for temporal causal reasoning in VLMs, with a strong and actionable position-bias finding, but the open-vs-closed gap is not yet cleanly separated from shortcut/artifact detection. read the letter →

arxiv 2505.15435 v1 pith:JIBZC2HE submitted 2025-05-21 cs.CV

Zeqing Wang , Shiyuan Zhang , Chengpei Tang , Keze Wang This is my paper

classification cs.CV

keywords temporalcausalityvision-languagemodelsbenchmarkcausalreasoningirreversibletransformationsimagepairspositionbiasVQAevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper introduces TimeCausality, a benchmark of 700 image pairs in which the same object appears before and after an irreversible real-world transformation, such as fruit rotting, metal rusting, or a person aging. Each pair comes with three questions: which image came first, why the later state occurred, and what caused the change. The reported results show that current vision-language models perform weakly and inconsistently on these questions, with many models answering correctly for one image order and wrong for the reversed order. The paper's central finding is that even the strongest closed-source model, GPT-4o, achieves only 43.43 on the group-consistency score for temporal-order judgment, while the best open-source model reaches 28.00. If the finding holds, standard visual question answering benchmarks are overstating the real-world visual reasoning abilities of these models.

What carries the argument

The machinery is the irreversible-transformation image pair: an original image and a GPT-4o-inpainted version of the same scene in which one object has undergone a causally irreversible change, with human-verified rationales for why the change is irreversible and what caused it. The benchmark also uses a three-aspect evaluation protocol: Aspect I asks the model to choose which image came first and tests both image orders, Aspect II asks for a free-text reason why the later state came after, and Aspect III asks what caused the transition. The consistency metrics Group Score and F1 combine accuracy on both the original and reversed image orders, so they expose position bias rather than rewarding it.

What would settle it

Take a random sample of TimeCausality pairs, strip the rationales, and have independent human annotators label which image came first. If human-human agreement falls below roughly 90%, the temporal order is not unambiguous, so the accuracy and consistency scores do not measure what the benchmark claims to measure.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that temporal causal reasoning—inferring the direction and cause of an irreversible state change from a pair of images—is a distinct capability that current vision-language models lack. On the 700-pair TimeCausality benchmark, GPT-4o reaches Group Score 43.43 and F1 67.83 on the multiple-choice temporal-order aspect, while the best open-source model, Qwen2.5-VL-7B, reaches Group Score 28.00 and F1 55.13; several models score above 70% accuracy on one image order and near 0% on the reversed order, indicating strong position bias. On the free-text reasoning and inferring aspects, all models receive low average ratings from the Llama3-8B judge, between roughly 0.4 and 2.8 on a 0–5 scale, with GPT-4o at 2.45 (reasoning) and 2.80 (inferring). The paper concludes that open-source VLMs have not closed the gap to closed-source models on this task and that temporal causality should be a separate evaluation axis.

Load-bearing premise

The benchmark assumes the GPT-4o-generated and human-verified image orders and rationales are unambiguous ground truth, and that Llama3-8B's semantic-similarity ratings measure the quality of causal reasoning rather than stylistic similarity to the reference answer.

Editorial extensions

If this is right

Accuracy on a single temporal-order question overstates ability; reporting consistency across reversed image orders is necessary to expose position bias.
Open-source VLMs need substantially better alignment between visual cues and temporal reasoning, not just better commonsense knowledge, to close the gap with closed-source models.
Temporal causality should be added as a distinct evaluation axis in VLM benchmarks, separate from spatial, scientific, and event-sequence reasoning.
Free-text reasoning scores are low across all models, indicating that explaining real-world state changes remains an open problem even for the best model tested.
The five causal categories (physical, chemical, natural, environmental, and artificial) provide a reusable taxonomy for future temporal benchmarks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An implication the authors leave implicit: the same three-aspect protocol could be applied to video or multi-frame inputs, where temporal order is given but the cause of a state change still has to be inferred.
Because all free-text scores are low, a human-evaluation subset would clarify whether the Llama3-8B judge scores reward reasoning content or merely LLM-like wording; the paper does not report judge-human agreement.
A testable extension: fine-tune an open-source VLM on temporal-causal rationales and measure whether Group Score improves while Aspect II and III scores stay flat, which would isolate the visual-alignment bottleneck the paper diagnoses.
The benchmark's design also suggests a broader use: irreversible transformations are a natural test bed for temporal consistency in image-editing models, not only for question answering.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The paper introduces TimeCausality, a benchmark for temporal-causal reasoning in VLMs built from COCO images and GPT-4o inpainting, with 700 human-verified pairs across five causal types. That is genuinely new: prior temporal benchmarks test ordering, speed, or attribute change, not irreversible causal transformations with explicit rationales. The three-aspect design (multiple-choice ordering, free-text reasoning, free-text inferring) and the reversed-pair consistency metrics are thoughtful, and the paper's strongest empirical observation is real: many models get high ACC and near-zero ACC-R, meaning they solve Aspect I by a fixed position preference. That is an actionable finding for the field.

The soft spots are the usual benchmark-construction suspects, and they matter. First, no artifact-leakage control. The 'later' image is always GPT-4o inpainting, so a model might be detecting edit artifacts rather than reasoning about time. Table 1 shows GPT-4o-mini at 98.57 ACC and 1.43 ACC-R, clearly a position-preference shortcut, and Llama3.2-vision behaves similarly. GPT-4o's balanced 70.86/64.86 is more credible as actual reasoning, but the gap between GPT-4o and the open-source group could still be inflated if GPT-4o is better at recognizing synthetic edits. The paper does not report an artifact control (e.g., real before/after pairs or a test with labels permuted), and the human verification in Appendix E checks plausibility and NSFW content, not artifact leakage.

Second, the LLM-as-a-judge for Aspects II/III is unvalidated: no correlation with human ratings, no inter-annotator agreement, and the judge is also an LLM (Llama3-8B) that may favor certain phrasing. Third, no error bars or significance tests anywhere; some of the cross-model differences on group score are large, but the per-sample variance is not reported. Fourth, the abstract's claim that GPT-4o shows a 'marked drop' on TimeCausality relative to other tasks is not quantified in the paper.

The circularity concern (GPT-4o generates rationales and edits, then is evaluated) is real but partially mitigated by human filtering; the paper should address it explicitly. I don't think the flaws kill the benchmark—the position-bias result is robust and useful regardless—but they do mean the headline open-vs-closed gap is not yet cleanly established.

This paper deserves a serious referee. It is a legitimate benchmark contribution with a clear diagnosis, and the missing controls are fixable. I would send it to review with a request for an artifact-leakage analysis, judge validation, and error bars. I'd bring it to reading group; it is a good case study in benchmark confounds.

Referee Report

4 major / 5 minor

Summary. The paper introduces TimeCausality, a benchmark of 700 image pairs constructed by editing COCO images with GPT-4o inpainting under five causal categories (Physical Change, Chemical Change, Natural Phenomenon, Environmental Modification, Artificial Processing). It evaluates VLMs on three aspects: temporal order judgment (Aspect I, multiple-choice tested in both image orders), causal reasoning (Aspect II, free-text 'why'), and causal inferring (Aspect III, free-text 'what'). The central empirical claim is that GPT-4o substantially outperforms open-source VLMs on Aspect I (Group Score 43.43 vs. 28.00 for Qwen2.5-VL-7B), while open-source models approach GPT-4o on Aspects II and III. The paper also reports extreme position bias for several models and concludes that current VLMs lack robust temporal-causal reasoning, with a notable gap for open-source models.

Significance. If the result holds, TimeCausality is a useful benchmark with a clear diagnostic: temporal-causal reasoning is distinct from standard VQA and from temporal perception, and current models are inconsistent across image order. The dataset is human-verified and publicly released, the three-aspect evaluation design is a sensible decomposition of temporal-causal ability, and the evaluation spans diverse open-source, closed-source, and contrastive models. The observation of extreme position bias in Table 1 is itself an interesting and actionable finding. However, the headline open-versus-closed gap is currently vulnerable to editing-artifact leakage and relies on an unvalidated automated judge, so the significance is conditional on the additional controls requested below.

major comments (4)

[§3.2.1, Appendix E] The headline Table 1 comparison is not yet secured against a shortcut: each 'later' image is produced by GPT-4o inpainting the COCO original, and the manual filter in Appendix E checks temporal order, content consistency, rationales, and NSFW content, but does not test whether low-level editing artifacts (seams, color and lighting shifts, geometry changes) encode the temporal label independently of causal semantics. The position-bias cases in Table 1 (e.g., Llama3.2-vision-11B with ACC 79.14 and ACC-R 1.43; GPT-4o-mini with ACC 98.57 and ACC-R 1.43) demonstrate that non-reasoning shortcuts dominate for several models, so the GPT-4o group-score advantage (43.43 vs. 28.00) can be attributed to temporal-causal reasoning only after an artifact-leakage control is reported, such as a simple perceptual classifier trained to distinguish original from edited images, or human annotation of whether the visible cues are genuinely causal.
[§4.2, Appendix D.2] The Aspect II and III scores are produced by an unvalidated Llama3-8B judge that rates 0–5 semantic similarity against GPT-4o-generated ground-truth rationales. No inter-annotator agreement, no judge-versus-human correlation, and no error analysis are reported. Because the paper uses these scores to claim that open-source models reach near-GPT-4o competence on 'why' and 'what' reasoning (Table 1: Qwen2.5-VL-7B 2.23/2.29 vs. GPT-4o 2.45/2.80), that conclusion is trustworthy only if the judge is shown to agree with human judgments; otherwise the scores may reflect stylistic similarity to LLM-generated rationales rather than reasoning quality.
[§3.2.1, §4.1] The benchmark construction uses GPT-4o for editing instructions, rationales, and inpainting, and GPT-4o is then the top-performing evaluated model on the same benchmark. Human verification provides some independence, but it does not eliminate the risk that the ground truth and the edited images are tailored to GPT-4o's editing and reasoning style; this is especially relevant for Aspect I, where GPT-4o's 70.86/64.86 ACC/ACC-R may partly reflect better recognition of its own generated outputs. Please report a robustness check, for example independent manual re-annotation of a sample or evaluation on a subset generated without GPT-4o, and state whether the ranking changes.
[Table 1, §4.3] All reported numbers are single-run point estimates without confidence intervals, repeated runs, or significance tests. Given stochastic decoding and API variability, the 'huge gap' between GPT-4o and open-source models is not yet quantified with uncertainty; please report at least three runs on a random subset or bootstrap confidence intervals for the main metrics.

minor comments (5)

[Throughout] The benchmark name alternates between TimeCausality and TimeCasualty (e.g., §2.2, §3.3, Table 1 caption, Figure 6); please standardize to TimeCausality.
[§3.4] The data statistics are described as 'In Table 6', but they are presented in Figure 6; please correct the cross-reference.
[Throughout] There are typographical errors including 'Evalutation' in §3.3, 'banna' in Figure 5, and 'Infferring' in Figure 7; a careful proofread is needed.
[References] The reference entry for 'Yifan Li, Yifan Du and Ji-Rong Wen' is malformed, as the author names appear to be merged incorrectly; please fix the citation.
[Table 1 caption] The caption's caveat that ACC and ACC-R may not reflect true performance due to position bias is important and should be explained in the main text before the results are presented, rather than appearing only in the table caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: TimeCausality is an empirical benchmark study, and its central claims are measured against human-verified data rather than derived from fitted parameters, self-citations, or definitional equivalences.

full rationale

The paper is a benchmark construction and evaluation study, not a derivation chain. The dataset is built as: COCO source images are auto-labeled, an LLM (GPT-4o) generates editing instructions and rationales, GPT-4o performs inpainting, and the resulting image pairs and rationales are manually verified by human annotators following the Appendix E guidelines. The central empirical claims — that open-source VLMs trail GPT-4o on the benchmark and that GPT-4o itself drops relative to other tasks — are measured outcomes on this fixed benchmark, not predictions recovered from fitted parameters. No equation in the paper defines a metric in terms of the model outputs it is used to explain, and no parameter is fitted to a subset of the benchmark and then reported as a prediction. GPT-4o's role as both data generator and evaluated model is a benchmark-contamination/validity concern, but it is not a circular reduction: the ground-truth labels are human-verified independently of model predictions, the free-text judge is a different model (Llama3-8B), and the paper does not claim that GPT-4o's performance is guaranteed by its authorship of the rationales. There are no load-bearing self-citations, no imported uniqueness theorems from the authors' prior work, and no ansatz smuggled in through citation. Accordingly, the appropriate finding is no significant circularity, with score 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The paper's claims rest on four domain assumptions about dataset validity and judge validity.

assumptions (4)

domain assumption The five causal types (CC, EM, AP, NP, PC) are an adequate taxonomy of temporal causality.
Section 3.1 defines these categories without evidence of completeness or mutual exclusivity.
domain assumption GPT-4o-generated rationales and image edits, after human filtering, are correct and unambiguous ground truth for temporal order.
Section 3.2.1 describes human verification but gives no agreement statistics; the annotation guidelines in Appendix E are qualitative.
domain assumption Llama3-8B's 0-5 semantic similarity ratings are a valid proxy for answer correctness in Aspects II and III.
Section 4.2 and Appendix D.2 use this judge without validation against human ratings or inter-annotator agreement.
domain assumption The inpainting model produces visually plausible later states that do not introduce detection artifacts dominant enough to solve the task without causal reasoning.
The pipeline in Section 3.2.1 relies on GPT-4o inpainting; artifact analysis is not provided.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models." pith.science (2026). https://pith.science/paper/JIBZC2HE

@misc{pith2026250515435,
  author       = {Pith},
  title        = {Pith review of: TimeCausality: Evaluating the Causal Ability in Time Dimension for Vision Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JIBZC2HE}},
  note         = {Machine review of arXiv:2505.15435}
}

read the original abstract

Reasoning about temporal causality, particularly irreversible transformations of objects governed by real-world knowledge (e.g., fruit decay and human aging), is a fundamental aspect of human visual understanding. Unlike temporal perception based on simple event sequences, this form of reasoning requires a deeper comprehension of how object states change over time. Although the current powerful Vision-Language Models (VLMs) have demonstrated impressive performance on a wide range of downstream tasks, their capacity to reason about temporal causality remains underexplored. To address this gap, we introduce \textbf{TimeCausality}, a novel benchmark specifically designed to evaluate the causal reasoning ability of VLMs in the temporal dimension. Based on our TimeCausality, we find that while the current SOTA open-source VLMs have achieved performance levels comparable to closed-source models like GPT-4o on various standard visual question answering tasks, they fall significantly behind on our benchmark compared with their closed-source competitors. Furthermore, even GPT-4o exhibits a marked drop in performance on TimeCausality compared to its results on other tasks. These findings underscore the critical need to incorporate temporal causality into the evaluation and development of VLMs, and they highlight an important challenge for the open-source VLM community moving forward. Code and Data are available at \href{https://github.com/Zeqing-Wang/TimeCausality }{TimeCausality}.

Figures

Figures reproduced from arXiv: 2505.15435 by the authors.

**Figure 2.** Causal types in our TimeCausality benchmark. We identify five dimensions of temporal causality: Artificial Processing (AP), Chemical Change (CC), Natural Phenomenon (NP), Environmental Modification (EM), and Physical Change (PC). The image depicting the causal result is generated by modifying the original using an inpainting model. The object marked in RED denotes the primary target of causality. conduct careful cur… view at source ↗

**Figure 3.** Illustration of TimeCausality generation pipeline. First, we use a grounding model with an auto-labeled [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: Evaluation aspects of our TimeCausality. [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 4.** Figure 4: Data demo and case study in our TimeCausality. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 6.** Figure 6: Data statistics in our TimeCasualty. Together, these three question types provide a holistic framework to evaluate VLMs’ temporalcausal reasoning abilities. Overall, the Aspect I is in the format of multiple-choice, and Aspect II & III are in free-text format. 3.4 Dat…

**Figure 7.** Figure 7: More Cases in Our Time Causality * https://huggingface.co/ * https://github.com/InternLM/lmdeploy 1 [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

58 extracted references · 10 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

arXiv 2024
[4]

Anthropic. 2024. https://www.anthropic.com/news/claude-3-family Claude3 system card

work page 2024
[5]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV), pages 2425--2433

work page 2015
[6]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023
[7]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025 a . Qwen2.5-vl technical ...

arXiv 2025
[8]

Zechen Bai, Hai Ci, and Mike Zheng Shou. 2025 b . Impossible videos. arXiv preprint arXiv:2503.14378

arXiv 2025

Show all 58 references

[9]

Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, et al. 2024. Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954

2024 arXiv
[10]

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

2020
[11]

Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. 2024 a . How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):220101

2024
[12]

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. 2024 b . Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Com...

2024
[13]

Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416

2022 arXiv
[14]

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2010. An image is worth 16x16 words: Transformers for image recognition at scale. arxiv 2020. ar...

2010 arXiv
[16]

Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406.12793

2024 arXiv
[17]

Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Computer Vision and Pattern Recognition (CVPR), pages 6904--6913

2017
[18]

Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[19]

Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. 2024. Efficient multimodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530

2024 arXiv
[20]

Wenyi Hong, Weihan Wang, Ming Ding, Wenmeng Yu, Qingsong Lv, Yan Wang, Yean Cheng, Shiyu Huang, Junhui Ji, Zhao Xue, et al. 2024. Cogvlm2: Visual language models for image and video understanding. arXiv preprint arXiv:2408.16500

2024 arXiv
[21]

Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Computer Vision and Pattern Recognition (CVPR), pages 6700--6709

2019
[22]

Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023. Mistral 7b. arXiv preprint arXiv:2310.06825

2023 arXiv
[23]

Baiqi Li, Zhiqiu Lin, Wenxuan Peng, Jean de Dieu Nyandwi, Daniel Jiang, Zixian Ma, Simran Khanuja, Ranjay Krishna, Graham Neubig, and Deva Ramanan. 2024 a . Naturalbench: Evaluating vision-language models on natural adversarial samples. In The Thirty-eight Conference on Neural...

2024
[24]

Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 a . Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125

2023 arXiv
[25]

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023 b . Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890

2023 arXiv
[26]

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 c . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597

2023 arXiv
[27]

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. 2024 b . Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages...

2024
[28]

Shicheng Li, Lei Li, Shuhuai Ren, Yuanxin Liu, Yi Liu, Rundong Gao, Xu Sun, and Lu Hou. 2023 d . Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. arXiv preprint arXiv:2311.17404

2023 arXiv
[29]

Yuan-Hong Liao, Rafid Mahmood, Sanja Fidler, and David Acuna. 2024. Reasoning paths with reference objects elicit quantitative spatial reasoning in large vision-language models. arXiv preprint arXiv:2409.09788

2024 arXiv
[30]

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll \'a r, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Computer vision--ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, ...

2014
[31]

Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024 a . https://llava-vl.github.io/blog/2024-01-30-llava-next/ Llava-next: Improved reasoning, ocr, and world knowledge

2024
[32]

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. 2024 b . Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476

2024 arXiv
[33]

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022 a . Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 3...

2022
[34]

Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022 b . Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neural Information Processing...

2022
[35]

Gen Luo, Xue Yang, Wenhan Dou, Zhaokai Wang, Jiawen Liu, Jifeng Dai, Yu Qiao, and Xizhou Zhu. 2024. Mono-internvl: Pushing the boundaries of monolithic multimodal large language models with endogenous visual pre-training. arXiv preprint arXiv:2410.08202

2024 arXiv
[36]

Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. 2023. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint arXiv:2311.16103

2023 arXiv
[37]

Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and Others

OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, and Others. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

2024 arXiv
[38]

OpenAI. 2023. https://openai.com/research/gpt-4v-system-card GPT-4V(ision) system card

2023
[39]

OpenAI. 2024. Hello gpt-4o. https://openai.com/index/hello-gpt-4o/

2024
[40]

Long Phan, Alice Gatti, Ziwen Han, et al. 2025. https://arxiv.org/abs/2501.14249 Humanity's last exam . Preprint, arXiv:2501.14249

2025 arXiv
[41]

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learni...

2021
[42]

Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang, Hongyang Li, Qing Jiang, and Lei Zhang. 2024. https://arxiv.org/abs/2401.14159 Grounded sam: Assembling open-world ...

2024 arXiv
[43]

Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv

2022
[44]

Kai Sun, Yushi Bai, Ji Qi, Lei Hou, and Juanzi Li. 2024. Mm-math: Advancing multimodal math evaluation with process evaluation and fine-grained classification. arXiv preprint arXiv:2404.05091

2024 arXiv
[45]

Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

2023 arXiv
[46]

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. 2024 a . Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530

2024 arXiv
[47]

Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivi \`e re, Mihir Sanjay Kale, Juliette Love, et al. 2024 b . Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295

2024 arXiv
[48]

Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, Austin Wang, Rob Fergus, Yann LeCun, and Saining Xie. 2024. https://arxiv.org/abs/2406.16860 Cambrian-1: A fully open, vision-centri...

2024 arXiv
[49]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023 a . Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

2023 arXiv
[50]

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023 b . Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[51]

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024. Qwen2-vl: Enhancing vision-language mode...

2024 arXiv
[52]

Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. 2024. https://arxiv.org/abs/2404.16994 Pllava : Parameter-free llava extension from images to videos for video dense captioning . Preprint, arXiv:2404.16994

2024 arXiv
[53]

Kun Zhou Jinpeng Wang Wayne Xin Zhao Yifan Li, Yifan Du and Ji-Rong Wen. 2023. https://openreview.net/forum?id=xozJw0kZXF Evaluating object hallucination in large vision-language models . In The 2023 Conference on Empirical Methods in Natural Language Processing

2023
[54]

Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024. Mmm...

2024
[55]

Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414

2022 arXiv
[56]

Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Ge Zhang, Linyang Li, et al. 2024. Anygpt: Unified multimodal llm with discrete sequence modeling. arXiv preprint arXiv:2402.12226

2024 arXiv
[57]

Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. 2024. Vision-language models for vision tasks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence

2024
[58]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

2023
[59]

Chengke Zou, Xingang Guo, Rui Yang, Junyu Zhang, Bin Hu, and Huan Zhang. 2024. Dynamath: A dynamic visual benchmark for evaluating mathematical reasoning robustness of vision language models. arXiv preprint arXiv:2411.00836

2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

arXiv 2024

[4] [4]

Anthropic. 2024. https://www.anthropic.com/news/claude-3-family Claude3 system card

work page 2024

[5] [5]

Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In International Conference on Computer Vision (ICCV), pages 2425--2433

work page 2015

[6] [6]

Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. 2023. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966

arXiv 2023

[7] [7]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. 2025 a . Qwen2.5-vl technical ...

arXiv 2025

[8] [8]

Zechen Bai, Hai Ci, and Mike Zheng Shou. 2025 b . Impossible videos. arXiv preprint arXiv:2503.14378

arXiv 2025