Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

CAViAR claims that pairing an iterative program-writing agent with a critic that selects among sampled video-reasoning strategies lifts accuracy on long-video benchmarks, reaching 62.0 percent on LVBench versus a prior best of 48.7.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

CAViAR, an agent-plus-critic system for long video reasoning, improves on direct video LLM inference across LVBench, Neptune, and ActivityNet-RTL.

T0 review reviewed 2026-08-04 challenge →

load-bearing objection A plausible agent+critic recipe for long-video reasoning with strong numbers, but the critic's few-shot examples look like they could come from the test sets, and until that is disclosed the headline gains are not independent. the 4 major comments →

arxiv 2509.07680 v1 pith:FGRRPB6L submitted 2025-09-09 cs.CV cs.LG

CAViAR: Critic-Augmented Video Agentic Reasoning

Sachit Menon , Ahmet Iscen , Arsha Nagrani , Tobias Weyand , Carl Vondrick , Cordelia Schmid This is my paper
classification cs.CV cs.LG
keywords video reasoninglong-video question answeringagentic reasoningtool usereasoning critictemporal localizationmultimodal large language modelsinference-time scaling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

CAViAR is a method for complex long-video reasoning that adds no training: a language-model agent writes and runs short Python programs that call video modules such as frame retrieval, temporal grounding, localization, and ASR understanding, reading each tool's output before choosing the next step. Because a single run can fail, the method samples several module subsets to produce distinct reasoning traces, then a natural-language critic compares the traces and selects the most promising one. The paper reports 62.0 percent accuracy on LVBench versus 48.7 for the prior best, gains over direct inference on Neptune and ActivityNet-RTL, and ablations showing the critic is responsible for most of the improvement. If these numbers hold, existing perception models can be assembled at inference time to outperform much larger or specially trained systems on reasoning-heavy video tasks.

Core claim

On its own terms, the work claims that adaptive, verifiable reasoning beats fixed procedures for long-video understanding: a reasoning agent that iteratively composes video modules into executable programs, observing each result before deciding the next step, outperforms single-shot program generation, and adding a critic that chooses among multiple such reasoning traces yields the largest gains. The key empirical evidence is a 13-point absolute improvement on LVBench over the prior state of the art (62.0 vs 48.7), plus improvements over direct inference on Neptune (77.2 vs 74.9 with ASR manually provided) and on ActivityNet-RTL temporal localization (32.3 vs 23.0 mIOU). Ablations show the a

What carries the argument

The central mechanism is a reasoning trace: a full sequence of programs and tool results produced by an agent that can inspect each result before writing the next program. The second component is the reasoning critic, a same-base-model natural-language ranker that is shown several sampled traces at once and, using a handful of in-context examples per dataset, names which strategies are likely to win. The video modules—retrieval_qa, get_segment, find_when, asr_understanding, think, and finish—are the tools the agent composes.

Load-bearing premise

The critic's four in-context examples per dataset must teach it which module strategies succeed on the evaluation queries without being drawn from those same test sets; the supplement calls them author-written and in-distribution, but their content resembles actual benchmark questions, so leakage would invalidate the reported gains.

What would settle it

Run CAViAR on LVBench with critic examples provably held out from the evaluation set—written from a different set of long videos—and check whether the 62.0 percent accuracy and the 13-point gap over the prior best persist. As a direct check, determine whether the critic's staged 'host suit color at 12:10' example appears in the LVBench evaluation set; if it does, the critic has been calibrated on test-like items.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • The same base model can be scaled at inference time without any training: Gemini 1.5 Flash goes from 46.0 to 62.0 on LVBench and from 23.0 to 32.3 mIOU on ActivityNet-RTL under CAViAR.
  • Single-program approaches fail on these tasks: generating one fixed program yields near-random accuracy (27.1 on LVBench, 28.7 on Neptune), whereas the iterative agent recovers to 47.1 and 72.5 respectively.
  • The critic is the load-bearing addition: replacing it with confidence-based self-evaluation drops LVBench accuracy from 47.1 to 39.9, while the critic raises it to 62.0.
  • Critic-based strategy selection outperforms even the best fixed module subset found with oracle validation, suggesting the gain comes from per-query flexibility rather than from tuning module choice per dataset.
  • The approach is model-agnostic in principle: a preliminary experiment with GPT-4o-mini on 25% of LVBench still shows a substantial gain over direct inference.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the critic's in-context examples are genuinely held out from the evaluation sets, CAViAR's procedure should transfer to new datasets with only a handful of hand-written examples; a direct test is to swap the critic examples for ones written from a separate pool of long videos and measure the accuracy change.
  • The same generate-and-critique loop could apply to other settings where tool outputs are noisy and strategies diverge, such as embodied instruction following or document question answering, because the critic only reads text traces and never re-processes the video.
  • The gap between agent-plus-optimal-fixed-modules (59.8 on LVBench) and agent-plus-critic (62.0) suggests that sampling more than three strategies, or making strategies differ more sharply, could yield further gains—an extension the paper does not test.
  • One open question the paper's design raises, not settled by its experiments, is leakage: the critic's examples resemble actual benchmark items, so their provenance determines whether the reported gains measure generalization or test-set fitting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces CAViAR, an inference-time framework for long-video reasoning that pairs a program-generating LLM agent with a natural-language critic. The agent iteratively composes executable programs from a small set of video modules (retrieval-QA, temporal grounding, temporal localization, ASR understanding), while the critic selects among several sampled reasoning traces produced from different module subsets. Experiments are reported on LVBench, Neptune, and ActivityNet-RTL, with additional results on EgoSchema and GPT-4o-mini in the supplement. The main claims are that CAViAR improves over direct inference with the same base model on all three tasks and achieves a 62.0% accuracy on LVBench, a 13.0-point absolute gain over the reported state of the art, and 32.3 mIOU on ActivityNet-RTL. The ablations show that the critic is the main driver of the LVBench gain (47.1 without critic vs 62.0 with critic) and that fixed single-program generation is far weaker.

Significance. If the reported numbers are reliable and independent, the paper makes a useful contribution: it demonstrates that iterative, adaptive tool use plus a lightweight critic can outperform much larger or specially trained video models on long-video reasoning, without additional training. The method is interpretable, and the ablations are well designed: comparing against single-program baselines, an oracle fixed-module selection, and a confidence self-evaluation baseline gives the reader a clear picture of where the gains come from. The supplement provides full prompts, module APIs, and critic examples, which is a genuine strength. The central concern is that the critic's only supervision is four in-context examples per dataset, whose provenance is not disclosed; because LVBench has no training/validation split, the examples may have been selected from or authored after inspecting the evaluation distribution. Since the critic is responsible for most of the reported improvement, the headline SOTA claim is not independently verifiable as written. The paper also does not provide code or error bars, further limiting verification.

major comments (4)
  1. [Supplement §4, Listing 4; §4.1] The critic's only supervision is four in-context examples per dataset, and the supplement states these are 'written by the authors with the aim of being in-distribution for the datasets shown.' The printed examples are not generic templates: they contain concrete LVBench-style queries (e.g., Jimmy Kimmel's suit color at 12:10, Harry Styles and a photo of six people, a vlogger buying makeup at Kicks), and §4.1 states LVBench has no training/validation split. If any of these examples were drawn from the evaluation set, or authored after inspecting it, the critic is effectively fitted to the test distribution, and the 62.0 vs 47.1 critic gain in Table 4 is not an independent measure of the method. The manuscript must disclose the exact source of every critic example, should rule out test-set overlap, and ideally should re-run with a development set or with examples that are clearly not draw
  2. [Supplement Listing 7 (EgoSchema critic examples)] The section labelled 'EgoSchema critic examples' repeats the LVBench in-context examples verbatim, including the Jimmy Kimmel suit-color and vlogger/makeup items. If this is not a formatting artifact of the supplement, then the EgoSchema critic was given LVBench examples, contradicting the claim that the examples are 'in-distribution for the datasets shown.' This also reinforces the need for full provenance disclosure of critic examples for every dataset.
  3. [§5, Table 4; NeurIPS Checklist §7] All headline results are single point estimates. The checklist justifies the absence of error bars by stating that experiments are 'run in a deterministic setting,' but the method invokes LLMs with sampling and multiple strategy traces, and no code is released to reproduce the runs. Given that the central claim is a 15-point gap on LVBench and a 9-point mIOU gain, the paper should report variance across repeated runs (or provide code and exact API configuration so the runs can be independently reproduced). Without this, it is impossible to assess whether the gains are stable or partly due to a favorable random seed / example choice.
  4. [§3.4, §5, Table 4] The strategy subsets given to the reasoning agent are a per-dataset design choice that is not specified a priori. The oracle 'Optimal Fixed Modules' row shows that much of the critic's gain can be replicated by choosing the right fixed module subset (59.8 vs 62.0 on LVBench; 76.5 vs 77.2 on Neptune). The paper does not report how the three subsets were selected, or whether this selection was made before or after seeing evaluation results. Since the contribution claims to 'avoid dataset-specific tuning,' the subset selection procedure should be stated in the method and, ideally, validated on a held-out development set.
minor comments (4)
  1. [§1, Introduction] Typo: 'in a a variety of domains' should read 'in a variety of domains.'
  2. [Supplement Listing 4, Strategy B critique] The critique text says 'Strategy B fails as the retrieval_qa tool alone cannot actually locate a given timestamp in a video,' but Strategy B is labelled 'direct retrieval_qa' and does not explicitly call retrieval_qa as a tool. Clarify the intended strategy description so readers can follow why the tool cannot localize timestamps.
  3. [Table 4 caption] The caption states items in italics require oracle validation accuracy, but the difference between 'Single Program (Optimal Fixed Modules)' and 'Agent (Optimal Fixed Modules)' is not explained in the main text. A sentence defining how the optimal fixed subset is found would improve interpretability.
  4. [Supplement §5.2] The GPT-4o-mini result is reported on 25% of LVBench without specifying how the 25% subset was sampled. Please state the sampling method and, if possible, whether the same subset is used for direct inference and CAViAR.

Circularity Check

1 steps flagged

LVBench SOTA gain is carried by critic examples that are, by the paper's own description, in-distribution for a benchmark with no training set.

specific steps
  1. fitted input called prediction [Supplement Section 4 (Critic Prompts), combined with Section 4.1 and Table 4]
    "The critic is given 4 in-context examples per prompt. These examples are written by the authors with the aim of being in-distribution for the datasets shown. ... No supervised methods are available to compare to as there is no training set."

    The headline LVBench gain (62.0 vs 47.1 for the agent alone, Table 4) is attributed to the critic's strategy selection. The critic's only task-specific supervision is four hand-written examples per dataset that label which strategies are 'winning.' The paper states LVBench has no training set, so 'in-distribution' examples cannot come from a legitimate development split; the only available distribution is the evaluation set. Thus the critic's selection rule is fitted to the evaluation distribution by construction, and the reported SOTA accuracy is not an independent measurement of the method. The fitted component (critic examples) is presented as a general critic that 'avoids dataset-specific tuning,' but the examples are dataset-specific and, as written, encode evaluation-distribution kno

full rationale

The derivation chain is otherwise self-contained: the reasoning agent iteratively calls video modules, and the critic compares the resulting traces. No formal result is derived from itself, and there is no self-citation chain or uniqueness argument. However, the central empirical claim on LVBench reduces to the critic's selection policy: without the critic the agent scores 47.1, with it 62.0 (Table 4). The critic's only learned input is four in-context examples per dataset, explicitly described as 'written by the authors with the aim of being in-distribution for the datasets shown.' Since LVBench has no training set (Section 4.1), the in-distribution examples must be drawn from or patterned after the evaluation queries, making the critic's strategy selector fitted to the test distribution. The paper does not disclose any other source for these examples. This is a partial circularity: the underlying visual modules are external and the method could be valid with properly held-out examples, but as written the main SOTA result is not an independent measure of the proposed method.

Axiom & Free-Parameter Ledger

3 free parameters · 3 axioms · 0 invented entities

CAViAR introduces no new physical entities. The free parameters are the per-dataset strategy subsets, the per-dataset critic in-context examples, and the module window sizes; all are hand-set components. The key axioms are domain assumptions about module reliability and about the critic examples not being drawn from the test set.

free parameters (3)
  • Strategy module subsets = 3 per dataset (e.g., LVBench: {get_segment, retrieval_qa}, {retrieval_qa}, {get_segment, retrieval_qa, find_when})
    Hand-designed per task; determines what strategies the critic chooses among; includes a per-dataset choice of which modules are relevant.
  • Critic in-context examples = 4 per dataset, printed in full in the Supplement
    Hand-written critiques with winning strategies; their source is undisclosed and they resemble evaluation queries, making them a potential test-set fitted component.
  • Module window sizes = find_when: 100-frame window; retrieval_qa: 64-frame window plus 56 uniform frames; base context 32k tokens
    Chosen by authors; affects module output quality and therefore the whole agent trace.
axioms (3)
  • domain assumption The video modules retrieval_qa, find_when, asr_understanding, and get_segment perform their described functions when prompted on the test videos.
    The entire agent trace is a composition of these prompted modules; if a module hallucinates or malfunctions, the trace fails (as the paper itself notes for find_when on long videos).
  • domain assumption The critic's four in-context examples per dataset are in-distribution but not drawn from the evaluation set.
    This is the load-bearing premise for the critic's generalization; the paper does not state the provenance of the examples.
  • domain assumption The evaluation sets used are the official test splits, and no strategy or critic example selection was performed using the evaluation labels.
    Needed for the reported accuracies to be unbiased; the paper does not explicitly rule out development on the evaluation set.

reviewed 2026-08-04 · how reviews work

0 comments
Cite this review

Pith. "Pith review of CAViAR: Critic-Augmented Video Agentic Reasoning." pith.science (2026). https://pith.science/paper/FGRRPB6L

@misc{pith2026250907680,
  author       = {Pith},
  title        = {Pith review of: CAViAR: Critic-Augmented Video Agentic Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGRRPB6L}},
  note         = {Machine review of arXiv:2509.07680}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Video understanding has seen significant progress in recent years, with models' performance on perception from short clips continuing to rise. Yet, multiple recent benchmarks, such as LVBench, Neptune, and ActivityNet-RTL, show performance wanes for tasks requiring complex reasoning on videos as queries grow more complex and videos grow longer. In this work, we ask: can existing perception capabilities be leveraged to successfully perform more complex video reasoning? In particular, we develop a large language model agent given access to video modules as subagents or tools. Rather than following a fixed procedure to solve queries as in previous work such as Visual Programming, ViperGPT, and MoReVQA, the agent uses the results of each call to a module to determine subsequent steps. Inspired by work in the textual reasoning domain, we introduce a critic to distinguish between instances of successful and unsuccessful sequences from the agent. We show that the combination of our agent and critic achieve strong performance on the previously-mentioned datasets.

Figures

Figures reproduced from arXiv: 2509.07680 by Ahmet Iscen, Arsha Nagrani, Carl Vondrick, Cordelia Schmid, Sachit Menon, Tobias Weyand.

Figure 1
Figure 1. Figure 1: CAViAR consists of a reasoning agent that produces sequences of programs to solve video queries with different strategies, followed by a critic that selects the most promising reasoning. Each program invokes visual modules that use the video as input, rather than it being providing as input a single time at the start. We show two strategies here for illustration. video modules into sequences of executable … view at source ↗
Figure 2
Figure 2. Figure 2: The CAViAR system. The reasoning agent generates reasoning traces to solve the query using video modules. The critic selects the best, yielding a final answer. Given a video and a prompt, the reasoning agent iteratively gener￾ates and executes programs, us￾ing provided video-processing mod￾ules and a Python interpreter to ultimately converge to a final an￾swer. We refer to the resulting se￾quences as reaso… view at source ↗
Figure 3
Figure 3. Figure 3: Reasoning temporal localization. Queries require identifying a time range for events/ac￾tions that requires reasoning. Here, the critic chooses a strategy which initially finds multiple ranges but correctly reasons which should be included in a final answer. 7 [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Module reliability. The find_when module often reports substantial amounts of tangen￾tially relevant, distracting information when applied to long videos, often leading the agent with all modules to failure. Reasoning agent vs single program How does generating a single program rather than using the reasoning agent to produce a reasoning trace perform? We present the base model the same API for the modules… view at source ↗
Figure 5
Figure 5. Figure 5: Modular assumptions hinder single program performance. In this case, the program assumes the answer appears within the first 15 seconds and that retrieval_qa output includes it exactly. Finding no birds in that segment, it lacks viable answers and resorts to a guess. Confidence self-evaluation vs critic. Wang et al. [30] introduce a ‘confidence’ module for deciding when to lock in a final answer. Rather th… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. EventCoT: Event-centric Video Chain-of-thought for Reasoning Temporal Localization

    cs.CV 2026-07 conditional novelty 6.5

    Event-centric tokenization plus two-step embedding matching lets a video-LLM jointly answer and timestamp RTL queries while using under 20% of LITA’s visual tokens.

Reference graph

Works this paper leans on

40 extracted references · 12 canonical work pages · cited by 1 Pith paper · 1 internal anchor

  1. [1]

    Egoschema: A diagnostic benchmark for very long-form video language understanding

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36: 0 46212--46244, 2023

    2023

  2. [2]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pages 58--76. Springer, 2024 a

    work page 2024

  3. [4]

    Self-chained image-language model for video localization and question answering

    Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36: 0 76749--76771, 2023

    work page 2023

  4. [5]

    A simple llm framework for long-range video question-answering

    Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple llm framework for long-range video question-answering. arXiv preprint arXiv:2312.17235, 2023

    Pith/arXiv arXiv 2023

  5. [6]

    Qwen2.5-vl technical report

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. a...

    Pith/arXiv arXiv 2025

  6. [7]

    How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites

    Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821, 2024

    Pith/arXiv arXiv 2024

  7. [8]

    Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476, 2024

    Pith/arXiv arXiv 2024

  8. [9]

    Deep reinforcement learning from human preferences

    Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017

    2017

  9. [10]

    Training verifiers to solve math word problems, 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/abs/2110.14168

    Pith/arXiv arXiv 2021

  10. [11]

    Visual programming: Compositional visual reasoning without training

    Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14953--14962, 2023

    work page 2023

  11. [12]

    V-star: Training verifiers for self-taught reasoners

    Arian Hosseini, Xingdi Yuan, Nikolay Malkin, Aaron Courville, Alessandro Sordoni, and Rishabh Agarwal. V-star: Training verifiers for self-taught reasoners. arXiv preprint arXiv:2402.06457, 2024

    Pith/arXiv arXiv 2024

  12. [13]

    Avis: Autonomous visual information seeking with large language model agent

    Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David Ross, Cordelia Schmid, and Alireza Fathi. Avis: Autonomous visual information seeking with large language model agent. Advances in Neural Information Processing Systems, 36: 0 867--878, 2023

    work page 2023

  13. [14]

    Lita: Language instructed temporal-localization assistant

    De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo Molchanov, Zhiding Yu, and Jan Kautz. Lita: Language instructed temporal-localization assistant. In European Conference on Computer Vision, pages 202--218. Springer, 2024

    work page 2024

  14. [15]

    Large language models cannot self-correct reasoning yet

    Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798, 2023

    Pith/arXiv arXiv 2023

  15. [16]

    Gpt-4o system card

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024

    Pith/arXiv arXiv 2024

  16. [17]

    Inferring and executing programs for visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE international conference on computer vision, pages 2989--2998, 2017

    work page 2017

  17. [18]

    When can llms actually correct their own mistakes? a critical survey of self-correction of llms

    Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. When can llms actually correct their own mistakes? a critical survey of self-correction of llms. Transactions of the Association for Computational Linguistics, 12: 0 1417--1440, 2024

    2024

  18. [19]

    Analyzing Modular Approaches for Visual Question Decomposition

    Apoorv Khandelwal, Ellie Pavlick, and Chen Sun. Analyzing modular approaches for visual question decomposition. ArXiv, abs/2311.06411, 2023. URL https://api.semanticscholar.org/CorpusID:265150517

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [20]

    Llava-onevision: Easy visual task transfer

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024

    Pith/arXiv arXiv 2024

  20. [21]

    Visual instruction tuning

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36: 0 34892--34916, 2023

    2023

  21. [22]

    Kangaroo: A powerful video-language model supporting long-context video input

    Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542, 2024

    Pith/arXiv arXiv 2024

  22. [23]

    Morevqa: Exploring modular reasoning models for video question answering

    Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reasoning models for video question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13235--13245, 2024

    work page 2024

  23. [24]

    Neptune: The long orbit to benchmarking long video understanding

    Arsha Nagrani, Mingda Zhang, Ramin Mehran, Rachel Hornung, Nitesh Bharadwaj Gundavarapu, Nilpa Jha, Austin Myers, Xingyi Zhou, Boqing Gong, Cordelia Schmid, et al. Neptune: The long orbit to benchmarking long video understanding. arXiv preprint arXiv:2412.09582, 2024

    Pith/arXiv arXiv 2024

  24. [25]

    Learning Transferable Visual Models From Natural Language Supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision . arXiv:2103.00020 [cs], February 2021. URL http://arxiv.org/abs/2103.00020. arXiv: 2103.00020

    Pith/arXiv arXiv 2021

  25. [26]

    Towards truly zero-shot compositional visual reasoning with llms as programmers

    Aleksandar Stani \'c , Sergi Caelles, and Michael Tschannen. Towards truly zero-shot compositional visual reasoning with llms as programmers. Trans. Mach. Learn. Res., 2024, 2024. URL https://api.semanticscholar.org/CorpusID:266755924

    work page 2024

  26. [27]

    Vipergpt: Visual inference via python execution for reasoning

    D \' dac Sur \' s, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11888--11898, 2023

    work page 2023

  27. [28]

    Gemini: a family of highly capable multimodal models

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023

    Pith/arXiv arXiv 2023

  28. [29]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530, 2024

    Pith/arXiv arXiv 2024

  29. [30]

    Eyes wide shut? exploring the visual shortcomings of multimodal llms

    Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9568--9578, 2024

    work page 2024

  30. [31]

    Rethinking the bounds of llm reasoning: Are multi-agent discussions the key?, 2024 a

    Qineng Wang, Zihao Wang, Ying Su, Hanghang Tong, and Yangqiu Song. Rethinking the bounds of llm reasoning: Are multi-agent discussions the key?, 2024 a . URL https://arxiv.org/abs/2402.18272

    Pith/arXiv arXiv 2024

  31. [32]

    Cogvlm: Visual expert for pretrained language models, 2023

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvlm: Visual expert for pretrained language models, 2023

    2023

  32. [33]

    Lvbench: An extreme long video understanding benchmark

    Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035, 2024 b

    Pith/arXiv arXiv 2024

  33. [34]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pages 58--76. Springer, 2024 c

    work page 2024

  34. [35]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos. arXiv preprint arXiv:2405.19209, 2024 d

    Pith/arXiv arXiv 2024

  35. [36]

    Minicpm-v: A gpt-4v level mllm on your phone

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024

    Pith/arXiv arXiv 2024

  36. [37]

    mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024

    Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-owl3: Towards long image-sequence understanding in multi-modal large language models, 2024. URL https://arxiv.org/abs/2408.04840

    Pith/arXiv arXiv 2024

  37. [38]

    Star: Bootstrapping reasoning with reasoning

    Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35: 0 15476--15488, 2022

    2022

  38. [39]

    Generative verifiers: Reward modeling as next-token prediction

    Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408.15240, 2024 a

    Pith/arXiv arXiv 2024

  39. [40]

    Video instruction tuning with synthetic data

    Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024 b

    Pith/arXiv arXiv 2024

  40. [41]

    Language agent tree search unifies reasoning acting and planning in language models

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. arXiv preprint arXiv:2310.04406, 2023

    Pith/arXiv arXiv 2023

This paper was first reviewed by deepseek-v4-flash on August 4, 2026.