Pith. sign in

REVIEW 4 major objections 6 minor 6 cited by

Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reinforcement-trained agent can answer questions about week-long first-person video by thinking, then calling one of three tools per step, reaching 46.0% on a new 300-question benchmark — 7.7 points above Gemini-1.5-Pro.

desk verdict The framework and external results are solid, but the week-long Ego-R1 Bench is likely circular—QA pairs and RAG memory come from the same clip logs—so the headline 46% should be treated with caution until the authors rule out leakage. read the letter →

arxiv 2506.13654 v1 pith:A5DAUK3U submitted 2025-06-16 cs.CV cs.AI

Shulin Tian , Ruiqi Wang , Hongming Guo , Penghao Wu , Yuhao Dong , Xiuying Wang , Jingkang Yang , Hao Zhang
show 2 more authors
Hongyuan Zhu Ziwei Liu
This is my paper
classification cs.CVcs.AI
keywords egocentricvideoquestionansweringchain-of-tool-thoughttoolcallingreinforcementlearningretrieval-augmentedgenerationhierarchicalRAGlong-formunderstanding
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a small, reinforcement-trained agent can answer questions about egocentric video spanning days to a week, where previous methods plateau at a few hours. The agent reasons in a Chain-of-Tool-Thought loop: it thinks, calls one of three specialized tools (hierarchical retrieval, a video-language model, a frame-level vision model), reads the observation, and thinks again until it can answer. To make this work, the authors build 25K synthetic reasoning traces for supervised fine-tuning and 4.4K QA pairs for reinforcement learning, plus a new 300-question benchmark over week-long first-person recordings. On that benchmark the 3B-parameter agent scores 46.0%, beating Gemini-1.5-Pro by 7.7 points and the strongest open baseline by 17.0 points.

What carries the argument

Chain-of-Tool-Thought (CoTT) is the central object: a reasoning trajectory written as a sequence of steps, each a triplet of a thought, a tool call, and the observation the tool returns. Two components carry the argument. First, the hierarchical RAG memory bank: each 30-second clip is summarized into a text log (visual captions plus speech transcripts), and these logs are aggregated bottom-up into 10-minute, hourly, and daily summaries, so retrieval can start at the week or day level and drill down to specific minutes without exhaustive search. Second, the trained orchestration policy: a 3B language model that, after SFT on 25K CoTT traces and GRPO reinforcement learning on 4.4K QA pairs, chooses at each step which tool to call, at what granularity, and whether enough evidence has accumulated to answer. The dynamic per-step tool selection — as opposed to a fixed pipeline — is what lets a small model navigate temporal structure that would be token-prohibitive to feed to a model whole.

What would settle it

For every question in Ego-R1 Bench, extract the human-verified evidence moment and check whether the decisive detail appears in the corresponding 30-second clip summary inside the memory bank; the fraction of questions whose decisive detail is missing gives an upper bound on what the whole retrieval-then-reason chain can ever achieve. A complementary experiment: when calling the visual tools, replace the RAG-chosen time windows with the ground-truth windows and measure the accuracy gain — that gap isolates how much of the 46.0% comes from retrieval quality versus the visual and reasoning components.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that dynamic, tool-augmented chain-of-thought reasoning — an agent that decides at every step which granularity and which perception tool to use — can handle the unique challenges of ultra-long egocentric video, extending reliable question answering from a few hours to a week. Instead of ingesting a week of footage into a context window, the agent projects the video into a text memory (30-second clip captions aggregated into 10-minute, hourly, and daily summaries) and navigates it top-down, then dispatches visual tools to inspect only the time ranges the retrieval singled out. The orchestration policy is a 3B language model cold-started by SFT on Chain-of-Tool-Thought traces and refined by GRPO reinforcement learning with rule-based rewards. On Ego-R1 Bench, the paper's 300-question benchmark drawn from week-long first-person recordings, the agent reaches 46.0% accuracy, surpassing Gemini-1.5-Pro (38.3%) and the strongest open baseline (29.0%), and both training stages are shown to be necessary.

Load-bearing premise

The whole pipeline assumes that the text summaries making up the hierarchical memory preserve every visual or conversational detail a question might hinge on; if a detail is missing from the summaries, keyword search can never find it, and the vision tools can only inspect time ranges the summaries pointed to.

Editorial extensions

If this is right

  • Video question answering over multi-day, week-long egocentric footage becomes tractable: 46.0% on Ego-R1 Bench, ahead of every baseline tested.
  • Orchestration of cheap specialized tools can substitute for raw context scale: a 3B model beats proprietary models many times its size on this task.
  • The approach transfers outside egocentric video: 64.9% on the VideoMME-long subset, second only to Gemini-1.5-Pro among all compared systems.
  • Both stages of training are load-bearing: RL alone drives task accuracy to 0% (while improving tool-call format), SFT alone reaches 34.3%, and the combination reaches 46.0%.
  • Retrieval and visual grounding are complementary: dropping the visual tools and using RAG alone lowers accuracy to 39.7%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The text memory is the ceiling: any question whose decisive evidence is left out of a 30-second clip summary is unanswerable no matter how well the agent reasons. A direct test is to measure how often the human-verified evidence for each Ego-R1 Bench question survives in the summary texts — that recall bounds the framework's achievable accuracy.
  • The recipe generalizes: when context windows hit their cost limit, train the orchestrator instead of enlarging the model. The same CoTT schema applies to other long-horizon multimodal tasks such as embodied memory or lifelong personal assistants.
  • A sharper test of the cross-day reasoning claim would plot accuracy against the time gap between question and evidence; performance staying flat over multi-day gaps would show the agent genuinely binds distant evidence, while a sharp decay would reveal that retrieval, not reasoning, is doing the work.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Ego-R1, a framework for reasoning over ultra-long egocentric videos (days to weeks). It uses a Chain-of-Tool-Thought (CoTT) process in which a small language model (Qwen-2.5-3B-Instruct) is trained via supervised fine-tuning and reinforcement learning (GRPO) to sequentially invoke three tools: a hierarchical RAG over summarized video logs, a video-LLM, and a VLM. The authors introduce a dataset Ego-R1 Data (25K CoTT traces and 4.4K QAs) and a new benchmark Ego-R1 Bench (300 questions over six week-long egocentric videos). They report 46.0% accuracy on Ego-R1 Bench, surpassing Gemini-1.5-Pro (38.3%) and open baselines, and strong results on VideoMME and EgoSchema.

Significance. If validated, the framework is a meaningful step toward practical week-long egocentric video understanding, and the two-stage RL training of a tool-calling orchestrator is a useful design. Strengths include the modular tool design, the public release of code and data, and the inclusion of external benchmarks (VideoMME, EgoSchema) that partly ground the method outside the proposed benchmark. The ablation study (Table 3) clearly shows the contribution of SFT and RL, and Table 4 demonstrates modularity by swapping Video-LLM components. However, the validity of the headline claim rests on the independence and fairness of Ego-R1 Bench, which is currently not established.

major comments (4)
  1. [Section 4.2, Appendix A, Table 2] The Ego-R1 Bench evaluation is not independent of the system's retrieval memory. Section 4.2 and Figure 2 describe the raw QA collection as generated from 'ASR transcripts with video captioning outputs from the 30-second segments'—the same 30-second clip logs that Appendix A uses to build the hierarchical RAG memory bank. Since Ego-R1 Bench is curated from the same hybrid sources (Section 6.1), a question derived from a day-level or clip-level summary can be answered by retrieving that same summary, making the 46.0% score in Table 2 reflect text retrieval rather than video reasoning. The paper never reports a leakage test (e.g., answering the benchmark with only the memory bank and a strong text LLM, or measuring how often the ground-truth evidence appears in the memory). This is load-bearing because the week-long claim rests entirely on Ego-R1 Bench.
  2. [Section B.1; Table 2] The headline comparison with Gemini-1.5-Pro is asymmetric in compute and tool access. At inference, Ego-R1 uses Gemini-1.5-Pro as its Video-LLM and GPT-4o as its VLM (Appendix B.1), while the Gemini-1.5-Pro baseline is limited to 512 uniformly sampled frames. The reported 7.7-point gain may therefore reflect repeated, selected queries to strong proprietary vision models rather than the 3B orchestrator's reasoning or the RL training. The paper should report and foreground the open-tool variant (Table 4, left: 43.7% with LLaVA-Video) and provide a controlled comparison that either gives baselines access to the same tools or restricts Ego-R1's tools to open-weight models.
  3. [Appendix B, Table 5] The data-generation prompt explicitly forces all tools to be used for every question: 'All tools provided are crucial to the solvement of the question. You MUST exploit the usage of all tools before answering the question' and 'Usually, solving a question requires over 5-10 steps of reasoning, and follows a hierarchical calling structure: rag => video_llm => vlm.' This predetermined structure is in tension with the claim of dynamic tool selection trained via RL. The paper should provide tool-usage statistics (e.g., the distribution of tool sequences and the fraction of trajectories that omit at least one tool) and analyze whether the final policy is genuinely adaptive or mostly follows the SFT-imposed order.
  4. [Appendix A] The assumption that the hierarchical memory bank preserves all question-relevant visual and conversational details is never tested. Because retrieval is keyword-based over the summaries, any detail omitted at the 30-second clip summarization step (Gemini-1.5-Pro) or during hierarchical aggregation (GPT-4.1) is unrecoverable. The paper should include a human audit or a recall measurement of the memory bank against ground-truth clip content, and report the fraction of Ego-R1 Bench questions whose evidence is present in the memory, together with a text-only baseline that reads the full memory.
minor comments (6)
  1. [Abstract; Table 2] The terms 'week-long' and '44.3 h average duration' should be reconciled; 44.3 hours is less than two full days of video, and it should be stated explicitly whether this is the total recorded footage per participant spread over a seven-day recording period.
  2. [Section 6.1] The 'additional summary model with a longer context window' used to conclude the reasoning trace is never named or specified; please identify the model and its role in the pipeline.
  3. [Table 2] The asterisk on the EgoLifeQA score (36.0*) is not defined in the caption; please add the footnote explaining the exclusion of training-overlap questions.
  4. [Reference [17]] Fleiss' kappa is usually attributed to Fleiss (1971), not Fleiss and Cohen (1973); please correct the citation.
  5. [Figure 2] The text in Figure 2 is very small and contains garbled spacing; a high-resolution version with clear labels would help readers follow the data-generation pipeline and make it easier to see the provenance of the benchmark questions.
  6. [Section 6.2] The claim of 'using far fewer parameters than most competitors' is misleading because the system invokes proprietary models at inference; please rephrase to refer to the orchestrator's size and clarify that the full system includes closed-weight vision components.

Circularity Check

1 steps flagged · score 6.0 of 10

Ego-R1 Bench answers are generated from the same 30-second Gemini clip logs that form the H-RAG memory bank, so the headline 46% may measure retrieval of question-generating summaries rather than independent video reasoning.

  1. other [Appendix A / Section 3.2 vs. Section 4.2, Figure 2, Section B.1]
    ""Videos are first segmented into 30-second clips, each summarized via a video captioning model and temporally aligned with the ASR results as clip logs." (Sec. 3.2) "During raw QA data collection for Ego-R1 Data and Ego-R1 Bench, we use Gemini 1.5 Pro for video-to-text processing, Gemini 2.5 Pro [20] for raw MCQ pairs generation, and Claude3.5 Sonnet [2] for MCQ pairs post-processing." (Sec. B.1)"

    The H-RAG memory bank and the Ego-R1 Bench questions are both produced from the same 30-second clip logs: Appendix A creates the memory bank by summarizing each clip with Gemini 1.5 Pro, and Section 4.2 generates QA candidates from ASR transcripts plus video-captioning outputs of the same 30-second segments, with Section B.1 applying that identical processing to the bench as well. A question authored from a day-level or clip-level log can therefore be answered by retrieving that very summary through H-RAG; the 46.0% Ego-R1 Bench score may reflect textual keyword retrieval of question-generating summaries rather than visual grounding or temporal reasoning over video.

full rationale

The central claim of reliable week-long egocentric video reasoning rests primarily on the new Ego-R1 Bench result (46.0% vs. 38.3% for Gemini-1.5-Pro in Table 2). That benchmark is not independent of the system's retrieval memory. Appendix A constructs the H-RAG memory by summarizing every 30-second clip with Gemini 1.5 Pro (visual captions plus ASR), and Section B.1 shows that the same Gemini 1.5 Pro video-to-text processing is used for raw QA collection for both Ego-R1 Data and Ego-R1 Bench, with Gemini 2.5 Pro generating the raw MCQs. Section 4.2 confirms that candidate questions are generated from 'ASR transcripts with video captioning outputs from the 30-second segments' aggregated across temporal granularities. Thus a question written from a day-level summary can be answered by H-RAG retrieving that exact summary; the benchmark does not establish that any correct answer requires visual evidence beyond the text logs, and no text-only baseline reading the full memory bank is reported. The paper also never states that the 300 Ego-R1 Bench questions are disjoint from the 4.4K Ego-QA training pairs, further weakening the benchmark as an independent test. External results on VideoMME and EgoSchema provide independent support that the framework generalizes, and the EgoLifeQA evaluation excludes training overlap, so the paper is not wholly circular; nevertheless, the specific week-long claim is scored against a benchmark constructed from the same summaries the agent searches, making the headline result partially circular by construction.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claim rests on domain assumptions about lossless textual summarization and keyword retrieval, plus assumptions about the fairness of the new benchmark. No new physical or mathematical entities are introduced.

free parameters (5)
  • Clip duration for RAG memory = 30 seconds
    Chosen in Section 3.2 and Appendix A as the base segmentation unit; affects how much information is compressed into each summary and how fine-grained retrieval can be.
  • Temporal hierarchy levels = week, day, hour, 10-minute
    Hand-designed granularity in Section 3.2 and Appendix B.3; assumes question timestamps align with these levels and that summaries at each level preserve needed detail.
  • Maximum reasoning steps N = not reported
    RL rollouts terminate at this cap (Section 5.2); it controls how many tool calls are allowed and therefore the likelihood of finding distant evidence, but no value is given.
  • SFT training epochs = 3
    Section 6.3 mentions "limited epochs (e.g., 3)"; no epoch sweep is reported, so the choice is an author-set hyperparameter rather than a fitted data constant.
  • GRPO hyperparameters (group size, clip epsilon, KL beta) = not reported
    The GRPO objective in Section 5.2 includes G, epsilon, and beta, but their values are not stated in the preprint.
assumptions (4)
  • domain assumption Gemini-1.5-Pro clip summaries preserve all question-relevant visual and dialogue details.
    The retrieval chain starts entirely from these summaries (Section 3.2, Appendix A); any omitted visual or conversational detail is invisible to the agent unless a later video-LLM or VLM call happens to inspect the right time range.
  • domain assumption Keyword search over textual summaries is a sufficient temporal localization signal.
    Hierarchical RAG retrieval is implemented as LLM-based keyword matching on text summaries (Appendix A); visual events that are not textualized cannot be found by the RAG tool.
  • domain assumption GPT-4.1-generated CoTT traces are valid supervision for tool-calling behavior.
    Ego-CoTT-25K is generated by a proprietary LLM and used directly for SFT without human verification of trace quality (Section 4.2); the trained agent inherits any systematic errors in those traces.
  • ad hoc to paper Ego-R1 Bench is a fair, unbiased measure of ultra-long egocentric video reasoning.
    The paper claims human verification but does not demonstrate that the 300 benchmark questions are disjoint from the training QA pairs (Section 5.2); this assumption is required for the headline benchmark result to be valid.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning." pith.science (2026). https://pith.science/paper/A5DAUK3U

@misc{pith2026250613654,
  author       = {Pith},
  title        = {Pith review of: Ego-R1: Chain-of-Tool-Thought for Ultra-Long Egocentric Video Reasoning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A5DAUK3U}},
  note         = {Machine review of arXiv:2506.13654}
}
read the original abstract

We introduce Ego-R1, a novel framework for reasoning over ultra-long (i.e., in days and weeks) egocentric videos, which leverages a structured Chain-of-Tool-Thought (CoTT) process, orchestrated by an Ego-R1 Agent trained via reinforcement learning (RL). Inspired by human problem-solving strategies, CoTT decomposes complex reasoning into modular steps, with the RL agent invoking specific tools, one per step, to iteratively and collaboratively answer sub-questions tackling such tasks as temporal retrieval and multi-modal understanding. We design a two-stage training paradigm involving supervised finetuning (SFT) of a pretrained language model using CoTT data and RL to enable our agent to dynamically propose step-by-step tools for long-range reasoning. To facilitate training, we construct a dataset called Ego-R1 Data, which consists of Ego-CoTT-25K for SFT and Ego-QA-4.4K for RL. Furthermore, our Ego-R1 agent is evaluated on a newly curated week-long video QA benchmark, Ego-R1 Bench, which contains human-verified QA pairs from hybrid sources. Extensive results demonstrate that the dynamic, tool-augmented chain-of-thought reasoning by our Ego-R1 Agent can effectively tackle the unique challenges of understanding ultra-long egocentric videos, significantly extending the time coverage from few hours to a week.

Figures

Figures reproduced from arXiv: 2506.13654 by the authors.

Figure 1
Figure 1. Overview of Ego-R1. In this figure, we demonstrate how the Ego-R1 Agent orchestrates specialized tools (e.g., Hierarchical_RAG, Video LLM, and VLM) to answer the question step-by-step, based on the observations and previous actions. The system effectively answers questions that require careful searching within ultra-long videos and precise analysis of frame details. Abstract We introduce Ego-R1, a novel framework fo… view at source ↗
Figure 2
Figure 2. Data generation pipeline of the Ego-R1 Data. We first obtained raw QA pairs from both AI-generated and human-annotated sources based on 6 raw videos collected from 6 participants and the corresponding log. The verified and processed Multiple Choice Questions (MCQs) serve as the foundation of the Ego-R1 Data (left). We take questions without answers for Chain-of-Tool-Thought (CoTT) generation, which involves creating… view at source ↗
Figure 3
Figure 3. Overview of the two-stage training strategies in Ego-R1. Ego-R1 employs a two-stage training approach: Stage 1 utilizes supervised fine-tuning with CoTT data to establish structured tool-calling capabilities, while Stage 2 applies multi-turn reinforcement learning with rule-based rewards to optimize iterative reasoning and tool execution across diverse question types. formatted tool calls as prescribed by the CoTT r… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Overview of the Hierarchical RAG system. Based on the raw video and its 30-second clips, we generate the memory bank for each video from its 30-second-level summaries to day-level summaries. During the keywords retrieval, the system searches efficiently by starting wit…
Figure 5
Figure 5. Figure 5: Qualitative results comparison with Video-R1. Case 1-3 illustrate successful examples where Ego-R1 outperforms Video-R1 by producing more detailed, interpretable step-by-step reasoning chains through dynamic tool-calling. In contrast, Case 4 highlights a failure case f…

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. EgoVITA: Learning to Plan and Verify for Egocentric Video Reasoning

    cs.CV 2025-11 conditional novelty 7.0 of 10

    EgoVITA, a GRPO-based plan-then-verify framework with dense visual-grounding rewards, improves egocentric video reasoning by up to +7.7 points and keeps exocentric video performance intact.

  2. Light-Omni: Reflex over Reasoning in Agentic Video Understanding with Long-Term Memory

    cs.CV 2026-07 conditional novelty 6.5 of 10

    Dual global+latent states with hierarchical episodic merging enable reflexive, low-latency long-video agents that beat iterative reasoning baselines on accuracy and efficiency.

  3. Video-DeepResearch: Towards the Next-Generation Multimodal Deepresearch Agent

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Video-DeepResearch trains video agents with a two-stage SFT plus GRPO pipeline to do visual web searches before text searches, and introduces a 200-question benchmark requiring both video grounding and open-web knowledge.

  4. Multimodal Reinforcement Learning with Adaptive Verifier for AI Agents

    cs.AI 2025-12 unverdicted novelty 6.0 of 10

    Argos is an agentic verifier that adaptively picks scoring functions to evaluate accuracy, localization, and reasoning quality, enabling stronger multimodal RL training for AI agents.

  5. LightMem-Ego: Your AI Memory for Everyday Life

    cs.CL 2026-07 conditional novelty 4.0 of 10

    A streaming hierarchical multimodal memory system captures egocentric video/audio, routes queries across current/short-term/long-term stores, and demos everyday recall on phones and AI glasses.

  6. Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle

    cs.CL 2025-09 conditional novelty 3.0 of 10

    A survey that maps reinforcement learning methods, datasets, benchmarks, and open-source tools across the full training lifecycle of large language models, focusing on verifiable-reward reasoning.

Reference graph

Works this paper leans on

86 extracted references · 23 canonical work pages · cited by 6 Pith papers

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    Claude 3.5 sonnet

    Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/news/ claude-3-5-sonnet/

  3. [3]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  4. [4]

    Hourvideo: 1-hour video- language understanding.Advances in Neural Information Processing Systems, 37:53168–53197, 2024

    Keshigeyan Chandrasegaran, Agrim Gupta, Lea M Hadzic, Taran Kota, Jimming He, Cristóbal Eyzaguirre, Zane Durante, Manling Li, Jiajun Wu, and Fei-Fei Li. Hourvideo: 1-hour video- language understanding.Advances in Neural Information Processing Systems, 37:53168–53197, 2024

    2024

  5. [5]

    Grounded multi-hop videoqa in long-form egocentric videos, 2024

    Qirui Chen, Shangzhe Di, and Weidi Xie. Grounded multi-hop videoqa in long-form egocentric videos, 2024

    2024

  6. [6]

    Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models.CoRR, 2023

    Yi Chen, Yuying Ge, Yixiao Ge, Mingyu Ding, Bohao Li, Rui Wang, Ruifeng Xu, Ying Shan, and Xihui Liu. Egoplan-bench: Benchmarking egocentric embodied planning with multimodal large language models.CoRR, 2023

    2023

  7. [7]

    Longvila: Scaling long-context visual language models for long videos.arXiv preprint arXiv:2408.10188, 2024

    Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos.arXiv preprint arXiv:2408.10188, 2024

    arXiv 2024

  8. [8]

    Egothink: Evaluating first-person perspective thinking capability of vision-language models

    Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. Egothink: Evaluating first-person perspective thinking capability of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14291–14302, 2024

    2024

Show all 86 references
  1. [9]

    Le, Sergey Levine, and Yi Ma

    Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V . Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training, 2025

    2025

  2. [10]

    Scaling egocentric vision: The epic-kitchens dataset

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Scaling egocentric vision: The epic-kitchens dataset. InProceedings of the European conference on compute...

    2018

  3. [11]

    Open- vlthinker: An early exploration to complex vision-language reasoning via iterative self- improvement.arXiv preprint arXiv:2503.17352, 2025

    Yihe Deng, Hritik Bansal, Fan Yin, Nanyun Peng, Wei Wang, and Kai-Wei Chang. Open- vlthinker: An early exploration to complex vision-language reasoning via iterative self- improvement.arXiv preprint arXiv:2503.17352, 2025

    2025 arXiv

  4. [12]

    Grounded question-answering in long egocentric videos

    Shangzhe Di and Weidi Xie. Grounded question-answering in long egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12934–12943, 2024

    2024

  5. [13]

    Insight-v: Exploring long-chain visual reasoning with multimodal large language models.arXiv preprint arXiv:2411.14432, 2024

    Yuhao Dong, Zuyan Liu, Hai-Long Sun, Jingkang Yang, Winston Hu, Yongming Rao, and Ziwei Liu. Insight-v: Exploring long-chain visual reasoning with multimodal large language models.arXiv preprint arXiv:2411.14432, 2024

    2024 arXiv

  6. [14]

    Videoagent: A memory-augmented multimodal agent for video understanding

    Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. InEuropean Conference on Computer Vision, pages 75–92. Springer, 2025

    2025

  7. [15]

    Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025

    Jiazhan Feng, Shijue Huang, Xingwei Qu, Ge Zhang, Yujia Qin, Baoquan Zhong, Chengquan Jiang, Jinxin Chi, and Wanjun Zhong. Retool: Reinforcement learning for strategic tool use in llms.arXiv preprint arXiv:2504.11536, 2025. 11

    2025 arXiv

  8. [16]

    Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms.arXiv preprint arXiv:2503.21776, 2025

    2025 arXiv

  9. [17]

    The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.Educational and psychological measurement, 33(3):613–619, 1973

    Joseph L Fleiss and Jacob Cohen. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.Educational and psychological measurement, 33(3):613–619, 1973

    1973

  10. [18]

    Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis.arXiv preprint arXiv:2405.21075, 2024

    2024 arXiv

  11. [19]

    Amego: Active memory from long egocentric videos

    Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. Amego: Active memory from long egocentric videos. InEuropean Conference on Computer Vision, pages 92–110. Springer, 2024

    2024

  12. [20]

    Gemini 2.5: Our most intelligent ai model, 2025

    Google DeepMind. Gemini 2.5: Our most intelligent ai model, 2025

    2025

  13. [21]

    Tora: A tool-integrated reasoning agent for mathematical problem solving

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving. arXiv preprint arXiv:2309.17452, 2023

    2023 arXiv

  14. [22]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF conference on computer vision a...

    2022

  15. [23]

    Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives

    Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyl- los Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, et al. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. InProceeding...

    2024

  16. [24]

    Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-r1: Incentivizing reasoning capability in multimodal large language models.arXiv preprint arXiv:2503.06749, 2025

    2025 arXiv

  17. [25]

    Token-efficient long video understanding for multimodal llms, 2025

    Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, Sungjin Ahn, Jan Kautz, Hongxu Yin, Yao Lu, Song Han, and Wonmin Byeon. Token-efficient long video understanding for multimodal llms, 2025

    2025

  18. [26]

    Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    Bowen Jin, Hansi Zeng, Zhenrui Yue, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning.arXiv preprint arXiv:2503.09516, 2025

    2025 arXiv

  19. [27]

    Dwim: Towards tool-aware visual rea- soning via discrepancy-aware workflow generation & instruct-masking tuning.arXiv preprint arXiv:2503.19263, 2025

    Fucai Ke, Xingjian Leng, Zhixi Cai, Zaid Khan, Weiqing Wang, Pari Delir Haghighi, Hamid Rezatofighi, Manmohan Chandraker, et al. Dwim: Towards tool-aware visual rea- soning via discrepancy-aware workflow generation & instruct-masking tuning.arXiv preprint arXiv:2503.19263, 2025

    2025

  20. [28]

    Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer.arXiv preprint arXiv:2408.03326, 2024

    2024 arXiv

  21. [29]

    Videochat: Chat-centric video understanding, 2024

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024

    2024

  22. [30]

    Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    Xuefeng Li, Haoyang Zou, and Pengfei Liu. Torl: Scaling tool-integrated rl.arXiv preprint arXiv:2503.23383, 2025

    2025 arXiv

  23. [31]

    Egotom: Benchmarking theory of mind reasoning from egocentric videos.arXiv preprint arXiv:2503.22152, 2025

    Yuxuan Li, Vijay Veerabadran, Michael L Iuzzolino, Brett D Roads, Asli Celikyilmaz, and Karl Ridgeway. Egotom: Benchmarking theory of mind reasoning from egocentric videos.arXiv preprint arXiv:2503.22152, 2025. 12

    2025 arXiv

  24. [32]

    Mario: Math reasoning with code interpreter output–a reproducible pipeline.arXiv preprint arXiv:2401.08190, 2024

    Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output–a reproducible pipeline.arXiv preprint arXiv:2401.08190, 2024

    2024 arXiv

  25. [33]

    Coarse correspondence elicit 3d spacetime understanding in multimodal language model.arXiv preprint arXiv:2408.00754, 2024

    Benlin Liu, Yuhao Dong, Yiqin Wang, Yongming Rao, Yansong Tang, Wei-Chiu Ma, and Ranjay Krishna. Coarse correspondence elicit 3d spacetime understanding in multimodal language model.arXiv preprint arXiv:2408.00754, 2024

    2024 arXiv

  26. [34]

    Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx mllm: On- demand spatial-temporal understanding at arbitrary resolution.arXiv preprint arXiv:2409.12961, 2024

    2024 arXiv

  27. [35]

    Chain-of-spot: Interactive reasoning improves large vision-language models.arXiv preprint arXiv:2403.12966, 2024

    Zuyan Liu, Yuhao Dong, Yongming Rao, Jie Zhou, and Jiwen Lu. Chain-of-spot: Interactive reasoning improves large vision-language models.arXiv preprint arXiv:2403.12966, 2024

    2024 arXiv

  28. [36]

    Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment

    Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers of omni-modal language model with progressive modality alignment. arXiv preprint arXiv:2502.04328, 2025

    2025 arXiv

  29. [37]

    Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension.arXiv preprint arXiv:2411.13093, 2024

    2024

  30. [38]

    Taco: Learning multi-modal action models with synthetic chains-of-thought-and-action, 2024

    Zixian Ma, Jianguo Zhang, Zhiwei Liu, Jieyu Zhang, Juntao Tan, Manli Shu, Juan Carlos Niebles, Shelby Heinecke, Huan Wang, Caiming Xiong, Ranjay Krishna, and Silvio Savarese. Taco: Learning multi-modal action models with synthetic chains-of-thought-and-action, 2024

    2024

  31. [39]

    Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models, 2024

    2024

  32. [40]

    Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding.Advances in Neural Information Processing Systems, 36:46212–46244, 2023

    2023

  33. [41]

    Hello gpt-4o, 2024

    OpenAI. Hello gpt-4o, 2024

    2024

  34. [42]

    Introducing gpt-4.1 in the api, 2025

    OpenAI. Introducing gpt-4.1 in the api, 2025

    2025

  35. [43]

    Openai o3 and o4-mini system card, 2025

    OpenAI. Openai o3 and o4-mini system card, 2025

    2025

  36. [44]

    Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022

    Aaron Parisi, Yao Zhao, and Noah Fiedel. Talm: Tool augmented language models.arXiv preprint arXiv:2205.12255, 2022

    2022 arXiv

  37. [45]

    Hd-epic: A highly-detailed egocentric video dataset.arXiv preprint arXiv:2502.04144, 2025

    Toby Perrett, Ahmad Darkhalil, Saptarshi Sinha, Omar Emara, Sam Pollard, Kranti Parida, Kaiting Liu, Prajwal Gatti, Siddhant Bansal, Kevin Flanagan, et al. Hd-epic: A highly-detailed egocentric video dataset.arXiv preprint arXiv:2502.04144, 2025

    2025 arXiv

  38. [46]

    Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    Cheng Qian, Emre Can Acikgoz, Qi He, Hongru Wang, Xiusi Chen, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. Toolrl: Reward is all tool learning needs.arXiv preprint arXiv:2504.13958, 2025

    2025 arXiv

  39. [47]

    Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. Toolllm: Facilitating large language models to master 16000+ real-world apis.arXiv preprint arXiv:2307.16789, 2023

    2023 arXiv

  40. [48]

    Does your vision-language model get lost in the long video sampling dilemma?, 2025

    Tianyuan Qu, Longxiang Tang, Bohao Peng, Senqiao Yang, Bei Yu, and Jiaya Jia. Does your vision-language model get lost in the long video sampling dilemma?, 2025

    2025

  41. [49]

    Action scene graphs for long-form understanding of egocentric videos

    Ivan Rodin, Antonino Furnari, Kyle Min, Subarna Tripathi, and Giovanni Maria Farinella. Action scene graphs for long-form understanding of egocentric videos. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18622–18632, 2024

    2024

  42. [50]

    Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36:68539– 68551, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools.Advances in Neural Information Processing Systems, 36:68539– 68551, 2023. 13

    2023

  43. [51]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.arXiv preprint arXiv:2402.03300, 2024

    2024 arXiv

  44. [52]

    Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra

    Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, Zhuang Liu, Hu Xu, Hyunwoo J. Kim, Bilge Soran, Raghuraman Krishnamoorthi, Mohamed Elhoseiny, and Vikas Chandra. Longvu: Spatiote...

    2024

  45. [53]

    Video-xl: Extra-long vision language model for hour-scale video understanding, 2024

    Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding, 2024

    2024

  46. [54]

    Moviechat: From dense token to sparse memory for long video understanding

    Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding. InProceedings of the IEEE/CVF Conf...

    2024

  47. [55]

    Ego4d goal-step: Toward hierarchical understanding of procedural activities.Advances in Neural Information Processing Systems, 36:38863–38886, 2023

    Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torre- sani. Ego4d goal-step: Toward hierarchical understanding of procedural activities.Advances in Neural Information Processing Systems, 36:38863–38886, 2023

    2023

  48. [56]

    Openthinkimg: Learning to think with images via visual tool reinforcement learning, 2025

    Zhaochen Su, Linjie Li, Mingyang Song, Yunzhuo Hao, Zhengyuan Yang, Jun Zhang, Guanjie Chen, Jiawei Gu, Juntao Li, Xiaoye Qu, and Yu Cheng. Openthinkimg: Learning to think with images via visual tool reinforcement learning, 2025

    2025

  49. [57]

    Egotracks: A long-term egocentric visual object tracking dataset.Advances in Neural Information Processing Systems, 36:75716–75739, 2023

    Hao Tang, Kevin J Liang, Kristen Grauman, Matt Feiszli, and Weiyao Wang. Egotracks: A long-term egocentric visual object tracking dataset.Advances in Neural Information Processing Systems, 36:75716–75739, 2023

    2023

  50. [58]

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.arXiv preprint arXiv:2403.05530, 2024

    2024 arXiv

  51. [59]

    Llamav-o1: Rethinking step-by-step visual reasoning in llms.arXiv preprint arXiv:2501.06186, 2025

    Omkar Thawakar, Dinura Dissanayake, Ketan More, Ritesh Thawkar, Ahmed Heakl, Noor Ahsan, Yuhao Li, Mohammed Zumri, Jean Lahoud, Rao Muhammad Anwer, et al. Llamav-o1: Rethinking step-by-step visual reasoning in llms.arXiv preprint arXiv:2501.06186, 2025

    2025 arXiv

  52. [60]

    Otc: Optimal tool calls via reinforcement learning

    Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. Otc: Optimal tool calls via reinforcement learning. arXiv preprint arXiv:2504.14870, 2025

    2025 arXiv

  53. [61]

    Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning.arXiv preprint arXiv:2310.03731, 2023

    Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in llms for enhanced mathematical reasoning.arXiv preprint arXiv:2310.03731, 2023

    arXiv

  54. [62]

    Resanything: Attribute prompting for arbitrary referring segmen- tation.arXiv preprint arXiv:2505.02867, 2025

    Ruiqi Wang and Hao Zhang. Resanything: Attribute prompting for arbitrary referring segmen- tation.arXiv preprint arXiv:2505.02867, 2025

    2025 arXiv

  55. [63]

    Videoagent: Long-form video understanding with large language model as agent

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. InEuropean Conference on Computer Vision, pages 58–76. Springer, 2024

    2024

  56. [64]

    Internvideo2

    Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Internvideo2. 5: Empowering video mllms with long and rich context modeling.arXiv preprint arXiv:2501.12386, 2025

    2025 arXiv

  57. [65]

    Lifelongmemory: Leveraging llms for answering queries in long-form egocentric videos.arXiv preprint arXiv:2312.05269, 2023

    Ying Wang, Yanlai Yang, and Mengye Ren. Lifelongmemory: Leveraging llms for answering queries in long-form egocentric videos.arXiv preprint arXiv:2312.05269, 2023. 14

    2023 arXiv

  58. [66]

    Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209, 2024

    Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for llm reasoning on long videos.arXiv preprint arXiv:2405.19209, 2024

    2024 arXiv

  59. [67]

    Chain-of-thought prompting elicits reasoning in large language models

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  60. [68]

    Longvlm: Efficient long video understanding via large language models, 2024

    Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models, 2024

    2024

  61. [69]

    Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023

    2023

  62. [70]

    Grounded chain-of-thought for multimodal large language models.arXiv preprint arXiv:2503.12799, 2025

    Qiong Wu, Xiangcong Yang, Yiyi Zhou, Chenxin Fang, Baiyang Song, Xiaoshuai Sun, and Rongrong Ji. Grounded chain-of-thought for multimodal large language models.arXiv preprint arXiv:2503.12799, 2025

    2025 arXiv

  63. [71]

    Llava-o1: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    Guowei Xu, Peng Jin, Li Hao, Yibing Song, Lichao Sun, and Li Yuan. Llava-o1: Let vision language models reason step-by-step.arXiv preprint arXiv:2411.10440, 2024

    2024 arXiv

  64. [72]

    Retrieval-augmented egocentric video captioning

    Jilan Xu, Yifei Huang, Junlin Hou, Guo Chen, Yuejie Zhang, Rui Feng, and Weidi Xie. Retrieval-augmented egocentric video captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13525–13536, 2024

    2024

  65. [73]

    An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement.arXiv preprint arXiv:2409.12122, 2024

    2024 arXiv

  66. [74]

    Octopus: Embodied vision-language programmer from environmental feedback

    Jingkang Yang, Yuhao Dong, Shuai Liu, Bo Li, Ziyue Wang, Haoran Tan, Chencheng Jiang, Jiamu Kang, Yuanhan Zhang, Kaiyang Zhou, et al. Octopus: Embodied vision-language programmer from environmental feedback. InEuropean Conference on Computer Vision, pages 20–38. Springer, 2024

    2024

  67. [75]

    Egolife: Towards egocentric life assistant, 2025

    Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Ya...

    2025

  68. [76]

    R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization.arXiv preprint arXiv:2503.10615, 2025

    Yi Yang, Xiaoxuan He, Hongkun Pan, Xiyan Jiang, Yan Deng, Xingtao Yang, Haoyu Lu, Dacheng Yin, Fengyun Rao, Minfeng Zhu, et al. R1-onevision: Advancing generalized multimodal reasoning through cross-modal formalization.arXiv preprint arXiv:2503.10615, 2025

    2025 arXiv

  69. [77]

    Fans–formal answer selection for natural language math reasoning using lean4.arXiv preprint arXiv:2503.03238, 2025

    Jiarui Yao, Ruida Wang, and Tong Zhang. Fans–formal answer selection for natural language math reasoning using lean4.arXiv preprint arXiv:2503.03238, 2025

    2025

  70. [78]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  71. [79]

    Re-thinking temporal search for long-form video understanding, 2025

    Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, Jiajun Wu, and Manling Li. Re-thinking temporal search for long-form video understanding, 2025

    2025

  72. [80]

    R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vl: Learning to reason with multimodal large language models via step-wise group relative policy optimization.arXiv preprint arXiv:2503.12937, 2025

    2025 arXiv

  73. [81]

    Long context transfer from language to vision

    Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406.16852, 2024. 15

    2024 arXiv

  74. [82]

    Llava-next: A strong zero-shot video understanding model, April 2024

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024

    2024

  75. [83]

    Deepresearcher: Scaling deep research via reinforcement learning in real-world environments.arXiv preprint arXiv:2504.03160, 2025

    Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments.arXiv preprint arXiv:2504.03160, 2025

    2025 arXiv

  76. [84]

    Videoagent2: Enhancing the llm-based agent system for long-form video understanding by uncertainty-aware cot, 2025

    Zhuo Zhi, Qiangqiang Wu, Minghe shen, Wenbo Li, Yinchuan Li, Kun Shao, and Kaiwen Zhou. Videoagent2: Enhancing the llm-based agent system for long-form video understanding by uncertainty-aware cot, 2025

    2025

  77. [85]

    day”levelbackwardfromDAY3_16391802,usingthekeyword“supermarket

    Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification.arXiv preprint arXiv:2308.07921, 2023. 16 Appendix T...

    2023 arXiv

  78. [86]

    name " :

    All tools provided are crucial to the solvement of the question. You MUST exploit the usage of all tools before answering the question. 2. You may want to use the same tool multiple times with different arguments to explore the problem from different angles, if needed. 3. Make...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.