Pith. sign in

REVIEW 9 cited by

AntGPT: Can Large Language Models Help Long-term Action Anticipation from Videos?

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2307.16368 v3 pith:GYHHUY2R submitted 2023-07-31 cs.CV

Qi Zhao , Shijie Wang , Ce Zhang , Changcheng Fu , Minh Quan Do , Nakul Agarwal , Kwonjoon Lee , Chen Sun This is my paper · ORCID
classification cs.CV
keywords goalactionsactorantgptprocedureactionapproachfuture
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Can we better anticipate an actor's future actions (e.g. mix eggs) by knowing what commonly happens after his/her current action (e.g. crack eggs)? What if we also know the longer-term goal of the actor (e.g. making egg fried rice)? The long-term action anticipation (LTA) task aims to predict an actor's future behavior from video observations in the form of verb and noun sequences, and it is crucial for human-machine interaction. We propose to formulate the LTA task from two perspectives: a bottom-up approach that predicts the next actions autoregressively by modeling temporal dynamics; and a top-down approach that infers the goal of the actor and plans the needed procedure to accomplish the goal. We hypothesize that large language models (LLMs), which have been pretrained on procedure text data (e.g. recipes, how-tos), have the potential to help LTA from both perspectives. It can help provide the prior knowledge on the possible next actions, and infer the goal given the observed part of a procedure, respectively. To leverage the LLMs, we propose a two-stage framework, AntGPT. It first recognizes the actions already performed in the observed videos and then asks an LLM to predict the future actions via conditioned generation, or to infer the goal and plan the whole procedure by chain-of-thought prompting. Empirical results on the Ego4D LTA v1 and v2 benchmarks, EPIC-Kitchens-55, as well as EGTEA GAZE+ demonstrate the effectiveness of our proposed approach. AntGPT achieves state-of-the-art performance on all above benchmarks, and can successfully infer the goal and thus perform goal-conditioned "counterfactual" prediction via qualitative analysis. Code and model will be released at https://brown-palm.github.io/AntGPT

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  2. Enhancing Visual Planning with Auxiliary Tasks and Multi-token Prediction

    cs.CV 2025-07 conditional novelty 6.0 of 10

    An MLLM trained with auxiliary goal-prediction tasks and multi-token prediction achieves SOTA on COIN and CrossTask visual planning and matches SOTA on Ego4D LTA.

  3. VisuLogic: A Benchmark for Evaluating Visual Reasoning in Multi-modal Large Language Models

    cs.CV 2025-04 conditional novelty 6.0 of 10

    On VisuLogic's 1,000 vision-centric puzzles, the best multimodal models reach 28.1% accuracy versus a 24.9% random baseline and 51.4% human accuracy, and an RL baseline lifts accuracy by up to 5.6 points.

  4. Vision and Intention Boost Large Language Model in Long-Term Action Anticipation

    cs.CV 2025-05 conditional novelty 5.0 of 10

    On Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+, intention-conditioned visual features plus multimodal in-context examples beat published long-term action anticipation models.

  5. ClassComet: Exploring and Designing AI-generated Danmaku in Educational Videos to Enhance Online Learning

    cs.HC 2025-04 conditional novelty 5.0 of 10

    AI-written danmaku, combining content and emotion types, can match human comment quality and significantly boost learner engagement and quiz gains in short educational videos.

  6. Technical Report for Ego4D Long-Term Action Anticipation Challenge 2025

    cs.CV 2025-06 conditional novelty 4.0 of 10

    A three-stage pipeline using the EgoVideo-V encoder, a verb-noun co-occurrence reranker, SAM2 hand-object features, and a fine-tuned Llama 2 model took first place in the Ego4D 2025 long-term action anticipation challenge.

  7. Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models

    cs.AI 2025-04 conditional novelty 4.0 of 10

    A structured survey of reinforcement-learning-based reasoning methods for multimodal large language models, including a taxonomy, reward design review, benchmark tables, and open challenges.

  8. Visual Large Language Models for Generalized and Specialized Applications

    cs.CV 2025-01 conditional novelty 3.0 of 10

    This paper reviews and taxonomizes VLLM applications into vision-to-text, vision-to-action, and text-to-vision, adding ethics and future-work discussion.

  9. Empowering Multimodal LLMs with External Tools: A Comprehensive Survey

    cs.CV 2025-08 unverdicted novelty 2.0 of 10

    A survey paper maps how external tools are used to augment multimodal large language models across data, tasks, evaluation, and future directions.

Pith tools