Pith. sign in

REVIEW 6 major objections 5 minor 1 cited by

Vision and Intention Boost Large Language Model in Long-Term Action Anticipation

T0 review · 6 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that inferring a person's behavioral intention from observed video and fusing that intention into visual embeddings through cross-attention lets an LLM predict long-term future actions more accurately than either…

desk verdict A useful, well-ablated recipe for LLM-based action anticipation, held back by an unvalidated intention module and reproducibility gaps; deserves review with major revision. read the letter →

arxiv 2505.01713 v1 pith:WSUGO54K submitted 2025-05-03 cs.CV

Congqi Cao , Lanshu Hu , Yating Yu , Yanning Zhang This is my paper · ORCID
classification cs.CV
keywords long-termactionanticipationbehavioralintentioninferencevision-languagemodelscross-attentionfusionin-contextlearningegocentricvideolargelanguageEgo4D
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that long-term action anticipation fails when a model sees only pixels or only text: visual features are redundant, and text substitutes such as action labels or captions lose detail. It proposes ICVL, which asks a vision-language model to state in words what the person in the video wants to do, then uses that intention sentence as the query in cross-attention over the video's visual embeddings to produce intention-enhanced visual representations. Those representations, together with a textual prompt and selected in-context examples, are fed into a LoRA-tuned LLM that generates the future action sequence. The paper reports state-of-the-art results on Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+ under the respective metrics, with ablations attributing the main gain to the intention-conditioned fusion.

What carries the argument

The load-bearing mechanism is the Intention-Context Attention Fusion (ICAF) module: a cross-attention layer in which the text embedding of the VLM-inferred intention is the query and the visual embeddings of the observed video are the keys and values. This turns a free-form sentence answering 'What does the person want to do?' into a reweighting of every visual token before the LLM reads them. The second mechanism is multi-modality example selection: training clips are ranked by a weighted sum of normalized L2 similarity in visual space and in observed-label text space, and the top-k clips are inserted into the prompt as in-context demonstrations. Both feed a LLaMA-3-8B LLM fine-tuned with LoRA, with frozen CLIP and BLIP2 encoders supplying the visual features.

What would settle it

Take the trained ICVL model on Ego4D and replace the VLM-inferred intention text with a fixed random or deliberately wrong intention, such as 'do the dishes' for a gardening clip, while keeping visual features and all other inputs identical. If edit distances do not worsen substantially relative to the reported 0.8570 action edit distance, the intention-conditioned fusion is not the source of the gain. A softer test is to have humans rate intention accuracy on a random sample of 100 clips and check whether per-clip prediction error tracks rated intention quality; the paper contains no such evaluation.

Watch

Extended reading notes

Core claim

The central claim is that a high-level behavioral intention inferred directly from observed video, stated as a free-form sentence by a VLM, can act as a cross-modal key that reshapes visual embeddings before they reach an LLM, and that this is what lets the LLM anticipate long-term actions better than text-only or vision-only pipelines. Concretely, with intention embeddings as queries and visual embeddings as keys and values, the attention output reweights the visual sequence toward cues consistent with the inferred goal; the resulting embeddings are passed to the LLM along with a prompt containing observed action labels and the k most similar training examples selected by a weighted visual-textual similarity. The paper claims this yields the lowest edit distances on Ego4D (noun 0.6194, verb 0.6516, action 0.8570) and the highest mean average precision on EPIC-Kitchens-55 (ALL 43.3) and EGTEA GAZE+ (ALL 81.0), while remaining robust to noisy observed action labels (7.97% verb-noun accuracy with a CLIP encoder).

Load-bearing premise

The method assumes that the free-form intention sentence a VLM produces when asked 'What does the person want to do?' on the last observed frame is accurate and stable enough to act as the attention query over all visual features, yet the paper reports no measurement of intention quality; a hallucinated intention would reweight every visual token around a wrong goal and propagate into the LLM's predictions.

Editorial extensions

If this is right

  • If ICVL is right, LLM-based long-term action anticipation does not have to choose between pixels and text: feeding intention-reweighted visual embeddings alongside textual prompts yields lower edit distances on Ego4D than the reported text-only or vision-only baselines.
  • Intention-conditioned fusion should make predictions more robust to noisy observed action labels, since the reported gains over AntGPT hold even though ICVL's action recognizer (CLIP, 7.97% verb-noun accuracy) is far weaker than EgoVideo's encoder.
  • Joint visual-textual example selection should transfer to other LLM-based video tasks that pair a query video with in-context demonstrations, because it only requires a visual encoder, a text encoder, and similarity scores.
  • The ablations predict that cross-attention with the intention as query is better than either concatenation or cross-attention with the visual as query, making the query choice itself part of the mechanism's effectiveness.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not measure intention quality, so the reported gains could in principle come from the extra visual tokens, the extra parameters, or the LoRA tuning rather than from the semantics of the intention sentence; replacing the intention text with a scrambled version of itself (same token count, random order) would separate those explanations.
  • The intention-as-query design suggests a latent-variable reading: the VLM sentence is a discrete proxy for a continuous intention variable, and a natural next step is to train the intention encoder end-to-end with a goal-consistency loss against the future action sequence rather than freezing it.
  • Because the example-selection weight α is a fixed scalar, a testable extension is to learn α per query video or condition it on the confidence of the action recognizer; low-confidence queries would lean more on visual similarity, which the ablation shows is the stronger single modality.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes Intention-Conditioned Vision-Language (ICVL), a framework for long-term action anticipation that: (1) uses a VLM to infer behavioral intentions from observed frames, (2) fuses intention embeddings with visual embeddings via a cross-attention module called ICAF (Eq. 2), and (3) selects in-context examples using a weighted combination of visual and textual similarity (Eq. 7). The model is evaluated on Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+, reporting state-of-the-art edit distances and mean average precision. Ablations show that both ICAF and example selection contribute to the reported improvements.

Significance. If validated, the paper makes a useful contribution by combining LLM reasoning with visual and intention information for long-term action anticipation. The ablations in Tables 3-6 are informative, and the use of a frozen, unfine-tuned VLM with held-out splits reduces internal circularity. However, the central claim that VLM-inferred intention semantics are responsible for the gains is not directly validated: the ICAF module adds learnable capacity, and no control experiment removes the specific semantic content of the intention. The reported improvements over prior SOTA are also small and lack error bars. The paper is a solid empirical study but requires additional controls and clarifications to establish the proposed mechanism.

major comments (6)
  1. [Sec 3.2, Eq. (2)] The paper attributes the improvement from ICAF to the semantic content of the VLM-inferred intention, but this is not isolated. Cross-attention with learnable projections adds capacity, and any text-like or learned query could produce similar gains. There is no evaluation of intention quality, no human inspection, and no control experiment with random or generic queries. Please add a control that removes intention semantics (e.g., using a fixed or random text query, or learnable query tokens) to demonstrate that the specific intention content is what matters.
  2. [Sec 4.4, Tables 3-4] The two 'Baseline' rows have inconsistent values: Table 3 reports 0.6927/0.6823/0.8944 for noun/verb/action, while Table 4 reports 0.6469/0.6661/0.8773. This inconsistency prevents a clean interpretation of the ablations. Please clarify the exact components included in each baseline and use consistent labels across tables.
  3. [Sec 3.3, Eq. (4)] The formulation of the top-k selection objective is incorrect as written: the sum runs over all elements in Ω rather than over the selected set U, so the objective is independent of U and the argmin is ill-defined. Please correct the index set to reflect selection from U.
  4. [Sec 4.4, Figure 4] The text states 'optimal results attained when seven examples are provided' but then says 'We report the final performance based on the use of three examples.' This is contradictory and affects reproducibility. Please clarify the exact number of examples used in the final experiments and whether the difference between three and seven is within noise.
  5. [Sec 4.4, Table 6] The claim that visual similarity beats textual similarity is not supported by the noun column (Text 0.7299 vs Visual 0.7330, lower is better). Additionally, the relation of these rows to the Baseline in Table 3 is unclear; the Text and Visual rows appear worse than Baseline, yet the Fused row is better. Please specify the configuration of Table 6 (e.g., whether ICAF is included) and discuss the per-metric pattern.
  6. [Sec 4.3, Table 1] The improvement over EgoVideo is small (relative improvements of 0.7%, 0.6%, and 0.49% on noun, verb, and action). With no error bars or multiple seeds, it is difficult to assess whether the state-of-the-art claim is statistically significant. Please report variance or significance tests, and ensure comparisons use comparable training conditions.
minor comments (5)
  1. [Sec 3.2] The mechanism by which E_ic is fed into the decoder-only LLM is underspecified. Please state whether E_ic tokens are prepended as prefix tokens and how they are projected to the LLM embedding space.
  2. [Sec 4.2] The role of BLIP2-OPT-2.7B as the 'frozen visual encoder' for ICAF is unclear, given that CLIP ViT-L/14 is described as the visual encoder for action recognition. Please clarify which visual features are used in Eq. (2).
  3. [Sec 3.3, Eq. (7)] The value of the weighting factor α is not reported. Please provide the value used in experiments and any sensitivity analysis.
  4. [Abstract] The phrase 'fully leverages the rich semantic information' overstates the evidence, since the paper does not quantify how much visual information is retained compared to text-only methods.
  5. [Sec 4.3] The statement that 'a direct relationship between the recognition accuracy and the final anticipation performance can be clearly observed' is not supported by a quantitative analysis; consider adding a correlation or scatter plot.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: ICVL predicts held-out future action sequences, its VLM intention comes from observed frames only, and ablations independently isolate the proposed modules.

full rationale

The derivation chain claimed by the paper — observed video, VLM-inferred intention (Eq. 1), ICAF cross-attention (Eq. 2), LLM decoding with next-token loss (Eq. 8) — does not reduce to its own inputs by construction. Intention inference uses only the last observed frame and previously inferred intentions from observed frames; no future action tokens are fed into the VLM or the LLM during inference. The reported Ego4D edit distances and EK-55/EGTEA mAP values are computed on held-out validation/test splits after end-to-end training of ICAF and LoRA, so the headline numbers are not fitted quantities renamed as predictions. The ablations in Tables 3–5 compare against a text-only baseline, no-intention visual embeddings, action-label-derived intentions, visual-token intentions, and different fusion schemes; these comparisons are internal and do not presuppose the conclusion that intention semantics work. The only author self-citation, [Cao et al., 2024b], is used as background support for multi-modality fusion being effective in short-term anticipation, but the paper's central claim does not rest on that citation because the present paper's own ablations demonstrate the effectiveness of its proposed fusion. Concerns that intention quality is never directly evaluated, or that cross-attention's extra capacity could explain the gains, are empirical underdetermination and robustness issues, not circularity: they do not make Eq. 2 equal to its input or make any reported result a tautology. Minor reporting inconsistencies, such as the text saying optimal results are attained with seven examples while the final setting uses three (Fig. 4 and Sec. 4.4), are clarity issues rather than definitional loops. The paper is therefore self-contained against its external benchmarks and exhibits no significant circularity.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The load-bearing machinery is external and frozen: CLIP ViT-L/14 (visual features), BLIP2-OPT-2.7B (visual encoder), LLaMA 3.2-9B (intention), and LLaMA 3-8B (text encoder and reasoning LLM). The paper contributes the ICAF cross-attention module (Eq. 2-3) and the weighted multimodal example selection (Eq. 4-7), which introduce two hand-chosen parameters, alpha and the example count k, neither reported with sensitivity analysis for all datasets. No new entities are postulated; 'intention' is free-form text from an existing VLM, which is why the two domain assumptions (intention quality and label-noise tolerance) carry the burden. The method is structural: a model combination with textbook components, so the ledger is short.

free parameters (2)
  • alpha (weight between text and visual similarity, Eq. 7) = not reported
    Balances text and visual similarity in example selection. Its value is never stated, while all reported results depend on it; no sensitivity study is given for EK-55 or EGTEA.
  • number of in-context examples (k) = 3 (used in final tables)
    Figure 4 shows action ED keeps improving up to 7 examples, yet the final configuration uses 3. The choice is manual and dataset-specific behavior is not reported.
assumptions (3)
  • domain assumption Behavioral intentions inferred from observed frames by an unfine-tuned VLM guide the evolution of future actions and improve anticipation.
    Section 3.1 and Eq. 2 make the intention text the query over all visual features; the paper never measures intention correctness, so this premise is unvalidated.
  • domain assumption Noisy observed action labels (7.97% verb-noun accuracy for the CLIP encoder on Ego4D, Sec. 4.3) remain informative enough to prompt the LLM and to drive text-similarity example selection.
    Sections 3.1 and 3.3 rely on predicted labels as inputs; the paper notes the recognition-accuracy and performance relationship but does not ablate the noise level.
  • standard math Scaled dot-product cross-attention behaves as specified by Vaswani et al. 2017.
    Equations 2 and 3 invoke standard attention without proof; a normal background assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Vision and Intention Boost Large Language Model in Long-Term Action Anticipation." pith.science (2026). https://pith.science/paper/WSUGO54K

@misc{pith2026250501713,
  author       = {Pith},
  title        = {Pith review of: Vision and Intention Boost Large Language Model in Long-Term Action Anticipation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WSUGO54K}},
  note         = {Machine review of arXiv:2505.01713}
}
read the original abstract

Long-term action anticipation (LTA) aims to predict future actions over an extended period. Previous approaches primarily focus on learning exclusively from video data but lack prior knowledge. Recent researches leverage large language models (LLMs) by utilizing text-based inputs which suffer severe information loss. To tackle these limitations single-modality methods face, we propose a novel Intention-Conditioned Vision-Language (ICVL) model in this study that fully leverages the rich semantic information of visual data and the powerful reasoning capabilities of LLMs. Considering intention as a high-level concept guiding the evolution of actions, we first propose to employ a vision-language model (VLM) to infer behavioral intentions as comprehensive textual features directly from video inputs. The inferred intentions are then fused with visual features through a multi-modality fusion strategy, resulting in intention-enhanced visual representations. These enhanced visual representations, along with textual prompts, are fed into LLM for future action anticipation. Furthermore, we propose an effective example selection strategy jointly considers visual and textual similarities, providing more relevant and informative examples for in-context learning. Extensive experiments with state-of-the-art performance on Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+ datasets fully demonstrate the effectiveness and superiority of the proposed method.

Figures

Figures reproduced from arXiv: 2505.01713 by the authors.

Figure 1
Figure 1. Illustration of different action anticipation methods. (a) [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Illustration of Intention-Conditioned Vision-Language (ICVL) model. Given a video, we use a VLM, a visual encoder, and an [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Illustration of prompt for LLMs using in-context learn [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Ablation study on the number of the Selected Examples. [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Compositional Benchmark Synthesis for Hierarchical Human Action Recognition

    cs.AI 2026-08 conditional novelty 7.0 of 10

    A synthesis protocol builds a regenerable four-level intention benchmark from flat action data and shows a consistent compositional held-out gap across four baseline model families.

Reference graph

Works this paper leans on

40 extracted references · 28 canonical work pages · cited by 1 Pith paper

  1. [1]

    Gpt-4 technical report

    [Achiam et al., 2023] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Alt- man, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774,

    arXiv 2023

  2. [5]

    The epic-kitchens dataset: Collection, challenges and baselines

    [Damen et al., 2020] Dima Damen, Hazel Doughty, Gio- vanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. The epic-kitchens dataset: Collection, challenges and baselines. IEEE Trans- actions on Pattern Analysis and Machine Intelligence , 43(11):4125–4141,

    work page 2020

  3. [6]

    A technique for computer detection and correction of spelling errors

    [Damerau, 1964] Fred J Damerau. A technique for computer detection and correction of spelling errors. Communica- tions of the ACM, 7(3):171–176,

    work page 1964

  4. [8]

    Rolling-unrolling lstms for action anticipation from first-person video

    [Furnari and Farinella, 2020] Antonino Furnari and Gio- vanni Maria Farinella. Rolling-unrolling lstms for action anticipation from first-person video. IEEE transactions on pattern analysis and machine intelligence , 43(11):4021– 4036,

    work page 2020

  5. [9]

    Future transformer for long-term action anticipation

    [Gong et al., 2022] Dayoung Gong, Joonseok Lee, Manjin Kim, Seong Jong Ha, and Minsu Cho. Future transformer for long-term action anticipation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3052–3061,

    work page 2022

  6. [11]

    Lora: Low-rank adaptation of large language models

    [Hu et al., 2021] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685,

    arXiv 2021

  7. [14]

    Technical report for ego4d long term action anticipation challenge

    [Ishibashi et al., 2023] Tatsuya Ishibashi, Kosuke Ono, Noriyuki Kugo, and Yuji Sato. Technical report for ego4d long term action anticipation challenge

    work page 2023

  8. [15]

    arXiv preprint arXiv:2307.01467,

    arXiv

Show all 40 references
  1. [16]

    Anticipating the start of user interaction for service robot in the wild

    [Ito et al., 2020] Koichiro Ito, Quan Kong, Shota Horiguchi, Takashi Sumiyoshi, and Kenji Nagamatsu. Anticipating the start of user interaction for service robot in the wild. In 2020 IEEE international conference on robotics and au- tomation (ICRA), pages 9687–9693. IEEE,

    2020

  2. [17]

    Palm: Predicting actions through language models

    [Kim et al., 2024] Sanghwan Kim, Daoji Huang, Yongqin Xian, Otmar Hilliges, Luc Van Gool, and Xi Wang. Palm: Predicting actions through language models. In European Conference on Computer Vision, pages 140–158. Springer,

    2024

  3. [18]

    Anticipating human activities using object affor- dances for reactive robotic response

    [Koppula and Saxena, 2015] Hema S Koppula and Ashutosh Saxena. Anticipating human activities using object affor- dances for reactive robotic response. IEEE transactions on pattern analysis and machine intelligence , 38(1):14– 29,

    2015

  4. [21]

    Llama-vid: An image is worth 2 tokens in large language models

    [Li et al., 2025] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision , pages 323–340. Springer,

    2025

  5. [23]

    Can’t make an omelette without breaking some eggs: Plausible action anticipation using large video-language models

    [Mittal et al., 2024] Himangi Mittal, Nakul Agarwal, Shao- Yuan Lo, and Kwonjoon Lee. Can’t make an omelette without breaking some eggs: Plausible action anticipation using large video-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Re...

    2024

  6. [24]

    Ego-topo: Environment affordances from egocentric video

    [Nagarajan et al., 2020] Tushar Nagarajan, Yanghao Li, Christoph Feichtenhofer, and Kristen Grauman. Ego-topo: Environment affordances from egocentric video. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 163–172,

    2020

  7. [25]

    Rethinking learning approaches for long- term action anticipation

    [Nawhal et al., 2022] Megha Nawhal, Akash Abdu Jyothi, and Greg Mori. Rethinking learning approaches for long- term action anticipation. InEuropean Conference on Com- puter Vision, pages 558–576. Springer,

    2022

  8. [26]

    Summarize the past to predict the future: Natural lan- guage descriptions of context boost multimodal object interaction anticipation

    [Pasca et al., 2024] Razvan-George Pasca, Alexey Gavryushin, Muhammad Hamza, Yen-Ling Kuo, Kaichun Mo, Luc Van Gool, Otmar Hilliges, and Xi Wang. Summarize the past to predict the future: Natural lan- guage descriptions of context boost multimodal object interaction anticipati...

    2024

  9. [27]

    Egovideo: Exploring egocen- tric foundation model and downstream adaptation

    [Pei et al., 2024] Baoqi Pei, Guo Chen, Jilan Xu, Yuping He, Yicheng Liu, Kanghua Pan, Yifei Huang, Yali Wang, Tong Lu, Limin Wang, et al. Egovideo: Exploring egocen- tric foundation model and downstream adaptation. arXiv preprint arXiv:2406.18070,

    2024 arXiv

  10. [28]

    Learning transferable visual models from nat- ural language supervision

    [Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from nat- ural language supervision. In International confere...

    2021

  11. [29]

    Pre- dicting the future from first person (egocentric) vision: A survey

    [Rodin et al., 2021] Ivan Rodin, Antonino Furnari, Dim- itrios Mavroeidis, and Giovanni Maria Farinella. Pre- dicting the future from first person (egocentric) vision: A survey. Computer Vision and Image Understanding , 211:103252,

    2021

  12. [30]

    En- couraging lstms to anticipate actions very early

    [Sadegh Aliakbarian et al., 2017] Mohammad Sadegh Ali- akbarian, Fatemeh Sadat Saleh, Mathieu Salzmann, Ba- sura Fernando, Lars Petersson, and Lars Andersson. En- couraging lstms to anticipate actions very early. In Pro- ceedings of the IEEE International Conference on Com- pu...

    2017

  13. [32]

    Llama: Open and efficient founda- tion language models

    [Touvron et al., 2023] Hugo Touvron, Thibaut Lavril, Gau- tier Izacard, Xavier Martinet, Marie-Anne Lachaux, Tim- oth´ee Lacroix, Baptiste Rozi`ere, Naman Goyal, Eric Ham- bro, Faisal Azhar, et al. Llama: Open and efficient founda- tion language models. arXiv preprint arXiv:23...

    2023 arXiv

  14. [33]

    Attention is all you need

    [Vaswani, 2017] A Vaswani. Attention is all you need. Ad- vances in Neural Information Processing Systems,

    2017

  15. [34]

    Memory-and-anticipation transformer for online action understanding

    [Wang et al., 2023] Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang, and Tong Lu. Memory-and-anticipation transformer for online action understanding. In Proceed- ings of the IEEE/CVF International Conference on Com- puter Vision, pages 13824–13835,

    2023

  16. [35]

    Visionllm: Large language model is also an open-ended decoder for vision- centric tasks

    [Wang et al., 2024] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also an open-ended decoder for vision- centric tasks. Advances in Neural Information Processing Systems, 36,

    2024

  17. [36]

    Black-box prompt tuning for vision-language model as a service

    [Yu et al., 2023] Lang Yu, Qin Chen, Jiaju Lin, and Liang He. Black-box prompt tuning for vision-language model as a service. In IJCAI, pages 1686–1694,

    2023

  18. [37]

    Antgpt: Can large language mod- els help long-term action anticipation from videos? arXiv preprint arXiv:2307.16368,

    [Zhao et al., 2023] Qi Zhao, Shijie Wang, Ce Zhang, Changcheng Fu, Minh Quan Do, Nakul Agarwal, Kwon- joon Lee, and Chen Sun. Antgpt: Can large language mod- els help long-term action anticipation from videos? arXiv preprint arXiv:2307.16368,

    2023 arXiv

  19. [38]

    Anticipative feature fusion transformer for multi-modal action anticipation

    [Zhong et al., 2023] Zeyun Zhong, David Schneider, Michael V oit, Rainer Stiefelhagen, and J ¨urgen Beyerer. Anticipative feature fusion transformer for multi-modal action anticipation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages...

    2023

  20. [39]

    [Zhong et al., 2024] Zeyun Zhong, Manuel Martin, Fred- erik Diederichs, and Juergen Beyerer. Querymamba: A mamba-based encoder-decoder architecture with a statisti- cal verb-noun interaction module for video action forecast- ing@ ego4d long-term action anticipation challenge

    2024

  21. [40]

    arXiv preprint arXiv:2407.04184, 2024

    2024 arXiv

  22. [1964]

    The llama 3 herd of models

    [Dubey et al., 2024] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783,

    2024 arXiv

  23. [2015]

    In the eye of beholder: Joint learning of gaze and actions in first person video

    [Li et al., 2018] Yin Li, Miao Liu, and James M Rehg. In the eye of beholder: Joint learning of gaze and actions in first person video. In Proceedings of the European conference on computer vision (ECCV), pages 619–635,

    2018

  24. [2017]

    Temporal aggregate representations for long- range video understanding

    [Sener et al., 2020] Fadime Sener, Dipika Singhania, and Angela Yao. Temporal aggregate representations for long- range video understanding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVI 16 , pages 154–171. Springer,

    2020

  25. [2018]

    Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models

    [Li et al., 2023] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre- training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR,

    2023

  26. [2019]

    Videograph: Rec- ognizing minutes-long human activities in videos

    [Hussein et al., 2019b] Noureldien Hussein, Efstratios Gavves, and Arnold WM Smeulders. Videograph: Rec- ognizing minutes-long human activities in videos. arXiv preprint arXiv:1905.05143,

    1905 arXiv

  27. [2020]

    Sgdcl: Semantic-guided dynamic correlation learning for explain- able autonomous driving

    [Cao et al., 2024a] Chengtai Cao, Xinhong Chen, Jianping Wang, Qun Song, Rui Tan, and Yung-Hui Li. Sgdcl: Semantic-guided dynamic correlation learning for explain- able autonomous driving. In 33rd International Joint Con- ference on Artificial Intelligence (IJCAI 2024), pages 596–

    2024

  28. [2021]

    Using gaze patterns to pre- dict task intent in collaboration

    [Huang et al., 2015] Chien-Ming Huang, Sean Andrist, Alli- son Saupp´e, and Bilge Mutlu. Using gaze patterns to pre- dict task intent in collaboration. Frontiers in psychology, 6:1049,

    2015

  29. [2022]

    Ego4d: Around the world in 3,000 hours of egocentric video

    [Grauman et al., 2022] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Ro- hit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Confe...

    2022

  30. [2023]

    Language models are few-shot learners

    [Brown et al., 2020] Tom Brown, Benjamin Mann, Nick Ry- der, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing sys- tems,...

    2020

  31. [2024]

    A survey on multi- modal large language models for autonomous driving

    [Cui et al., 2024] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multi- modal large language models for autonomous driving. In Proceedings of the IEEE/CVF Winter Conference on Ap- plic...

    2024

  32. [2025]

    Ego- centric video-language pretraining

    [Lin et al., 2022] Kevin Qinghong Lin, Jinpeng Wang, Mat- tia Soldan, Michael Wray, Rui Yan, Eric Z Xu, Difei Gao, Rong-Cheng Tu, Wenzhe Zhao, Weijie Kong, et al. Ego- centric video-language pretraining. Advances in Neural Information Processing Systems, 35:7575–7586,

    2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.