{"id":"0fee5c18-f5e9-471a-9cc4-d36fc4b815b3","arxiv_id":"2505.01713","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+, intention-conditioned visual features plus multimodal in-context examples beat published long-term action anticipation models.","lead":"An egocentric video system that asks a vision-language model what the person in the video intends to do, then fuses that text with the video features and feeds both to a fine-tuned LLM, reports new best results on three action-anticipation benchmarks. The recipe is a practical template for injecting high-level goals into LLM-based video prediction, with caveats around missing code, error bars, and unvalidated intentions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The core mechanism—VLM-inferred intention as the ICAF query (Eq. 2)—is never validated; end-task gains alone cannot establish that intention semantics, rather than extra capacity or attention flexibility, drive the improvement.","rationale":"The weakest assumption identified by the reader is the unvalidated VLM intention, and that is also the most load-bearing point in this paper. The central contribution is not the LLM or the example selection per se; it is the claim that inferred behavioral intentions, injected as the query in Eq. 2, make the visual representation more discriminative and causally guide anticipation. The ablations are directionally supportive: ICAF improves Ego4D action edit distance from 0.8944 to 0.8643, and VLM intentions outperform action-label intentions and learned tokens. However, those ablations compare complete pipelines and do not isolate whether the semantic correctness of the intention text matters, or whether any learned query vector would have produced similar gains. The paper does not report intention accuracy, human evaluation, error analysis, or even example outputs of inferred intentions. Since Eq. 2 forces all visual information through the intention query, a wrong or hallucinated intention could systematically reweight visual features around irrelevant content; the claimed robustness to noise is asserted, not measured. The lack of code and error bars further weakens the ability to verify the reported margins, though this is a reproducibility concern rather than a direct refutation. The likely verdict remains CONDITIONAL: the approach is plausible and internally consistent, but the central causal claim needs an explicit intention-quality check or a perturbation experiment to be accepted at face value.","tokens_in":12696,"tokens_out":5720,"duration_ms":61289,"concrete_test":"On Ego4D, train the ICAF module with identical settings under three intention conditions: (a) reported LLaMA-generated intentions, (b) an oracle intention derived from ground-truth future actions, and (c) intentions randomly permuted across training videos. Compare validation action edit distance. If (c) is not significantly worse than (a), semantic intention content is not the driver; if (a) is far from (b), the VLM inference leaves large room for error. As a complementary check, have annotators or a second VLM score whether each inferred intention matches the observed and future action labels, and report the correlation between intention correctness and per-video edit distance.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is the claim that VLM-inferred behavioral intentions provide high-level semantic guidance, and ICAF (Eq. 2: E_ic = Attention(E_i, E_v, E_v)) reroutes the visual sequence through the intention query. This claim rests entirely on end-task edit-distance improvements; the paper reports no measurement of intention accuracy, no human inspection, and no analysis of cases where the inferred intention is wrong. Because cross-attention with learnable projections has extra capacity, the observed gain could come from additional attention parameters or from any text-like query forcing useful pooling of visual tokens, rather than from the specific semantic content of \"what the person wants to do.\" This is load-bearing: Eq. 2 uses the intention as the only query, so if LLaMA 3.2-9B hallucinates or produces a generic intention, the visual features are reweighted around an uninformative vector and the \"intention-enhanced visual embeddings\" mechanism is not established. Related reporting gaps (no code, no error bars, inconsistent \"Baseline\" labels in Tables 3/4, and final three examples despite seven being called optimal) further prevent checking whether the SOTA margins are significant.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Intention-Conditioned Vision-Language (ICVL), a framework for long-term action anticipation that: (1) uses a VLM to infer behavioral intentions from observed frames, (2) fuses intention embeddings with visual embeddings via a cross-attention module called ICAF (Eq. 2), and (3) selects in-context examples using a weighted combination of visual and textual similarity (Eq. 7). The model is evaluated on Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+, reporting state-of-the-art edit distances and mean average precision. Ablations show that both ICAF and example selection contribute to the reported improvements.","tokens_in":12739,"tokens_out":5990,"duration_ms":53938,"significance":"If validated, the paper makes a useful contribution by combining LLM reasoning with visual and intention information for long-term action anticipation. The ablations in Tables 3-6 are informative, and the use of a frozen, unfine-tuned VLM with held-out splits reduces internal circularity. However, the central claim that VLM-inferred intention semantics are responsible for the gains is not directly validated: the ICAF module adds learnable capacity, and no control experiment removes the specific semantic content of the intention. The reported improvements over prior SOTA are also small and lack error bars. The paper is a solid empirical study but requires additional controls and clarifications to establish the proposed mechanism.","major_comments":[{"comment":"The paper attributes the improvement from ICAF to the semantic content of the VLM-inferred intention, but this is not isolated. Cross-attention with learnable projections adds capacity, and any text-like or learned query could produce similar gains. There is no evaluation of intention quality, no human inspection, and no control experiment with random or generic queries. Please add a control that removes intention semantics (e.g., using a fixed or random text query, or learnable query tokens) to demonstrate that the specific intention content is what matters.","section":"Sec 3.2, Eq. (2)"},{"comment":"The two 'Baseline' rows have inconsistent values: Table 3 reports 0.6927/0.6823/0.8944 for noun/verb/action, while Table 4 reports 0.6469/0.6661/0.8773. This inconsistency prevents a clean interpretation of the ablations. Please clarify the exact components included in each baseline and use consistent labels across tables.","section":"Sec 4.4, Tables 3-4"},{"comment":"The formulation of the top-k selection objective is incorrect as written: the sum runs over all elements in Ω rather than over the selected set U, so the objective is independent of U and the argmin is ill-defined. Please correct the index set to reflect selection from U.","section":"Sec 3.3, Eq. (4)"},{"comment":"The text states 'optimal results attained when seven examples are provided' but then says 'We report the final performance based on the use of three examples.' This is contradictory and affects reproducibility. Please clarify the exact number of examples used in the final experiments and whether the difference between three and seven is within noise.","section":"Sec 4.4, Figure 4"},{"comment":"The claim that visual similarity beats textual similarity is not supported by the noun column (Text 0.7299 vs Visual 0.7330, lower is better). Additionally, the relation of these rows to the Baseline in Table 3 is unclear; the Text and Visual rows appear worse than Baseline, yet the Fused row is better. Please specify the configuration of Table 6 (e.g., whether ICAF is included) and discuss the per-metric pattern.","section":"Sec 4.4, Table 6"},{"comment":"The improvement over EgoVideo is small (relative improvements of 0.7%, 0.6%, and 0.49% on noun, verb, and action). With no error bars or multiple seeds, it is difficult to assess whether the state-of-the-art claim is statistically significant. Please report variance or significance tests, and ensure comparisons use comparable training conditions.","section":"Sec 4.3, Table 1"}],"minor_comments":[{"comment":"The mechanism by which E_ic is fed into the decoder-only LLM is underspecified. Please state whether E_ic tokens are prepended as prefix tokens and how they are projected to the LLM embedding space.","section":"Sec 3.2"},{"comment":"The role of BLIP2-OPT-2.7B as the 'frozen visual encoder' for ICAF is unclear, given that CLIP ViT-L/14 is described as the visual encoder for action recognition. Please clarify which visual features are used in Eq. (2).","section":"Sec 4.2"},{"comment":"The value of the weighting factor α is not reported. Please provide the value used in experiments and any sensitivity analysis.","section":"Sec 3.3, Eq. (7)"},{"comment":"The phrase 'fully leverages the rich semantic information' overstates the evidence, since the paper does not quantify how much visual information is retained compared to text-only methods.","section":"Abstract"},{"comment":"The statement that 'a direct relationship between the recognition accuracy and the final anticipation performance can be clearly observed' is not supported by a quantitative analysis; consider adding a correlation or scatter plot.","section":"Sec 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of the journal, but the central mechanism—VLM-inferred intention as the driver of ICAF improvements—needs stronger validation before acceptance. The inconsistencies in baseline tables, the contradictory statement about the number of examples, and the lack of error bars should be addressed. I recommend major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a legitimate empirical paper that improves LLM-based long-term action anticipation by fusing VLM-inferred intentions into visual features. The real novelty is the combination: VLM-produced free-form intention text used as the cross-attention query over visual tokens, plus a multi-modal example selector. Each ingredient has a visible prior, but the specific pairing is new, and the ablations support the direction—ICAF moves Ego4D action edit distance from 0.8944 to 0.8643, VLM intentions beat action-label and learned-token intentions, and fused example selection beats single-modality selection. That is real work, and it is honestly reported at the task level.\n\nThe paper does several things well. It evaluates on three datasets, compares against AntGPT under the same CLIP encoder, uses parameter-efficient tuning (frozen encoders, LoRA), and includes a sensible discussion of how noisy action labels interact with the approach. The acknowledgments in Section 5 that logical coherence is not enforced is a fair limitation statement.\n\nThe soft spots, in order of concern. First, the load-bearing assumption: the VLM-inferred intention sentence is never validated. Since Eq. 2 uses that sentence as the only query over all visual features, a hallucinated or generic intention would reweight the whole visual sequence around an uninformative vector. The ablation against action labels and learned tokens gives indirect evidence that the VLM text carries useful semantics, but it does not show how often the intention is wrong, nor rule out that part of the gain comes simply from a longer, more flexible query giving cross-attention extra capacity. A human-rated or automatic correctness check on a sample of intentions would settle this cleanly. Second, reporting gaps: the \"Baseline\" row means different things in Tables 3 and 4, the text says seven examples are optimal but the final tables use three, and alpha, LoRA rank, the full prompt template, and frame sampling are absent. There is no code, data, or error bars. Some of these are minor, but together they block reproducing the SOTA claim. Third, the margins over the strongest baselines on Ego4D and EGTEA are under one percent, so the abstract's \"fully demonstrate superiority\" is stronger than the numbers justify. The EK-55 gain (2.9 mAP) is more substantial.\n\nWho this is for: researchers working on egocentric action anticipation, especially LLM-based pipelines, will get a useful recipe and a clear set of comparisons. It is not a field-reshaping result, but it is a plausible step forward.\n\nRecommendation: send it to peer review. The mechanism is plausible and the ablations support the main claim, which is more than many papers in this area offer. It needs major revision: validate the intentions, fix the reporting inconsistencies, release code or at least detailed hyperparameters, and calibrate the abstract to the actual margins. If the authors can show intention quality correlates with end-task gains, this becomes a solid contribution.","headline":"A useful, well-ablated recipe for LLM-based action anticipation, held back by an unvalidated intention module and reproducibility gaps; deserves review with major revision.","tokens_in":13509,"tokens_out":2350,"would_cite":false,"duration_ms":24823,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that inferring a person's behavioral intention from observed video and fusing that intention into visual embeddings through cross-attention lets an LLM predict long-term future actions more accurately than either…","keywords":["long-term action anticipation","behavioral intention inference","vision-language models","cross-attention fusion","in-context learning","egocentric video","large language models","Ego4D"],"falsifier":"Take the trained ICVL model on Ego4D and replace the VLM-inferred intention text with a fixed random or deliberately wrong intention, such as 'do the dishes' for a gardening clip, while keeping visual features and all other inputs identical. If edit distances do not worsen substantially relative to the reported 0.8570 action edit distance, the intention-conditioned fusion is not the source of the gain. A softer test is to have humans rate intention accuracy on a random sample of 100 clips and check whether per-clip prediction error tracks rated intention quality; the paper contains no such evaluation.","tokens_in":12280,"feed_emoji":"🎯","tokens_out":6173,"duration_ms":54327,"temperature":0.7,"pith_summary":"The paper argues that long-term action anticipation fails when a model sees only pixels or only text: visual features are redundant, and text substitutes such as action labels or captions lose detail. It proposes ICVL, which asks a vision-language model to state in words what the person in the video wants to do, then uses that intention sentence as the query in cross-attention over the video's visual embeddings to produce intention-enhanced visual representations. Those representations, together with a textual prompt and selected in-context examples, are fed into a LoRA-tuned LLM that generates the future action sequence. The paper reports state-of-the-art results on Ego4D, EPIC-Kitchens-55, and EGTEA GAZE+ under the respective metrics, with ablations attributing the main gain to the intention-conditioned fusion.","feed_headline":"Inferred intentions push LLM action forecasting past text-only models","feed_subtitle":"Fusing inferred intentions into visual embeddings beats text-only LLM baselines on Ego4D, EK-55, and EGTEA.","key_machinery":"The load-bearing mechanism is the Intention-Context Attention Fusion (ICAF) module: a cross-attention layer in which the text embedding of the VLM-inferred intention is the query and the visual embeddings of the observed video are the keys and values. This turns a free-form sentence answering 'What does the person want to do?' into a reweighting of every visual token before the LLM reads them. The second mechanism is multi-modality example selection: training clips are ranked by a weighted sum of normalized L2 similarity in visual space and in observed-label text space, and the top-k clips are inserted into the prompt as in-context demonstrations. Both feed a LLaMA-3-8B LLM fine-tuned with LoRA, with frozen CLIP and BLIP2 encoders supplying the visual features.","core_discovery":"The central claim is that a high-level behavioral intention inferred directly from observed video, stated as a free-form sentence by a VLM, can act as a cross-modal key that reshapes visual embeddings before they reach an LLM, and that this is what lets the LLM anticipate long-term actions better than text-only or vision-only pipelines. Concretely, with intention embeddings as queries and visual embeddings as keys and values, the attention output reweights the visual sequence toward cues consistent with the inferred goal; the resulting embeddings are passed to the LLM along with a prompt containing observed action labels and the k most similar training examples selected by a weighted visual-textual similarity. The paper claims this yields the lowest edit distances on Ego4D (noun 0.6194, verb 0.6516, action 0.8570) and the highest mean average precision on EPIC-Kitchens-55 (ALL 43.3) and EGTEA GAZE+ (ALL 81.0), while remaining robust to noisy observed action labels (7.97% verb-noun accuracy with a CLIP encoder).","pith_inferences":["The paper does not measure intention quality, so the reported gains could in principle come from the extra visual tokens, the extra parameters, or the LoRA tuning rather than from the semantics of the intention sentence; replacing the intention text with a scrambled version of itself (same token count, random order) would separate those explanations.","The intention-as-query design suggests a latent-variable reading: the VLM sentence is a discrete proxy for a continuous intention variable, and a natural next step is to train the intention encoder end-to-end with a goal-consistency loss against the future action sequence rather than freezing it.","Because the example-selection weight α is a fixed scalar, a testable extension is to learn α per query video or condition it on the confidence of the action recognizer; low-confidence queries would lean more on visual similarity, which the ablation shows is the stronger single modality."],"forward_implications":["If ICVL is right, LLM-based long-term action anticipation does not have to choose between pixels and text: feeding intention-reweighted visual embeddings alongside textual prompts yields lower edit distances on Ego4D than the reported text-only or vision-only baselines.","Intention-conditioned fusion should make predictions more robust to noisy observed action labels, since the reported gains over AntGPT hold even though ICVL's action recognizer (CLIP, 7.97% verb-noun accuracy) is far weaker than EgoVideo's encoder.","Joint visual-textual example selection should transfer to other LLM-based video tasks that pair a query video with in-context demonstrations, because it only requires a visual encoder, a text encoder, and similarity scores.","The ablations predict that cross-attention with the intention as query is better than either concatenation or cross-attention with the visual as query, making the query choice itself part of the mechanism's effectiveness."],"supporting_citations":[{"why":"Introduces the text-only LLM baseline AntGPT on Ego4D that ICVL extends and beats under the same CLIP encoder.","marker":"[Zhao et al., 2023]"},{"why":"PALM is the prior caption-based LLM method that ICVL outperforms on all three datasets, representing the text-only approach ICVL argues loses information.","marker":"[Kim et al., 2024]"},{"why":"EgoVideo supplies the strongest visual-encoder baseline; ICVL beats it despite using a much weaker CLIP encoder (7.97% versus 27.64% recognition accuracy).","marker":"[Pei et al., 2024]"},{"why":"Establishes in-context learning, the mechanism the paper's multi-modality example selection is designed to improve.","marker":"[Brown et al., 2020]"},{"why":"BLIP2-OPT-2.7B is the frozen visual encoder used to extract the visual embeddings that ICAF reweights.","marker":"[Li et al., 2023]"},{"why":"Supplies LLaMA 3-8B, the LLM that generates the future action sequence, and LLaMA 3.2-9B, the VLM that infers behavioral intentions.","marker":"[Dubey et al., 2024]"},{"why":"LoRA is the parameter-efficient fine-tuning method used to adapt the LLM in ICVL.","marker":"[Hu et al., 2021]"},{"why":"Ego4D provides the benchmark and evaluation protocol (edit distance over five predicted sequences) used in the main experiments.","marker":"[Grauman et al., 2022]"}],"fun_headline_variants":["Inferred intentions sharpen LLM long-term action forecasts","Intention-infused vision boosts LLM action anticipation","Fusing inferred goals into visual cues improves LLM anticipation","Intention-conditioned video-language model leads action anticipation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the free-form intention sentence a VLM produces when asked 'What does the person want to do?' on the last observed frame is accurate and stable enough to act as the attention query over all visual features, yet the paper reports no measurement of intention quality; a hallucinated intention would reweight every visual token around a wrong goal and propagate into the LLM's predictions.","fun_headline_variants_meta":{"raw":{"variants":["Inferred intentions sharpen LLM long-term action forecasts","Intention-infused vision boosts LLM action anticipation","Fusing inferred goals into visual cues improves LLM anticipation","Intention-conditioned video-language model leads action anticipation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000596,"raw_usage":{"total_tokens":2818,"prompt_tokens":1005,"completion_tokens":1813,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":1749}},"tokens_in":621,"tokens_out":1813,"duration_ms":14686,"temperature":1.0,"reasoning_tokens":1749,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:12:40.146690+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained ICVL model on Ego4D and replace the VLM-inferred intention text with a fixed random or deliberately wrong intention, such as 'do the dishes' for a gardening clip, while keeping visual features and all other inputs identical. If edit distances do not worsen substantially relative to the reported 0.8570 action edit distance, the intention-conditioned fusion is not the source of the gain. A softer test is to have humans rate intention accuracy on a random sample of 100 clips and check whether per-clip prediction error tracks rated intention quality; the paper contains no such evaluation.","supporting_citations":[{"cited_title":"Palm: Predicting actions through language models","cited_arxiv_id":null,"evidence_quote":"PALM is the prior caption-based LLM method that ICVL outperforms on all three datasets, representing the text-only approach ICVL argues loses information."},{"cited_title":"Ego4d: Around the world in 3,000 hours of egocentric video","cited_arxiv_id":null,"evidence_quote":"Ego4D provides the benchmark and evaluation protocol (edit distance over five predicted sequences) used in the main experiments."}],"review_version":1}