{"id":"5769d428-ec80-4a14-a91c-ef36b4e02a99","arxiv_id":"2508.15903","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"VT-LVLM-AR converts video into quantized 'visual event sentences' and classifies actions with a frozen LLaVA-1.5 using prompt tuning, claiming 94.1% on NTU RGB+D X-Sub.","lead":"The paper describes a system that turns videos into short token sequences, like sentences, and feeds these into a frozen vision-language model to recognize human actions. The authors report top accuracy on standard action recognition benchmarks, but the method description is missing key implementation details.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The VTEM-to-LLaVA interface is unspecified: 1024-dim event tokens cannot be 'directly fed' into frozen LLaVA-1.5, which expects CLIP patch embeddings; a hidden trainable projection would break the frozen-core/1.2M-parameter claim.","rationale":"The reader's weakest assumption pinpoints the same load-bearing issue: the paper never specifies how 1024-dim visual event tokens are fed into a frozen LLaVA-1.5. This is not a minor implementation detail; it determines whether the proposed architecture is even instantiable. LLaVA-1.5 has a rigid vision interface: a CLIP ViT produces a fixed grid of visual tokens, then an MLP projects them to language embeddings. The paper's Eq. (3) says the token dimension is 'designed to match' LLaVA's visual input, but no matching mechanism is described. If the authors inserted a trainable projector, the systemtrainable-parameter count and the 'frozen LLaVA' narrative are inaccurate; if they did not, the input is incompatible. This internal inconsistency undermines the central claim more directly than the other cited issues (missing code, error bars, skeleton-only baselines) because it affects the core architecture rather than just the reporting quality. I agree with the reader's rejection. The concrete test—running the described forward pass or inspecting the model definition—would decisively settle whether the interface exists and whether the frozen-core claim holds.","tokens_in":12296,"tokens_out":3113,"duration_ms":31742,"concrete_test":"Obtain or reimplement the exact forward pass: take 256 1024-dim visual event tokens from a trained VTEM, a frozen LLaVA-1.5 checkpoint, and 16 P-Tuning v2 soft prompts. Run the pipeline as described in Eqs. (5)–(7). If a shape mismatch occurs or a trainable projector must be added to map tokens into the LLM embedding space, the frozen-core claim fails. Report input tensor shapes, the insertion layer, and whether all projector/ViT weights remain frozen.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of a frozen-LLaVA pipeline achieving state-of-the-art accuracy depends on the VTEM output being directly ingestible by LLaVA-1.5. Section III-A states (Eq. 3) that each visual event token e_j is a 1024-dimensional vector, 'designed to match the visual input dimension expected by the subsequent LVLM.' Section III-B then says the visual event sentence E is 'directly fed as visual input to the LLaVA-1.5 model.' But LLaVA-1.5's visual pathway is not a generic 1024-dim token input: it takes image patches through a CLIP ViT, producing a fixed number of visual tokens (e.g., 576 for 336×336 inputs), which are then projected by an MLP into the LLM embedding space. A frozen LLaVA-1.5 cannot accept 256 arbitrary 1024-dim codebook vectors in place of these patch embeddings without a new projection or interface. The paper never specifies this interface—whether a trainable projector is inserted, whether the LLaVA vision tower is bypassed, or how the sequence-length mismatch is handled. If a trainable projection is required, the 'frozen' claim and the 1.2 million trainable-parameter count (Table IV) are violated. If no projection is used, the described input is architecturally incompatible with LLaVA-1.5. Either way, the central claim as written is not reproducible or verifiable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"VT-LVLM-AR proposes a two-stage action recognition pipeline: a Video-to-Event Mapper (VTEM) converts raw RGB video into M discrete 'visual event tokens' via a spatio-temporal extractor (left unspecified), adaptive temporal pooling, and codebook quantization; the resulting token sequence ('visual event sentence') is then fed to a frozen LLaVA-1.5 model adapted by P-Tuning v2 soft prompts for action classification. The paper reports 94.1% on NTU-60 X-Sub, 96.8% X-View, 87.0% on NTU-120 X-Sub, and 88.5% X-Set, together with ablations, a human evaluation of interpretability, and a parameter-efficiency analysis.","tokens_in":97,"tokens_out":5075,"duration_ms":102531,"significance":"The paper targets an important and timely problem: adapting a powerful, frozen LVLM to long, fine-grained video action recognition through a compact intermediate representation. The modular idea of a discrete, temporally coherent 'visual event sentence' is appealing and, if carefully validated, could be a meaningful step toward interpretable and parameter-efficient video understanding. The manuscript has clear conceptual value, and the human evaluation of interpretability is a nice addition. However, as written, the central architecture is under-specified and the empirical claims are not well supported: there is no code, no error bars, and the baselines are predominantly skeleton-based. The strengths of the paper are therefore only potential; the current evidence is insufficient to verify the SOTA claim.","major_comments":[{"comment":"The core interface between VTEM and LLaVA-1.5 is not specified and appears architecturally incompatible. Section III-B says the visual event sentence E is 'directly fed as visual input' to LLaVA-1.5. LLaVA-1.5's visual pathway expects CLIP patch embeddings (e.g., 576 tokens from a ViT) and projects them with an MLP; it does not accept 256 arbitrary 1024-dimensional codebook vectors. If a trainable projection is inserted, the 'frozen LLaVA' claim and the 1.2M trainable-parameter count in Table IV are violated; if no projection is used, the described input cannot be ingested. The authors must specify the exact interface, including sequence-length handling and whether the vision tower is bypassed.","section":"III-B, Eqs. (3)-(5)"},{"comment":"The VTEM training objective is not computable as stated. L_VTEM = αL_rec(P,E) + βL_cont(E) requires an explicit decoder that reconstructs pooled features P from event tokens E, but no decoder is described. In addition, the spatio-temporal feature extractor is left open ('can be based on architectures such as ...'), yet the experiments never name the actual extractor used. Without these details, the reported 94.1% result cannot be reproduced or independently verified.","section":"III-A, Eq. (4)"},{"comment":"The ablation differences are very small (e.g., 94.1 vs. 94.0, 93.8, and 92.7) and no repeated runs, error bars, or statistical significance tests are reported. The conclusion that 'each component of the VTEM module is vital' is not supported by these deltas. At minimum, the authors should report mean±std over multiple seeds and provide significance tests. Also, choices of M and K are tuned on the validation set; the 'optimal' 94.1% should not be presented as evidence beyond the selected configuration.","section":"IV-D, Tables II, V, VI"},{"comment":"The state-of-the-art claim is not established by the comparison set. Table I lists ST-GCN, Shift-GCN, InfoGCN, PoseC3D, FR-Head, Koopman, GAP, HD-GCN, and STC-Net, which are predominantly skeleton-based. The paper does not compare against recent RGB-video or video-LVLM baselines. To support the claim of SOTA for video input, the authors need to include contemporary RGB/transformer and video-LLM methods under the same protocols.","section":"IV-B, Table I"}],"minor_comments":[{"comment":"References [8] and [9] do not point to LLaVA-1.5 or P-Tuning v2, respectively; the citations appear mismatched and should be corrected.","section":"References"},{"comment":"The reported 1.2M trainable parameters is not derived. With 16 soft prompt tokens inserted into specific layers, the count depends on the number of layers and hidden dimensions; please specify the calculation.","section":"IV-F, Table IV"},{"comment":"The human evaluation lacks details: number of clips, annotator selection, blinding, and inter-annotator agreement are not reported, so the interpretability claim is difficult to assess.","section":"IV-E"},{"comment":"There are repeated typographical issues (e.g., 'LLaV A' instead of 'LLaVA') and inconsistent notation in Eqs. (1)-(2) (mixing h_j..h_k and p_i). A careful proofread is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears very early-stage: several citations are incorrect and core implementation details are absent. If a revised version is submitted, I would strongly encourage the authors to release code or a detailed reproducibility appendix, and to re-run experiments with standard error reporting. The paper's fit for a vision journal is appropriate in principle, but the current version is not verifiable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The idea here is genuinely worth a look: adaptive temporal pooling plus codebook quantization to turn video into a compact sequence of “visual event tokens,” then feeding those to a frozen LLaVA-1.5 with prompt tuning. That combination is not in the cited literature, and the framing of a “visual event sentence” as an interpretable intermediate is a nice instinct. The ablations and human evaluation are the right way to probe the contribution, and the comparison to full fine-tuning (94.0 vs 94.1) fairly supports the efficiency argument.\n\nBut there is a load-bearing gap at the center of the method. LLaVA-1.5 does not accept arbitrary 1024-dim vectors as visual input; it expects CLIP patch embeddings through a vision tower and a specific MLP projector. The paper says the event tokens are “designed to match the visual input dimension” and are “directly fed” to LLaVA, but it never says how the sequence of 256 codebook vectors bypasses or replaces the patch embeddings, how the projector is handled, or what happens to the sequence-length mismatch. If a trainable projection is inserted, the frozen-core and 1.2M-parameter claims are violated; if not, the input is architecturally incompatible. This is not a minor omission, it breaks the reproducibility of the central claim.\n\nOther soft spots compound it: no error bars reported, and the ablation margins (93.3 vs 94.1, 93.8 vs 94.1) are small enough to be noise. Most baselines are skeleton-based, which is a mismatch for a video-LVLM method. The feature extractor is left open (“can be based on a 3D ResNet or Swin Transformer”), and some references do not bear the weight of the claims they are attached to. These are fixable, but together they mean the quantitative results are not yet verifiable.\n\nThe paper is clearly written and the authors are not sloppy in their reasoning; the interface problem is a missing detail rather than an internal contradiction. I would not cite this in its current form, and I would not trust the numbers until the architecture is pinned down and code or detailed specs are released. But the core idea deserves a serious referee: a good reviewer could push the authors to specify the projection, report variance, and run fairer RGB baselines. That would give the field a much more useful paper.\n\nRecommendation: send to peer review, with expected major revision.","headline":"Nice idea, missing interface: the paper never specifies how 1024-dim event tokens become LLaVA-1.5's visual input, so the frozen-core and SOTA claims don't yet hold up.","tokens_in":13151,"tokens_out":1703,"would_cite":false,"duration_ms":20610,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A video-to-event token pipeline lets a frozen LLaVA-1.5 outperform dedicated action recognizers on NTU RGB+D and NTU RGB+D 120.","keywords":["long-term video action recognition","large vision-language models","visual event tokens","conceptual quantization","adaptive temporal pooling","prompt tuning","NTU RGB+D","parameter-efficient adaptation"],"falsifier":"Freeze every LLaVA-1.5 parameter, including the vision encoder and visual projector, train only the 16 soft prompts, and reproduce the 94.1% NTU RGB+D X-Sub result while logging the tensor path from the 1024-dimensional event tokens into the model. If a learned linear projection or image-patch embedding is required and updated, the frozen-core premise fails; if replacing LLaVA with a non-reasoning classifier on the same event tokens still gives about 94% accuracy, then the claimed contribution of LVLM reasoning is falsified.","tokens_in":1589,"feed_emoji":"🎬","tokens_out":2165,"duration_ms":118291,"temperature":0.7,"pith_summary":"VT-LVLM-AR's central claim is that a long video can be summarized as a short, language-like sequence of discrete \"visual event tokens,\" and that a frozen Large Vision-Language Model can classify actions from that sequence as accurately as specialized models. The Video-to-Event Mapper learns these tokens through lightweight spatio-temporal feature extraction, adaptive temporal pooling, and vector quantization against a learned codebook, plus a coherence loss that keeps adjacent tokens narratively ordered. The frozen LLaVA-1.5 is then adapted with only 16 soft prompts via prompt tuning, leaving about 1.2 million trainable parameters. The paper reports state-of-the-art results on NTU RGB+D and NTU RGB+D 120, with the largest tested gains coming from the quantization step. A reader should care because the claim, if right, offers a parameter-efficient and more interpretable route from raw video into the reasoning machinery of a general vision-language model.","feed_headline":"Video event tokens let a frozen LLaVA hit 94.1% on NTU-60","feed_subtitle":"Only 1.2 million soft-prompt parameters are trained; the 7-billion-parameter LLaVA backbone stays frozen.","key_machinery":"The Video-to-Event Mapper (VTEM) is the load-bearing component. It converts raw frames into roughly 256 discrete event tokens by (1) extracting lightweight spatio-temporal segment features, (2) adaptively pooling those features over varying temporal windows, (3) quantizing each pooled feature to the nearest entry in a learned codebook of about 2048 visual concepts, and (4) applying an event-coherence bias through reconstruction and contrastive losses. The resulting visual event sentence is concatenated with a natural-language task instruction and 16 learnable soft-prompt tokens, and fed into a frozen LLaVA-1.5 model; only the soft prompts are updated.","core_discovery":"The paper claims that long videos can be reduced to a few hundred discrete tokens, called visual event sentences, and that reading those tokens with a frozen large vision-language model is enough to match or beat specialized action-recognition models. On NTU RGB+D X-Sub it reports 94.1%, on X-View 96.8%, and on NTU RGB+D 120 X-Sub 87.0% and X-Set 88.5%. The gain is attributed to the event mapping itself: without conceptual quantization accuracy falls from 94.1% to 91.5%, removing adaptive temporal pooling drops it to 92.8%, and dropping the event-coherence bias costs 0.8 points. The authors also report that prompt tuning matches full fine-tuning (94.1% vs 94.0%) while training only about 1.2","pith_inferences":["Editorial inconsistency check: the claim of consistently state-of-the-art is not supported by the paper's own Table I on NTU RGB+D 120 X-Set, where PoseC3D is listed at 89.7% and VT-LVLM-AR at 88.5%; the text does not comment on this exception.","Editorial extension: the paper does not ablate the choice of LVLM itself; swapping LLaVA for a smaller LVLM or a non-reasoning classifier on the same tokens would reveal whether the gain comes from LLM-scale reasoning or from the tokenization.","Editorial extension: the reported margins over several baselines are small (0.4 to 0.8 points on some protocols), and the baseline set mixes skeleton and RGB methods; repeated-seed variance and a fine-grained long-video benchmark would strengthen the attribution.","Editorial extension: the coherence-loss ablation has the smallest effect in classification (0.8 points), so a more direct test of the visual event sentence idea would be to evaluate the same tokens on captioning or retrieval, where coherence should matter more."],"forward_implications":["If the result holds, action recognition no longer requires training a specialized video classification head; a frozen general-purpose vision-language model can classify actions from compact event tokens.","The same event-sentence interface could be reused for other video-language tasks, such as captioning or retrieval, by swapping the natural-language instruction.","The frozen-backbone plus prompt-tuning setup gives a practical parameter budget of about 1.2 million trainable parameters for adapting a multi-billion-parameter LVLM to video tasks.","The ablation identifies conceptual quantization as the most important ingredient, suggesting that the discrete event-token representation, rather than expensive full fine-tuning, drives the accuracy.","Performance peaks at 256 event tokens and a codebook of 2048; both over-compression and over-large vocabularies hurt, making event-sentence length and codebook size tunable hyperparameters."],"supporting_citations":[{"why":"Supplies the NTU RGB+D (NTU-60) benchmark and its X-Sub/X-View evaluation protocols, which carry all main accuracy comparisons.","marker":"[10]"},{"why":"Supplies the NTU RGB+D 120 benchmark and its X-Sub/X-Set protocols used in the second set of experiments.","marker":"[1]"},{"why":"The paper cites this as the source of the frozen LLaVA-1.5 vision-language backbone that receives the visual event sentences.","marker":"[8]"},{"why":"Supplies the P-Tuning v2-style soft-prompt method used to adapt the frozen LVLM with only 16 learnable prompt tokens.","marker":"[9]"}],"fun_headline_variants":["Frozen LLaVA + event tokens: 94.1% action accuracy","Video events as sentences: frozen LLaVA hits 94.1% on NTU","Long video to event tokens, frozen LLaVA reads them: 94.1%","Only 1.2M params: prompt-tuned LLaVA achieves 94.1% on NTU","VTEM compresses long videos into events; LLaVA decodes: 94.1%"],"cache_read_input_tokens":14848,"weakest_assumption_plain":"The load-bearing premise is that the mapper's 256 discrete 1024-dimensional event tokens are directly accepted as visual input by the frozen LLaVA-1.5 model; the paper (Section III-B) never specifies the projection or interface, so if the tokens must pass through a learned projection or be embedded as image patches, the frozen-backbone claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["Frozen LLaVA + event tokens: 94.1% action accuracy","Video events as sentences: frozen LLaVA hits 94.1% on NTU","Long video to event tokens, frozen LLaVA reads them: 94.1%","Only 1.2M params: prompt-tuned LLaVA achieves 94.1% on NTU","VTEM compresses long videos into events; LLaVA decodes: 94.1%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000483,"raw_usage":{"total_tokens":2304,"prompt_tokens":908,"completion_tokens":1396,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":1275}},"tokens_in":652,"tokens_out":1396,"duration_ms":13388,"temperature":1.0,"reasoning_tokens":1275,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T17:39:49.840522+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Freeze every LLaVA-1.5 parameter, including the vision encoder and visual projector, train only the 16 soft prompts, and reproduce the 94.1% NTU RGB+D X-Sub result while logging the tensor path from the 1024-dimensional event tokens into the model. If a learned linear projection or image-patch embedding is required and updated, the frozen-core premise fails; if replacing LLaVA with a non-reasoning classifier on the same event tokens still gives about 94% accuracy, then the claimed contribution of LVLM reasoning is falsified.","supporting_citations":[{"cited_title":"NTU RGB+D: A large scale dataset for 3d human activity analysis,","cited_arxiv_id":null,"evidence_quote":"Supplies the NTU RGB+D (NTU-60) benchmark and its X-Sub/X-View evaluation protocols, which carry all main accuracy comparisons."},{"cited_title":"Human action recognition from various data modalities: A review,","cited_arxiv_id":null,"evidence_quote":"Supplies the NTU RGB+D 120 benchmark and its X-Sub/X-Set protocols used in the second set of experiments."},{"cited_title":"Llava-more: A comparative study of llms and visual backbones for enhanced visual instruction tuning,","cited_arxiv_id":null,"evidence_quote":"The paper cites this as the source of the frozen LLaVA-1.5 vision-language backbone that receives the visual event sentences."},{"cited_title":"Adaptive prompt: Unlocking the power of visual prompt tuning,","cited_arxiv_id":null,"evidence_quote":"Supplies the P-Tuning v2-style soft-prompt method used to adapt the frozen LVLM with only 16 learnable prompt tokens."}],"review_version":1}