{"id":"788c8374-4b2a-43db-a1dc-abf2f21a5d4e","arxiv_id":"2412.09230","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A VideoQA pipeline using question-guided frame selection, MiniGPT-4 object grounding, and local-global graph fusion reports improved accuracy on five of six benchmarks, with one dataset contradicting the all-task claim.","lead":"This paper presents LGQAVE, a video question answering model that selects question-relevant frames, builds object graphs with MiniGPT-4, and combines local and global video features to generate answers. The authors report accuracy gains over prior systems on most benchmarks, but their own table contradicts the claim of improvement on every dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim rests on an unvalidated MiniGPT-4 grounding step: if it cannot return accurate, parseable question-relevant boxes, the §3.2 object graphs and the C-4→C-5 gains collapse.","rationale":"The reader's conditional verdict is the right one. I read the architecture and Table 1 in good faith: the pipeline is coherent, but the central empirical claim is not fully supported. The most load-bearing point is indeed the MiniGPT-4 grounding step in §3.2. The graph nodes are exactly the objects MiniGPT-4 chooses to box, no later module re-introduces missed objects, and the ablation evidence (C-3, C-4, C-5 in Table 3) shows that most of the final accuracy gain is tied to this component. Yet the paper provides no parse-rate statistics, no bounding-box quality evaluation, no prompt specification, and no failure analysis. This is a correctness risk, not a stylistic preference: if MiniGPT-4 cannot reliably emit parseable, question-relevant boxes, the method as described cannot produce the reported graphs. Separately, the 'surpasses previous SOTAs on all tasks' statement is internally inconsistent with Table 1 on MSRVTT-QA and ActivityNet-QA, which strengthens the need for protocol clarification or corrected numbers. No formal verification or code is provided, so reproducibility is also unresolved. Because the reader already asks for exactly these conditions (corrected table, code, and grounding evidence), I would keep the verdict as CONDITIONAL rather than escalate or downgrade.","tokens_in":14311,"tokens_out":5366,"duration_ms":55680,"concrete_test":"Implement the exact §3.2 protocol (same MiniGPT-4 checkpoint, prompt, ±2 frame context) and run it on 500 stratified (frame, question) examples from NExT-QA and STAR-QA. Measure (i) parse success rate for 4-coordinate bounding boxes and (ii) recall@IoU=0.5 of question-relevant objects against human-annotated objects or CoVGT's object proposals. Then rerun Table 3 C-3/C-5 with MiniGPT boxes replaced by proposals from an off-the-shelf object detector. If parse success is below 90%, or recall is below 70%, or detector boxes match MiniGPT boxes in accuracy, the claimed 9.29%/6.61% margin cannot be attributed to question-aware grounding.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 makes the entire object-graph pathway depend on MiniGPT-4's visual grounding: bounding boxes B^{t'}_i become the m+1 graph nodes, Eq. 3 propagates only those nodes, and Q-DGT (Eqs. 4-5) cannot recover objects that were never boxed. The paper cites MiniGPT-4 [56], a dialogue-oriented VLM, but gives no detection or grounding head, prompt template, confidence threshold, or grounding accuracy. This is not a cosmetic omission: Table 3 shows C-4 (graphs without MiniGPT grounding, using all-frame objects) drops Acc@All from 66.69 to 58.13, and C-3 vs C-5 attributes a further large gain to adding MiniGPT boxes with local features. Thus the claimed improvement is concentrated in the single least-validated component. Moreover, the §4.1 statement that LGQAVE 'surpasses previous SOTAs on all tasks' is contradicted by Table 1: LGQAVE scores 44.81 on MSRVTT-QA versus VideoLlava's 45.30, and 61.48 on ActivityNet-QA versus VideoLlava's 62.25, so the headline is overstated even before the grounding question is settled.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LGQAVE, a video question answering framework that combines three components: a cross-attention-based question-aware frame selection module operating on CLIP frame features and RoBERTa question embeddings; construction of spatial graphs from bounding boxes returned by a MiniGPT-4-based visual grounding step; and a question-conditioned dynamic graph transformer (Q-DGT) that produces local and global video representations refined by cross-attention and used for answer generation. The authors report results on NExT-QA, TGIF-FrameQA, MSRVTT-QA, ActivityNet-QA, STAR-QA, and Causal-VidQA, and claim in Section 4.1 that LGQAVE surpasses previous state-of-the-art methods on all tasks, with average gains of 9.29% over CoVGT and 6.61% over VideoLlava.","tokens_in":14592,"tokens_out":4111,"duration_ms":35807,"significance":"The paper addresses a relevant problem and proposes an interesting integration of question-guided frame selection, visual grounding, and graph-based reasoning for VideoQA. The inclusion of ablations (Table 3), comparisons with multiple baselines, and analysis of the beta and gamma parameters (Figure 3) is a strength. If the performance claims were fully supported, the work would be a useful contribution. However, the manuscript contains numerical inconsistencies that directly contradict the headline claim, and the most novel component, the MiniGPT-4 grounding step, is neither specified nor validated, so the significance of the reported improvements cannot be assessed from the presented evidence.","major_comments":[{"comment":"The claim that \"LGQA VE significantly surpasses the previous SOTAs on all tasks\" is contradicted by the paper's own Table 1: on MSRVTT-QA, LGQAVE scores 44.81, below VideoLlava's 45.30, and on ActivityNet-QA it scores 61.48, below VideoLlava's 62.25. The text should be revised to acknowledge these exceptions, and the reported average improvements of 9.29% and 6.61% need to be recomputed or justified with a precise definition of the averaging procedure.","section":"Section 4.1 / Table 1"},{"comment":"The entire object-graph pathway depends on MiniGPT-4's visual grounding to produce the bounding boxes that become graph nodes in Eq. 3, but the paper provides no details on how grounding is performed: no prompt template, confidence threshold, grounding head, or post-processing is described, and no grounding accuracy or failure analysis is reported. Since ablation C-4, which removes MiniGPT grounding, drops Acc@All from 66.69 to 58.13 on NExT-QA, the claimed gains are concentrated in this unvalidated component. The authors must supply implementation details and a quantitative validation of the grounding step, or the central performance claim cannot be supported.","section":"Section 3.2 / Table 3"},{"comment":"The claimed benefit of fine frame selection (FFS) is not visible in the reported numbers. The text states \"An average increase of 4.23% is observed with this method alone,\" but Table 2 shows HQGA+FFS at 49.40 Acc@All, which is below the HQGA baseline of 51.34 in Table 1, while CoVGT+FFS at 61.47 is above CoVGT's 57.40. This inconsistency undermines the conclusion that FFS helps existing graph-based methods and should be explained or corrected.","section":"Section 4.1 / Table 2"},{"comment":"The cross-attention score st is defined as Mean(Softmax(˜Et_i · ˜Q^⊤_i) · ˜Qi). Since ˜Et_i is N×C and ˜Qi is M×C, the product is N×M; after softmax and multiplication by ˜Qi, the result is N×C, and the mean over N yields a C-dimensional vector, not the scalar score used to compare frames. The reduction dimension and the intended scalarization need to be specified precisely.","section":"Section 3.1 / Eq. (2)"},{"comment":"The loss functions Lvqa and Lvq are used in Eqs. (11) and (12) but are never defined in the paper; the text says the formulation resembles prior work but does not give the exact functional form. Without these definitions, the training objective is not reproducible.","section":"Section 3.6 / Eqs. (11-12)"}],"minor_comments":[{"comment":"The abstract uses \"miniGPT model\" while the rest of the paper refers to \"MiniGPT-4\"; the naming should be consistent.","section":"Abstract"},{"comment":"The statement that including two preceding and two subsequent frames was \"fixed through empirical validation\" is not supported by any reported experiment; please provide the relevant ablation or remove the claim.","section":"Section 3.2"},{"comment":"The notation Q-DGTcm is introduced in Eqs. (6) and (7) without being defined in the text; please clarify what this cross-modal encoder is and how it differs from the Q-DGT module described earlier.","section":"Section 3.3"},{"comment":"The ActivityNet-QA value for VGT (PT) is reported as 3.70, which appears to be a typographical error; please verify and correct this entry.","section":"Table 1"},{"comment":"The computational cost figures (289 GFlops training, 138 GFlops testing) are reported without context or comparison to the baselines; please provide a comparison or state the hardware and batch size used.","section":"Section 4.1"},{"comment":"Reference [46] lists three authors in a single string; the author list should be separated properly.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely a student-led project. The central claim is overstated and the MiniGPT-4 grounding component needs substantial validation. The issues are fixable within the manuscript's scope, but the revision must include corrected tables, a clarified comparison protocol, and a rigorous description or ablation of the grounding step. If these cannot be provided, the paper should not be accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read on arXiv:2412.09230. The architecture is a sensible combination: question-conditioned cross-attention for frame selection, MiniGPT-4 as a grounding head for object boxes, and CoVGT's DGT as the graph engine. The frame selection module is clean and easily reusable, and the ablation (C-1 through C-5) is genuinely informative—it shows each component pulling its weight, especially the jump when local representations are added on top of grounding.\n\nThe problems are where the paper overclaims. The text says LGQAVE 'significantly surpasses previous SOTAs on all tasks,' but Table 1 shows it behind VideoLlava on MSRVTT-QA (44.81 vs 45.30) and ActivityNet-QA (61.48 vs 62.25). That is a direct internal contradiction, not a minor formatting issue. Table 2 is also confusing: HQGA+FFS gets 49.40, which is below the HQGA baseline of 51.34 in Table 1, so the frame selection module appears to hurt HQGA—the paper doesn't explain this. The stress-test on MiniGPT grounding lands. Section 3.2 provides no prompt template, confidence threshold, or grounding accuracy numbers. The entire object-graph pathway depends on boxes that are never validated, and the ablation C-4 vs C-5 (58.13 vs 66.69) shows just how much rides on that unvalidated component. Without code or error bars, the reported gains are hard to assess.\n\nThat said, this isn't a careless paper. The math is coherent, the citation pattern is honest (they credit CoVGT and [33] appropriately), and the idea of using a VLM for question-guided grounding is a reasonable direction even if the execution is incomplete. The soft spots are empirical, not conceptual. I'd send this to peer review—the architecture is plausible and the community would benefit from a careful check of the grounding step and the benchmark numbers. But it needs major revision: fix the table inconsistencies, provide grounding validation, release code and error bars, and tone down the SOTA claim to match the data.","headline":"Plausible architecture, but the SOTA claim is contradicted by the paper's own tables and the load-bearing MiniGPT grounding step is never validated.","tokens_in":15176,"tokens_out":1668,"would_cite":false,"duration_ms":16059,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LGQAVE achieves new state-of-the-art VideoQA accuracy by grounding object graphs in the question, averaging 9.29% over the strongest non-LLM baseline and 6.61% over the strongest LLM baseline.","keywords":["video question answering","question-aware frame selection","visual grounding","dynamic graph transformer","local-global video representation","cross-attention","foundation models","video reasoning"],"falsifier":"Take a subset of NExT-QA frames with human-annotated ground-truth boxes for the objects named in each question, compute the per-question recall of MiniGPT-4's bounding boxes, and check whether accuracy on questions with full grounding recall is noticeably higher than on questions with missed objects; if it is not, the grounding stage is not doing the causal work the paper attributes to it.","tokens_in":14116,"feed_emoji":"🎬","tokens_out":9595,"duration_ms":83195,"temperature":0.7,"pith_summary":"This paper argues that video question answering works better when every modeling choice is driven by the question itself rather than by generic video features. The proposed LGQAVE system picks only the frames a question is relevant to, asks MiniGPT-4 to box the objects the question depends on, and builds per-frame spatial graphs from those boxes. A question-conditioned dynamic graph transformer then produces both frame-level local representations and a global video representation, and a cross-attention fusion step refines the global view with local detail before a language model produces the answer. On NExT-QA, TGIF-FrameQA, MSRVTT-QA, ActivityNet-QA, STAR-QA, and Causal-VidQA, the authors report average accuracy gains of 9.29% over CoVGT and 6.61% over VideoLlava. If the results hold, question-conditioned selection is a practical way to remove redundant video content while sharpening answers to both multiple-choice and open-ended questions.","feed_headline":"Question-aware frame picking lifts video QA accuracy by 9 points","feed_subtitle":"Graphs of question-relevant objects plus local-global fusion beat LLM-only video models on six benchmarks.","key_machinery":"The load-bearing mechanism is a cascade of three question-conditioned modules. First, a learnable cross-attention between projected CLIP frame features and RoBERTa question features computes a relevance score $s_t$ for each frame, and only frames whose score exceeds a threshold $\\beta$ are kept for further processing. Second, MiniGPT-4 performs visual grounding on each selected frame together with its two temporal neighbors on either side, returning up to $m \\leq 10$ bounding boxes around question-relevant objects; these boxes become the nodes of a frame-specific spatial graph whose edge weights are learned from appearance and spatial features. Third, the question-aware dynamic graph transformer (Q-DGT) processes the graphs with a masked question embedding to emit per-frame local representations and a mean-pooled global representation, and a final cross-attention uses the local representations as keys and values to refine the global one with mixing weight $\\gamma$ before answer decoding.","core_discovery":"The central claim is that LGQAVE, the Local-Global Question Aware Video Embedding model, establishes a new state of the art in VideoQA by making frame selection, object selection, and representation fusion all question-aware. The authors show that a cross-attention score between question tokens and frame features identifies the frames worth keeping; that grounding those frames with MiniGPT-4 and building object interaction graphs yields richer local representations than using all detected objects; and that combining these local representations with a global video representation through cross-attention beats either representation used alone. The ablation on NExT-QA attributes steady gains to each component: full model Acc@All 66.69 versus 57.40 without sampling, MiniGPT-4 grounding, and local representations. The paper's own summary of the result is that LGQAVE surpasses previous state-of-the-art methods on all tested benchmarks, improving accuracy on average by 9.29% over non-LLM methods like CoVGT and 6.61% over LLM models like VideoLlava.","pith_inferences":["A testable extension beyond the paper is to use the frame-selection cross-attention scores as a temporal relevance map and check them against human key-frame annotations; if they align, the same module could serve video grounding and highlight generation.","The ceiling of the whole pipeline is likely set by the recall of MiniGPT-4's visual grounding, since an object it fails to box never enters the graphs; benchmarking per-question grounding recall would show where the method loses accuracy.","If the gains generalize to longer videos, question-conditioned frame selection could become the default front-end for other video-language tasks such as moment retrieval and video captioning, where most frames are also irrelevant to the target output.","One implicit claim worth testing is whether the average gain is a pure frame-sampling effect; rerunning the strongest baseline with the same sampled frames and boxes would isolate the contribution of the graph and local-global modules."],"forward_implications":["Because only question-relevant frames are processed, VideoQA systems can avoid the computational cost and redundancy of analyzing every frame while focusing attention where the question points.","Question-grounded object graphs let the model answer fine-grained causal and temporal questions that global-only representations miss, which is where the reported gains over CoVGT concentrate.","The frame-selection and grounding modules are separable: the authors show they improve HQGA and CoVGT when added to those architectures, so the same selection idea could be reused in other VideoQA pipelines.","Cross-attention fusion of local and global representations outperforms pooling or concatenating them, so the interaction pattern, not just the features themselves, contributes to the accuracy gain.","The pipeline does not rely on video captions or transcript-style supervision, so it can be applied to videos whose content is outside the distribution of caption-based training data."],"supporting_citations":[{"why":"Supplies the Q-DGT graph transformer module and the CoVGT baseline that LGQAVE extends and claims to beat by 9.29% on average.","marker":"[44]"},{"why":"Provides MiniGPT-4, whose question-guided visual grounding produces the bounding boxes that seed the object graphs.","marker":"[56]"},{"why":"VideoLlava is the strongest LLM-based baseline the paper compares against, claiming a 6.61% average improvement over it.","marker":"[23]"},{"why":"Video-LLaMA is another LLM-based VideoQA baseline in the main comparison table that LGQAVE outperforms.","marker":"[51]"},{"why":"NExT-QA is the primary benchmark used for the main results and the full component ablation study.","marker":"[40]"},{"why":"TGIF-FrameQA is one of the six evaluation benchmarks on which the state-of-the-art accuracy claim rests.","marker":"[14]"},{"why":"MSRVTT-QA is one of the six evaluation benchmarks used to support the open-ended VideoQA accuracy claim.","marker":"[45]"},{"why":"ActivityNet-QA is one of the six evaluation benchmarks used to support the open-ended VideoQA accuracy claim.","marker":"[4]"},{"why":"STAR-QA is one of the six evaluation benchmarks used to support the claim on situated reasoning questions.","marker":"[38]"},{"why":"Causal-VidQA is one of the six evaluation benchmarks used to support the claim on causal and commonsense reasoning questions.","marker":"[20]"}],"fun_headline_variants":["Local-global question-aware embedding wins video QA across six benchmarks","Question-guided frame selection and object graphs lift video QA","Cross-attention and object graphs fuse question-aware video representations","LGQAVE: question-aware fusion beats LLM-only video QA","Query-aware frame picking and object graphs boost video QA by 9%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes that MiniGPT-4 reliably finds and boxes every object the question depends on; any object it misses never enters the object graphs and cannot be recovered at a later stage.","fun_headline_variants_meta":{"raw":{"variants":["Local-global question-aware embedding wins video QA across six benchmarks","Question-guided frame selection and object graphs lift video QA","Cross-attention and object graphs fuse question-aware video representations","LGQAVE: question-aware fusion beats LLM-only video QA","Query-aware frame picking and object graphs boost video QA by 9%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001833,"raw_usage":{"total_tokens":7207,"prompt_tokens":946,"completion_tokens":6261,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":6175}},"tokens_in":562,"tokens_out":6261,"duration_ms":42960,"temperature":1.0,"reasoning_tokens":6175,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:11:53.112722+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a subset of NExT-QA frames with human-annotated ground-truth boxes for the objects named in each question, compute the per-question recall of MiniGPT-4's bounding boxes, and check whether accuracy on questions with full grounding recall is noticeably higher than on questions with missed objects; if it is not, the grounding stage is not doing the causal work the paper attributes to it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Q-DGT graph transformer module and the CoVGT baseline that LGQAVE extends and claims to beat by 9.29% on average."},{"cited_title":"Next-qa: Next phase of question-answering to explaining temporal actions","cited_arxiv_id":null,"evidence_quote":"NExT-QA is the primary benchmark used for the main results and the full component ablation study."},{"cited_title":"Tgif-qa: Toward spatio-temporal reasoning in visual question answering","cited_arxiv_id":null,"evidence_quote":"TGIF-FrameQA is one of the six evaluation benchmarks on which the state-of-the-art accuracy claim rests."},{"cited_title":"Video question answer- ing via gradually refined attention over appearance and mo- tion","cited_arxiv_id":null,"evidence_quote":"MSRVTT-QA is one of the six evaluation benchmarks used to support the open-ended VideoQA accuracy claim."},{"cited_title":"Activitynet: A large-scale video benchmark for human activity understanding","cited_arxiv_id":null,"evidence_quote":"ActivityNet-QA is one of the six evaluation benchmarks used to support the open-ended VideoQA accuracy claim."},{"cited_title":"From representa- tion to reasoning: Towards both evidence and commonsense reasoning for video question-answering","cited_arxiv_id":null,"evidence_quote":"Causal-VidQA is one of the six evaluation benchmarks used to support the claim on causal and commonsense reasoning questions."}],"review_version":1}