{"id":"172eecc2-6f82-4751-855f-977a6b590fec","arxiv_id":"2608.07932","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A proposal-aided interleaved grounding framework with action-aware supervision reports SOTA accuracy on dense sports video QA, evaluated on the authors' self-curated test sets.","lead":"SportsGrounder is a video reasoning system that combines an open-vocabulary object detector (OV-DINO) with a large multimodal model, interleaving bounding-box tokens and global video features frame by frame, and adding action-aware supervision and preference optimization. On self-curated sports QA datasets from SoccerNet and FineSports, it reports state-of-the-art accuracy, but the datasets and code are not released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AAS in §3.3 may be satisfied by reading the answer text from h_EOS, so the +2.4% action gain in Table 2 need not reflect visual motion grounding.","rationale":"The reader's concern about the self-curated QA benchmark is legitimate, but the single most load-bearing issue is internal to the proposed method. The AAS mechanism is the component explicitly designed to reduce language-bias over-reliance, and its validation rests entirely on the ablation in Table 2. Under the standard implementation of an autoregressive LMM, h_EOS is computed after the target answer has been seen, so the action-classification head can exploit the answer text. This makes the claimed +2.4% Action improvement and the overall 'learn accurate motion representations' conclusion unverifiable from the paper as written. The issue is not an ad hominem attack and does not require assuming bad faith; it is an ambiguity that a careful reader must resolve before accepting the mechanism. If the authors intended a pre-answer sentinel EOS, that is easily clarified and tested. The IGF architecture, the proposal-aided representation, and the reported SOTA numbers may all still hold, and the baselines are fairly described (same LoRA protocol, with and without prompt injection). Therefore the correct disposition remains a conditional acceptance, with the added condition that the AAS pathway must be shown to be blind to answer tokens. I set verdict_should_be to UNCHANGED because the overall verdict (conditional) does not change; the condition is more specific than the reader's dataset-bias condition.","tokens_in":16039,"tokens_out":7397,"duration_ms":82287,"concrete_test":"Re-run Stage 2 on SoccerNet with the AAS head receiving the hidden state of the last prompt token before Y_ans (i.e., mask answer-token attention for the AAS path) instead of h_EOS after the full answer. If the +2.4% Action accuracy gain over the IGF-only baseline in Table 2 shrinks or disappears, the reported AAS benefit came from answer-token leakage; if the gain persists, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The Action-Aware Supervision module is claimed to force the LMM to ground predictions in video motion, but as written it can be minimized without any visual grounding. In Stage 2 (Eq. 9), training uses the interleaved prefix X and the ground-truth answer sequence Y_ans; the AAS loss in Eq. 7 is computed from h_EOS, the hidden state at the EOS token after the full autoregressive forward pass. For action questions, the answer string almost always contains the action label, so the action head can predict it from the text answer alone. The paper's own remark that causal attention 'can naturally heavily weigh the adjacent textual inputs at the terminal tokens' names the exact shortcut, yet no mask, stop-gradient, or answer-token exclusion is described for the AAS pathway. If h_EOS is instead intended to be a sentinel EOS before answer generation, the text must say so and verify it; as written, the claimed mechanism for reducing language bias is not established, and the ablation gain attributed to AAS in Table 2 is confounded.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SportsGrounder augments InternVL3.5-2B with OV-DINO object proposals, fuses them frame-by-frame through Interleaved Grounding Fusion (IGF), adds an Action-Aware Supervision (AAS) loss on the EOS hidden state, and trains in three stages (representation alignment, LoRA-based SFT, and Mixed Preference Optimization). The authors programmatically generate 26k/24k four-option QA pairs from SoccerNet and FineSports and report 51.8% and 53.6% overall accuracy, outperforming LoRA-finetuned baselines including MiniCPM-V 4B.","tokens_in":16296,"tokens_out":3896,"duration_ms":44653,"significance":"If verified, this is a useful architectural contribution to dense sports video QA: the IGF mechanism is a clean solution to the sequence-length explosion and temporal misalignment problems that arise when injecting object-level groundings, and the use of a frozen open-vocabulary detector keeps the added cost modest. The paper is also clearly specified: the three-stage curriculum, losses, and fusion equations are explicit, and the ablation design is reasonable. However, the headline claim depends on a self-curated, unreleased benchmark, single-run results without uncertainty, and an AAS mechanism whose stated mechanism is not established by the experiments as written. These are load-bearing gaps, but they are fixable in revision.","major_comments":[{"comment":"The central state-of-the-art claim is evaluated only on QA pairs the authors generated from SoccerNet and FineSports metadata, and the resulting benchmark is not released. Because the test set is not public and the generation protocol is described only at a high level, the reader cannot verify that the questions measure fine-grained visual reasoning rather than metadata-derived textual regularities. Please release the data and generation scripts, or additionally evaluate on existing public benchmarks such as Sports-QA and SportU, and specify the exact templates, distractor sampling, and filtering used.","section":"§4.1 (Datasets and QA Construction)"},{"comment":"Top-K = 15, the action-supervision weight λ = 0.1, and the MPO weights are chosen 'empirically', but no validation split is described and all results appear to come from a single run. The key ablation gains (e.g., +2.4% Action for AAS and +4.4% Spatial for MPO) therefore have no stated uncertainty. Please report means and standard deviations across multiple seeds and specify the validation set used for hyperparameter selection.","section":"§4.1 (Implementation Details) and Table 2"},{"comment":"The AAS loss is computed from h_EOS after the full answer sequence has been generated, and the ground-truth answer text almost always contains the action label. Since causal attention at the terminal EOS position can attend to the preceding answer tokens, the action head can minimize Eq. 7 by reading the label from the text, so the claim that AAS 'forces' visual motion aggregation is not established. The paper itself acknowledges that causal attention can heavily weigh adjacent textual inputs at terminal tokens, but does not describe a stop-gradient, a masking of answer tokens, or an alternative placement of h_EOS before answer generation. Please clarify the exact token position used, exclude answer-token gradients if necessary, and add a text-only control (e.g., answers with visual input ablated) to show that the +2.4% Action gain in Table 2 reflects visual grounding rather than text copying.","section":"§3.3 (AAS, Eq. 7)"},{"comment":"The construction of the preference pairs for Mixed Preference Optimization is not described: how are the positive responses y_c and hard-negative responses y_r generated for the four-option QA tasks, and what exactly is the frozen reference model π_ref? Without this information, the Stage 3 gain in Table 2 is not reproducible, and the claim that MPO 'distinguishes deceptive distractors' cannot be separated from the specific choice of negatives. Please specify the negative sampling procedure and its relation to the distractor options in the generated QA pairs.","section":"§3.4 and §4.1 (MPO Stage 3)"}],"minor_comments":[{"comment":"The wording that AAS 'forces the network to learn accurate motion representations' is stronger than what a λ = 0.1 auxiliary loss can guarantee; consider softer phrasing such as 'encourages'.","section":"Abstract and §1"},{"comment":"The claimed efficiency benefit depends on the actual sequence length, but the values of T, N, and L_box are not reported for the two datasets; please include the average token counts and compute cost.","section":"§3.2, Eq. (5)"},{"comment":"Several baseline differences are small (e.g., 45.2 vs. 43.8 on SoccerNet), and no significance tests or confidence intervals are reported; with single runs, those differences may not be meaningful.","section":"Table 1"},{"comment":"The options for Task 3 Action Anticipation are printed in a non-alphabetical order (Fallback, Pass, Other, ToBasket), which is confusing in a multiple-choice question figure.","section":"Figure 4"},{"comment":"Please confirm that the MPO protocol described as 'natively introduced by our base model, InternVL3.5' is indeed the same as the MPO in reference [41], since that reference predates InternVL3.5 and may correspond to an earlier InternVL version.","section":"References [41] and [42]"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an ACM MM submission, and the main novelty claim hinges on a benchmark that is neither released nor externally validated. I would urge the editor to require data/code release or a demonstration on an existing public benchmark before accepting the state-of-the-art claim. The AAS mechanism, as written, has a plausible text-copying shortcut that the authors should be asked to close experimentally; this is a scientific correctness concern rather than a mere presentation issue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth a read if you work on grounded video QA: the IGF mechanism—interleaving OV-DINO proposals frame-by-frame with grid features—is a clean, practical answer to the sequence-explosion problem, and the prompt-injection baselines are a fair way to show that feeding boxes as text is not the same as fusing them as visual tokens. The benchmark curation is also reasonably careful; excluding coordinates from the question text is the right instinct.\n\nBut the stress-test concern about AAS is correct on the text as written. In Stage 2, h_EOS is extracted after the full autoregressive forward pass over the ground-truth answer sequence. For action questions, the action label is right there in the answer string, and nothing in Eq. 7 or the surrounding text stops the action head from reading it. The paper even acknowledges that causal attention \"can naturally heavily weigh the adjacent textual inputs at the terminal tokens,\" then asserts that backpropagating L_act through h_EOS mitigates it. That does not follow: backprop through a state that has already attended to the answer text can still be satisfied by encoding the answer label. The fix is straightforward—use the hidden state before answer tokens, or mask answer-token positions in the attention for the AAS head—but the paper doesn't describe it. So the +2.4% action gain in Table 2 is not evidence of visual motion grounding; it is evidence that the auxiliary head helps, possibly by a shallow route.\n\nThe other weaknesses are more conventional: no error bars, single runs, hyperparameters (K, lambda) chosen without a described validation split, and the benchmark is not released. These limit the strength of the \"state-of-the-art\" claim but are addressable in a revision.\n\nOverall, this is a solid engineering contribution with a plausible architecture and fair internal baselines. The central mechanism of one of the three modules is not established, but the work is coherent, clearly written, and the IGF integration is a useful data point for the field. I would send it to peer review, but I would ask for the AAS issue to be resolved—either by redefining h_EOS or by showing the action gain survives when the shortcut is blocked—and for error bars or a released benchmark.","headline":"The IGF architecture is a solid, well-specified subfield contribution, but the AAS module's claimed anti-language-bias mechanism is undercut by a likely shortcut, and the SOTA claim rests on an unpublished self-curated benchmark.","tokens_in":16774,"tokens_out":1731,"would_cite":true,"duration_ms":22008,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 2B model with interleaved object grounding and action supervision sets the top accuracy on dense sports video QA.","keywords":["Video Question Answering","Dense Sports Video Reasoning","Visual Grounding","Large Multimodal Models","Open-Vocabulary Detection","Interleaved Grounding Fusion","Preference Optimization","Action-Aware Supervision"],"falsifier":"Construct a text-only control by running the same QA pairs with the video frames removed or replaced by static gray frames while keeping the question text identical; if a text-only model approaches or matches the reported accuracy, the benchmark leaks non-visual cues. A second check: swap in a different action clip with identical player tracks and see whether answers track the visual change; if accuracy does not move, the model is still using language priors.","tokens_in":15876,"feed_emoji":"⚽","tokens_out":6203,"duration_ms":64048,"temperature":0.7,"pith_summary":"The paper argues that dense sports video question answering fails in current large multimodal models because grid features lose small, visually similar entities and the models fall back on language priors. To fix this, SportsGrounder adds an open-vocabulary detector that selects the top relevant objects per frame, fuses their bounding-box coordinates and semantic vectors into the visual sequence frame by frame, and supervises the model's hidden states so action queries are answered from motion rather than text. Trained with a mixed preference objective, the 2B-parameter model reaches 51.8% overall accuracy on SoccerNet and 53.6% on FineSports, beating larger baselines including a 4B model. The paper also introduces newly curated multiple-choice QA sets derived from both datasets. If right, object-centric grounding plus motion-level regularization is a cheaper route to fine-grained video reasoning than scaling parameters.","feed_headline":"2B sports-video QA model beats 4B rivals via object grounding","feed_subtitle":"Interleaving per-frame object boxes with global video features lifts accuracy to 51.8% on SoccerNet and 53.6% on FineSports.","key_machinery":"The central object is the Interleaved Grounding Fusion (IGF) mechanism, which builds the visual prefix as per-frame blocks of global grid tokens followed by a compact hybrid token per selected object: the object's normalized box coordinates tokenized through the LLM text embedding plus its OV-DINO semantic feature projected by an MLP. Because selection is restricted to the top 15 domain-relevant objects, sequence length stays bounded relative to naive concatenation of hundreds of detector queries. The Action-Aware Supervision (AAS) branch is the corrective mechanism: it maps the terminal hidden state to an action label through a linear head and back-propagates a masked cross-entropy loss only for action queries, preventing the model from treating the final state as pure language. Mixed Preference Optimization (MPO) is the decision-boundary calibrator: it optimizes preferred versus dispreferred answer sets, with DPO-style preference, quality, and generation losses, following the base model's training protocol.","core_discovery":"On its own terms, the paper's discovery is that interleaved, frame-aligned fusion of explicit box coordinates and implicit object semantics with global ViT features lets a small multimodal model reason about dense sports scenes accurately, while injecting boxes as text does not. Action-Aware Supervision forces the end-of-sequence hidden state to carry action information, countering language-prior guessing. Mixed Preference Optimization sharpens choice among hard-negative distractors. The result is the top accuracy on the paper's SoccerNet and FineSports QA benchmarks, including the fine-grained action, spatial reasoning, and dense disambiguation subtasks, with the action and spatial gains coming from the visual mechanism rather than from model scale.","pith_inferences":["The paper does not test for answer leakage in its generated QA pairs; a text-only or frame-scrambled control would establish whether the visual components, rather than question templates, drive the reported margins.","A natural next step the paper leaves implicit is applying IGF to other dense multi-agent scenes, such as traffic or surgical video, where small homogeneous subjects and fine actions dominate.","Because the Jersey Number subtask remains the one place a larger model wins, combining the object branch with higher-resolution crops or an OCR prober is a plausible extension the paper does not explore.","Deploying the method on raw broadcasts would need an action taxonomy or pseudo-labels, since AAS supervision depends on dataset action metadata."],"forward_implications":["A 2B-parameter LMM with this grounding architecture outperforms a 4B model on dense sports reasoning, suggesting parameter count is not the binding constraint for fine-grained video QA.","Feeding detector boxes to the model as text consistently hurts action accuracy; interleaving them as visual tokens preserves temporal attention and gives the better trade-off.","The AAS loss alone adds roughly 2.4 percentage points on action accuracy over the IGF-only baseline, so motion supervision is a separable, transferable component.","The MPO stage contributes most to spatial reasoning and team-comparison subtasks, so preference optimization is where relational accuracy is won.","Because IGF avoids sequence-length explosion, the architecture can be applied to longer videos and more frames per clip without quadratic attention blow-up."],"supporting_citations":[{"why":"SoccerNet supplies the dense-soccer action labels and timestamps used to build the SoccerNet QA pairs and the AAS action space.","marker":"[3]"},{"why":"OV-DINO is the open-vocabulary detector whose language-aware queries are filtered to top-K proposals.","marker":"[38]"},{"why":"Provides the Mixed Preference Optimization objective used in Stage 3.","marker":"[41]"},{"why":"InternVL3.5 is the base LMM and supplies the ViT global branch and the native MPO recipe.","marker":"[42]"},{"why":"FineSports supplies hierarchical multi-person basketball annotations used for the generalization benchmark.","marker":"[47]"},{"why":"DPO is the preference loss at the core of the MPO objective.","marker":"[30]"}],"fun_headline_variants":["Interleaved object grounding lifts sports video QA accuracy","Small model beats big rivals with frame-aligned object boxes","SportsGrounder: action-aware supervision halves language-bias guessing","Interleaved boxes + global features top sports video reasoning","Object-based grounding boosts dense sports QA without scale"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the programmatically generated multiple-choice questions from SoccerNet and FineSports genuinely test fine-grained visual reasoning; if the questions leak the answer through wording, template, or distractor patterns, the reported accuracy gains would not reflect visual understanding.","fun_headline_variants_meta":{"raw":{"variants":["Interleaved object grounding lifts sports video QA accuracy","Small model beats big rivals with frame-aligned object boxes","SportsGrounder: action-aware supervision halves language-bias guessing","Interleaved boxes + global features top sports video reasoning","Object-based grounding boosts dense sports QA without scale"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1413,"prompt_tokens":963,"completion_tokens":450,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":371}},"tokens_in":579,"tokens_out":450,"duration_ms":5829,"temperature":1.0,"reasoning_tokens":371,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:38:47.027940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a text-only control by running the same QA pairs with the video frames removed or replaced by static gray frames while keeping the question text identical; if a text-only model approaches or matches the reported accuracy, the benchmark leaks non-visual cues. A second check: swap in a different action clip with identical player tracks and see whether answers track the visual change; if accuracy does not move, the model is still using language priors.","supporting_citations":[{"cited_title":"Manning, Stefano Ermon, and Chelsea Finn","cited_arxiv_id":null,"evidence_quote":"DPO is the preference loss at the core of the MPO objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SoccerNet supplies the dense-soccer action labels and timestamps used to build the SoccerNet QA pairs and the AAS action space."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"FineSports supplies hierarchical multi-person basketball annotations used for the generalization benchmark."}],"review_version":1}