{"id":"8af117cb-a0d2-4ae6-b049-14b5e587a421","arxiv_id":"2507.07744","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SDST is a parameter-efficient, anchor-free side-tuning architecture for video temporal grounding that matches or beats state-of-the-art methods with about 73% fewer trainable parameters.","lead":"This paper introduces SDST, a lightweight side module that lets video grounding models use internal video-text features without retraining the large backbone. It reports competitive or better benchmark results while using roughly 73% fewer trainable parameters than the current state of the art.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 and Table B report incompatible QVHighlights val scores for Ours (R1@0.7 60.90 vs 56.76±0.53); the SOTA claim rests on which number is canonical.","rationale":"The reader's weakest assumption concerns the frozen AdaptivePool reuse in Eq. 15 and the resulting depth-pooling trade-off. That is a plausible technical risk, but the paper directly ablate pooling strategies (Table 3) and acknowledges the trade-off in Sec. 6.1.3, so even if the pooling transfer is suboptimal, the main result does not necessarily collapse. In contrast, I found a concrete internal inconsistency between the headline results (Table 1) and the multi-seed fair comparison (Table B) for the same method and backbone on the same split. The R1@0.7 value differs by 4.14 points, far beyond any plausible seed spread given the reported standard error. This is not a matter of disagreement with community consensus; it is a correctness-of-reporting issue. The central claim that SDST is statistically equivalent to SG-DETR while using 27% of its parameters is directly falsified if Table B is the accurate set of numbers. Consequently, the paper cannot be accepted as-is; it needs either a correction of Table B or a reproduction of Table 1's values with multiple seeds and a clear statement of selection protocol (best seed, ensemble, or post-hoc choice). I keep the verdict CONDITIONAL because the issue is empirically resolvable and the code is public, but the condition should be explicitly tied to resolving this discrepancy, not merely to adding error bars or PEFT baselines. I disagree with the reader's choice of weakest assumption because the AdaptivePool concern, while legitimate, is less decisive for the paper's main claim than the Table 1/Table B mismatch.","tokens_in":26849,"tokens_out":7986,"duration_ms":75545,"concrete_test":"Run the official released code (github.com/davidpujol/SDST) on QVHighlights val with the exact Table 1 configuration, across 3 independent seeds, and report mean±std for R1@0.5, R1@0.7, mAP, HD mAP, HIT@1. If the mean R1@0.7 is approximately 60.9, then Table B is outdated or computed under a different protocol and must be corrected. If the mean R1@0.7 is approximately 56.8, then Table 1's single-run numbers are not representative, and the paper's SOTA-equivalence claim over SG-DETR (57.91) should be downgraded to 'competitive but below SOTA on MR.' The test also checks whether Section M's Nemenyi conclusion survives when using the multi-seed numbers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The most load-bearing issue is an internal inconsistency in the reported QVHighlights val numbers. Table 1 reports Ours† val R1@0.7 = 60.90 and mAP = 55.60, but Section E, Table B, which evaluates the same model on 3 seeds with InternVideo2-1B features, reports Ours R1@0.7 = 56.76 ± 0.53 and mAP = 55.31 ± 0.23. The R1@0.7 gap is 4.14, which is roughly 13 standard errors of the mean (0.53/sqrt(3) ≈ 0.306), so it cannot be explained by seed variance. For SG-DETR the two tables agree exactly (57.91), so the discrepancy is specific to Ours. The central claim of 'SOTA or highly competitive performance' depends entirely on which set of numbers is taken as canonical. Using Table B, SDST is below SG-DETR on all three MR metrics (R1@0.5 73.20 vs 73.52, R1@0.7 56.76 vs 57.91, mAP 55.31 vs 55.64), so the claims that SDST 'surpasses on several metrics' and is 'statistically on par' are no longer supported. Since Table 1 is the headline result, this is a load-bearing reproducibility issue, not a stylistic point. The reader's identified AdaptivePool concern is real but secondary; even if pooling transfer is imperfect, it is explicitly ablated and acknowledged, whereas the Table 1/Table B mismatch directly determines whether the main performance claim holds.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SDST, a side-tuning (ST) architecture for video temporal grounding (VTG) that combines a dense stream for highlight detection (HD) with a sparse stream for moment retrieval (MR). Three contributions are claimed: (1) the first anchor-free ST architecture for VTG, based on recurrent decoder queries; (2) a new Reference-based Deformable Self-Attention (RDSA) mechanism that addresses a context limitation of standard deformable cross-attention; and (3) the first effective integration of the InternVideo2 backbone into an ST framework, via reusing its frozen AdaptivePool module across intermediate layers. Experiments on QVHighlights, TACoS, and Charades-STA report highly competitive or state-of-the-art results with a parameter count of 4.1M, roughly 27% of SG-DETR's 15M parameters.","tokens_in":27201,"tokens_out":7229,"duration_ms":70491,"significance":"If the reported numbers are reproducible, the paper makes a useful contribution to parameter- and memory-efficient VTG. The efficiency claim is substantial and well supported by the parameter counts and memory measurements in Tables 1, D, and H. The RDSA idea is technically interesting and is backed by a careful ablation (Tables 5, M, N) and an offset analysis (Fig. 4). The paper also ships public code and includes an extensive ablation study, a statistical significance analysis, and a discussion of a previously underappreciated depth-pooling trade-off. These strengths make the work of potential interest to the VTG and efficient-transfer-learning communities, provided the main results are consistent across the paper.","major_comments":[{"comment":"The QVHighlights val results for the proposed method are mutually inconsistent between the main text and the supplementary material. Table 1 reports Ours† with R1@0.5 = 73.68, R1@0.7 = 60.90, and mAP = 55.60. Table B, which evaluates the same model on three seeds with InternVideo2-1B features, reports R1@0.5 = 73.20 ± 0.23, R1@0.7 = 56.76 ± 0.53, and mAP = 55.31 ± 0.23. The R1@0.7 gap of 4.14 is over ten times the reported standard error of the mean (0.53/√3 ≈ 0.31) and cannot be explained by seed variation. The SG-DETR row matches exactly between the two tables (57.91), so the inconsistency is specific to the proposed method. The central claims that the method 'surpasses on several metrics' and is 'statistically on par' with SG-DETR rely directly on the Table 1 numbers. The authors must clarify which numbers are canonical, reconcile the two tables, and re-evaluate the conclusions if the lower numbers from Table B are the correct ones.","section":"§5.2 (Table 1) vs. §E (Table B)"},{"comment":"The Friedman test section contains an inverted statistical statement. The text reports χ²_F = 5.640 with p = 0.933 and then states 'Hence, we can reject the null hypothesis, and conclude that there is no significant difference.' With p > 0.05, the correct conclusion is that the null hypothesis of no difference cannot be rejected. As written, the sentence is contradictory: rejecting the null would imply the opposite of the stated conclusion. This error undermines the credibility of the statistical analysis and should be corrected. The Nemenyi test in §M.2 appears to be computed correctly, but the Friedman test explanation needs to be fixed.","section":"§M.1"}],"minor_comments":[{"comment":"The sentence 'which is our goal in this this work' contains a duplicated word 'this' and should be corrected.","section":"§3.3.4"},{"comment":"The main results table carries no error bars, while the supplementary includes mean±std over three seeds. Given the large discrepancy between the two, the authors should report error bars in the main text or add a clear footnote explaining which run the main-table numbers correspond to.","section":"Table 1 and Table B"},{"comment":"The comparison with other PEFT methods is limited to side-tuning baselines; adapter-, LoRA-, and prompt-based methods are not evaluated due to computational constraints. The paper would be strengthened by citing existing experimental comparisons against such methods (e.g., from R2-Tuning) to justify the claim that ST methods are competitive.","section":"§H"},{"comment":"The spelling 'R 2-Tuning' is used inconsistently; the paper should use a single consistent form, preferably matching the reference title 'R2-Tuning'.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper fits the scope of the journal and addresses a relevant efficiency problem. The main concern is reproducibility: the QVHighlights val numbers in Table 1 and Table B are incompatible, and this directly affects the paper's headline claims. The authors should be asked to verify and reconcile these numbers, and to correct the Friedman test interpretation, before the manuscript can be considered for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis is a real contribution with a reporting problem that needs to be fixed before the SOTA claim can be trusted. The new pieces are worth knowing: SDST is the first anchor-free side-tuning model for VTG, RDSA sensibly reformulates deformable cross-attention as reference-based self-attention, and the InternVideo2 integration into a side-tuner is the first that works. The ablations are extensive — pooling, refinement depth, attention variants, module order, loss weights — and the code is public.\n\nThe load-bearing issue: Table 1 gives QVHighlights val R1@0.7 = 60.90 for SDST, while Sec. E, Table B, gives the same model/features as 56.76 ± 0.53. That gap (~4 points) is roughly 13 standard errors, so seed variance won't explain it. SG-DETR's numbers match across both tables, so the inconsistency is specific to SDST. If Table B is canonical, SDST is below SG-DETR on all three MR metrics on QVHighlights val, and the abstract's \"highly competitive or SOTA\" claim rests on a number that is contradicted inside the paper. The Nemenyi \"statistically on par\" conclusion can coexist with Table B, but the main text's \"surpasses on several metrics\" cannot.\n\nThere are two smaller soft spots. The Friedman test write-up in Sec. M.1 is inverted: p=0.933 means you fail to reject the null, so there is no significant ranking difference; the text says \"we can reject the null hypothesis\" right before drawing the correct conclusion. And the PEFT comparison omits adapters, LoRA, and prompt methods under the same backbone; the memory rationale is legitimate, but it means the comparison to those families is inherited from prior work, not shown here.\n\nNone of this kills the architecture. RDSA is a good idea, the efficiency story is plausible, and the paper is generally honest about its trade-offs. But the main table must be reconciled with Table B before the central claim is taken at face value. I would send it to peer review, with a clear request to resolve the val-number discrepancy and fix the statistical text. If Table 1 was a lucky seed, it should be replaced by the multi-seed mean; if Table B is a typo, that needs to be shown. The authors are capable of fixing it.\n\nFor the right reader — anyone doing efficient video grounding or PEFT — this is useful material, and the reading group could learn a lot from the discrepancy itself.","headline":"A genuinely useful side-tuning architecture, but the headline QVHighlights val numbers are internally inconsistent, so the SOTA claim is currently unverifiable.","tokens_in":27724,"tokens_out":6261,"would_cite":true,"duration_ms":58550,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new sparse-dense side-tuner for video temporal grounding, SDST, is anchor-free and matches or beats state-of-the-art DETR-style methods on QVHighlights, TACoS, and Charades-STA while using only about 27% of the parameters.","keywords":["Video temporal grounding","Moment retrieval","Highlight detection","Side-tuning","Deformable attention","Parameter-efficient fine-tuning","InternVideo2","Recurrent decoder queries"],"falsifier":"Take the trained SDST and replace the frozen AdaptivePool with a per-layer learned pooling module (or with no pooling reuse at all, e.g., average pooling) on the same three benchmarks; if the resulting accuracy does not drop below the reported numbers, the claim that AdaptivePool reuse is the enabling choice would be falsified. A more direct check is to measure the feature distribution distance between intermediate-layer pooled outputs and last-layer pooled outputs and test whether the performance gap across the paper's pooling ablation correlates with that distance.","tokens_in":26614,"feed_emoji":"🎯","tokens_out":8860,"duration_ms":82870,"temperature":0.7,"pith_summary":"Video temporal grounding (VTG) asks a model to find the moments a text query describes in a video, both by retrieving the time interval (moment retrieval, MR) and by scoring each frame's relevance (highlight detection, HD). Most competitive models use DETR-style decoders over final-layer features of a frozen image-text backbone, which limits adaptation to video domains and is parameter-heavy. The paper introduces SDST, a side-tuning architecture that is anchor-free—the first for VTG—and that refines frozen intermediate features from InternVideo2 through a shared dual stream: a dense stream for frame-level highlights and a sparse stream of learnable recurrent moment queries. It claims SDST beats prior side-tuning methods and matches or surpasses state-of-the-art DETR methods on QVHighlights, TACoS, and Charades-STA while using about 27% of the parameters of the strongest competitor. The central insight is that deformable attention only works naturally as self-attention, not cross-attention, because cross-attention queries lack local context of the key space—a limitation fixed by the paper's Reference-based Deformable Self-Attention.","feed_headline":"At 27% of parameters, it matches top video grounding","feed_subtitle":"A sparse-dense side-tuner with reference-based attention localizes moments and highlights from queries.","key_machinery":"Reference-based Deformable Self-Attention (RDSA), a deformable-attention variant that uses reference-derived query embeddings instead of the raw decoder queries. For each moment proposal, a learned CNN gives the dense frame embeddings local context, then bilinear sampling extracts three action embeddings—left-most, center, right-most—which are concatenated and linearly projected to predict the deformable offsets and attention scores. Because these query embeddings are derived from the same dense features as the keys, the query and key spaces coincide, turning the deformable cross-attention into a deformable self-attention and providing the local context the standard mechanism lacks. The paper's other load-bearing mechanism is the cross-layer reuse of InternVideo2's frozen AdaptivePool module: the same last-layer pooling function is applied at every sampled intermediate layer, avoiding backprop and extra memory at the cost of a distribution shift between intermediate and last-layer features, a trade-off the authors call the 'depth-pooling trade-off'.","core_discovery":"The paper's central claim is that a parameter- and memory-efficient side-tuner can reach the accuracy of conventional DETR-style VTG systems. SDST recursively applies a weight-shared dual-stream module to K intermediate visual and textual embeddings of a frozen InternVideo2 backbone. The dense stream produces frame-level embeddings for highlight detection, while the sparse stream maintains M learnable center-width moment proposals with latent embeddings, refined across levels for moment retrieval. The key discovery is diagnostic: in the deformable attention used by anchor-free methods, the query is fully independent of the key/value space, so when used as cross-attention its offset predictions collapse near initialization and cannot look beyond the current estimated boundaries. SDST replaces the queries with bilinearly sampled left-most, center, and right-most action embeddings extracted from a CNN-refined dense embedding, converting deformable cross-attention into deformable self-attention while keeping the efficiency of sparse key sampling. The paper also shows that InternVideo2's frozen AdaptivePool—trained only for the last layer—can be reused across intermediate layers to pool spatio-temporal tokens, and that this pooling choice is worth several points over CLS pooling. With these pieces, SDST reports highly competitive or state-of-the-art results on QVHighlights, TACoS, and Charades-STA at 4.1M trainable parameters, about 27% of the parameters of the SG-DETR baseline.","pith_inferences":["The paper's diagnosis of deformable attention—that it is naturally suited to self-attention and loses context in cross-attention—is a generic mechanism claim, not limited to VTG; a direct test is to swap deformable cross-attention for an RDSA-style reference-derived self-attention in an image detection decoder and measure whether offset collapse disappears.","The depth-pooling trade-off points to a concrete follow-up: distilling the last-layer AdaptivePool into lightweight per-layer pooling heads on a small proxy set could remove the distribution shift and let SDST exploit shallower layers without full backpropagation.","Because the paper's Fig. 3 shows last-layer-only features with two or three refinement steps can beat intermediate features, part of the improvement attributed to 'intermediate features' may actually come from the number of recurrent refinement steps; separating these two variables is a natural ablation the paper does not fully isolate.","The authors leave cross-domain generalization for future work; a straightforward experiment would be training SDST on Charades-STA and testing on TACoS (and vice versa) to see whether the parameter savings and sparse-dense multitask training also improve robustness to domain shift."],"forward_implications":["On QVHighlights val, SDST improves the previous side-tuning method R2-Tuning by 3.82% average mAP on moment retrieval and 2.21% mAP on highlight detection, and is statistically indistinguishable from the SG-DETR state of the art while using about a quarter of its parameters.","On Charades-STA and TACoS, SDST sets a new state of the art: for example, R1@0.7 improves by 2.71% on Charades-STA and 2.39% on TACoS over the previous best results.","The paper shows RDSA's offsets move beyond the currently estimated moment boundaries, which makes it especially effective for long actions: mAP on long actions improves by 2.84% over the standard deformable cross-attention.","The dual-stream interaction is bidirectional in practice: conditioning the sparse stream on the dense stream's non-linear output improves both moment retrieval and highlight detection compared to conditioning on raw video features.","Reusing InternVideo2's frozen AdaptivePool across intermediate layers outperforms CLS pooling by 5.07% average mAP on MR and 7.74% HIT@1 on HD, establishing that pooling strategy is decisive for side-tuning with spatio-temporal backbones."],"supporting_citations":[{"why":"R2-Tuning is the prior side-tuning method for VTG that SDST builds on and outperforms; it supplies the anchor-based ST baseline.","marker":"[25]"},{"why":"Moment-DETR and QVHighlights provide the main benchmark dataset and the anchor-free DETR baseline for MR and HD.","marker":"[18]"},{"why":"Deformable DETR supplies the deformable attention mechanism whose cross-attention limitation RDSA addresses.","marker":"[50]"},{"why":"SG-DETR is the state-of-the-art anchor-free method with InternVideo2 features that SDST compares against for accuracy and parameter count.","marker":"[8]"},{"why":"InternVideo2 is the spatio-temporal backbone whose intermediate features and frozen AdaptivePool module the ST framework integrates.","marker":"[40]"},{"why":"DAB-DETR inspires the recurrent decoder queries with learnable center-width moment references used in the sparse stream.","marker":"[23]"},{"why":"DINO's decoder query initialization strategies are used in the ablation that motivates RDSA over initialization-based fixes.","marker":"[48]"}],"fun_headline_variants":["Video grounding with 73% fewer parameters, SOTA results","Side-tuner for VTG: 27% parameters, top accuracy","Reference-based deformable self-attention boosts VTG","Sparse-dense side-tuner matches SOTA at 27% params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy depends on reusing InternVideo2's frozen AdaptivePool, which was trained only for the last layer, on intermediate layers; if that pooling transfer causes a harmful distribution shift, the gains attributed to the backbone integration would shrink or vanish.","fun_headline_variants_meta":{"raw":{"variants":["Video grounding with 73% fewer parameters, SOTA results","Side-tuner for VTG: 27% parameters, top accuracy","Reference-based deformable self-attention boosts VTG","Sparse-dense side-tuner matches SOTA at 27% params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3589,"prompt_tokens":1065,"completion_tokens":2524,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":681,"completion_tokens_details":{"reasoning_tokens":2450}},"tokens_in":681,"tokens_out":2524,"duration_ms":20346,"temperature":1.0,"reasoning_tokens":2450,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:33:22.193039+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the trained SDST and replace the frozen AdaptivePool with a per-layer learned pooling module (or with no pooling reuse at all, e.g., average pooling) on the same three benchmarks; if the resulting accuracy does not drop below the reported numbers, the claim that AdaptivePool reuse is the enabling choice would be falsified. A more direct check is to measure the feature distribution distance between intermediate-layer pooled outputs and last-layer pooled outputs and test whether the performance gap across the paper's pooling ablation correlates with that distance.","supporting_citations":[{"cited_title":"Detecting moments and highlights in videos via natural language queries","cited_arxiv_id":null,"evidence_quote":"Moment-DETR and QVHighlights provide the main benchmark dataset and the anchor-free DETR baseline for MR and HD."},{"cited_title":"Internvideo2: Scaling foundation models for mul- timodal video understanding","cited_arxiv_id":null,"evidence_quote":"InternVideo2 is the spatio-temporal backbone whose intermediate features and frozen AdaptivePool module the ST framework integrates."}],"review_version":1}