{"id":"6e6361ca-2224-4640-9af1-77ffbd26bfb3","arxiv_id":"2509.05604","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Language-guided, recursively refined spatiotemporal object and frame graphs yield state-of-the-art video summarization scores on standard benchmarks.","lead":"VideoGraph summarizes videos by building object-level graphs inside each frame and frame-level graphs across time, then refining both with language cues. The system reports top F-scores on SumMe, TVSum, and QFVS in both supervised and unsupervised settings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline SumMe/TVSum F-scores (58.1/69.5) are produced with N=16 and W=8 selected by maximizing F-score on the same five test splits used in Table 3 (Tables 9-10), with no validation split and no per-split variance; test-set-adaptive selection may account for a large share of the claimed…","rationale":"I read the central claim as a quantitative one: VideoGraph with word-level queries reaches state-of-the-art F-scores on SumMe, TVSum, and QFVS in both supervised and unsupervised settings. The chain of support is (i) the reported numbers, (ii) the fairness of the comparison protocol, and (iii) the ablations attributing gains to the proposed components. The weakest link is (i)-(ii): the exact configuration behind the headline numbers (N=16, W=8/25) is chosen by maximizing average F-score on the same test splits that are later reported, and no variance information accompanies the five-split averages. The swings caused by N and W (up to 9.3 points across the N range) are the same order of magnitude as the claimed margins over the strongest baseline (3.2-3.9 points), so test-set-adaptive selection is large enough to be load-bearing. This is a methodological fix, not a fatal flaw: the model is plausible, the code is released, the ablations are extensive, and the transfer-setting gains are large. I disagree with the reader's identification of the weakest assumption. The shared-detector issue is real and worth a condition (an ablation with independently sourced word queries), and the paper itself concedes in the Future Work section that 'inappropriate biases from the pretrained models' can propagate, but it attacks the novelty interpretation ('language-guided semantic knowledge') more than the quantitative claim: the QFVS query-focused results use independent groundtruth captions and still beat the prior best on all four test videos, so the performance story does not collapse if the detector is circular. Test-set-adaptive hyperparameter selection, by contrast, directly threatens the numbers themselves. The reader did list a 'proper validation protocol for N and W' among the conditions, so the CONDITIONAL verdict stands unchanged; I would simply elevate that condition to the primary one. I weighed the other flagged issues: the QFVS single-test-video protocol weakens but does not overturn that claim (gains are consistent, +2.6 to +4.0 per video); Fig. 8 and Table 7 inconsistencies are reporting errors, noted above; I found no mathematical error in the core equations (Eqs. 7-17), though the unbounded residual additions to a row-stochastic adjacency matrix in Eqs. 14-15 are unconventional and worth checking against the released code. Net: keep CONDITIONAL, with the validation-protocol re-run as the decisive check.","tokens_in":28838,"tokens_out":19524,"duration_ms":160285,"concrete_test":"One check would settle the concern: re-run the supervised standard-setting experiments on SumMe and TVSum with N and W selected on a validation split instead of on the test splits. Concretely, carve a 10% validation subset out of each training split (or hold out one of the five test splits for selection and evaluate on the remaining four), choose N in {1,2,4,8,16,20,36} and W in {1,2,4,8,10,25,30} on that validation subset, then fix N and W and report mean and per-split F-scores on the held-out test splits. If the resulting averages fall short of the reported 58.1 (SumMe) and 69.5 (TVSum) by more than roughly 1.5 points, the headline SOTA margins largely reflect test-set-adaptive hyperparameter selection. The same run should report per-split values for VideoGraph and for Narasimhan et al. [37] so that a paired test of the 3.2-3.9 point margins is possible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in the quantitative claim, not the architecture: the two headline configuration choices are selected directly on the test benchmarks. Section 5.5.3 (Table 9) fixes N=16 because it gives the highest F-score on the TVSum benchmark ('we evaluated our model primarily using the top 16 object feature representations based on the ablation study'); Section 5.5.4 (Table 10) fixes W=8 on SumMe/TVSum and W=25 on QFVS, each at the peak of its curve ('In the main experiments, we thus use 8 words on SumMe and TVSum, and 25 words on QFVS'). The standard data configuration (Sec 5.1.3) defines only an 80/20 train/test split with no validation subset, and Sec 5.2.3 reports only averages over five random splits, without per-split variance. The selection effect is material: the N-ablation spans 60.2-69.5 F-score (9.3 points), N=16 vs N=8 is +2.6, and the W curve moves about 1 point around its peak, while the SOTA margin over the strongest baseline [37] is only 3.9 (SumMe) and 3.2 (TVSum) in the supervised standard setting (Table 3). Choosing the best of seven N values and seven W values on the very splits that produce the headline numbers can plausibly account for most of that margin, so the 'state-of-the-art' claim is not established at the reported magnitude, and with five splits and no reported variance the 3.2-point margin cannot be assessed for significance. Secondary reporting inconsistencies (Fig. 8 shows identical per-iteration F-scores in (a)-(c) yet final F-scores 50.3/62.1/70.2, and the text's 6.8% gain does not match the 11.8-point difference; Table 7's full-model row reports word-level numbers on SumMe/TVSum while the text says the ablation uses the sentence-level query) reduce confidence in reporting care.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes VideoGraph, a method for generic and query-focused video summarization that formulates the task as language-guided spatiotemporal graph modeling. Detected objects in each frame are treated as nodes of a per-frame spatial graph, and averaged object representations with positional encodings form the nodes of a temporal graph over frames. Both graphs use language-query-conditioned multi-head cross-attention, and the adjacency matrices are refined recursively by residual updates. The authors compare sentence-level and word-level language queries and evaluate supervised and unsupervised settings on SumMe, TVSum, and QFVS, reporting state-of-the-art F-scores, for example 58.1 on SumMe and 69.5 on TVSum in the supervised standard setting.","tokens_in":29170,"tokens_out":9102,"duration_ms":84889,"significance":"If the reported results hold, this is a meaningful advance: object-level spatial and temporal graph reasoning with recursive adjacency refinement, combined with low-cost word-level language queries, would outperform prior frame-level and transformer-based baselines on standard benchmarks. The architecture is specified precisely (Eqs. 4-17, Table 2), the ablations isolate the contribution of each component, the code is released, and the runtime/parameter comparison in Table 11 is a useful contribution. The main caveat is empirical rather than architectural: the two headline hyperparameters are selected directly on the test benchmarks, and the reported margins over the strongest baseline are comparable in size to the effect of that selection. The missing variance reporting further prevents assessing whether the claimed state-of-the-art margins are statistically meaningful.","major_comments":[{"comment":"The two headline configuration choices are selected directly on the test benchmarks. Section 5.2.2 states that N=16 is used 'based on the ablation study', and Table 9 shows that N=16 is simply the peak of the TVSum F-score curve (60.2 at N=1 rising to 69.5 at N=16 and then falling). Similarly, Table 10 selects W=8 on SumMe and TVSum and W=25 on QFVS at the peak of each curve. The standard data configuration in Section 5.1.3 defines only an 80/20 train/test split with no validation subset, so these choices appear to be made on the same test splits that produce the headline numbers in Tables 3 and 5. The magnitude of the selection effect is comparable to the claimed improvement: the supervised standard margins over [37] are 3.9 on SumMe and 3.2 on TVSum, while the W-ablation spans 54.6-58.1 on SumMe and 65.4-69.5 on TVSum, and the N-ablation spans 60.2-69.5 on TVSum. Please either select N and W on held-out validation splits, fix them a priori, or report the full sensitivity analysis as the basis for a more cautious claim. As written, the state-of-the-art claim at the reported magnitude is not established.","section":"Sections 5.2.2, 5.5.3, 5.5.4; Tables 9 and 10"},{"comment":"Section 5.2.3 states that experiments are run five times on five random splits and reported as averages, but no standard deviations, per-split scores, or significance tests are provided anywhere in the paper. For the supervised standard setting, the margin over [37] is only 3.9 F-score on SumMe and 3.2 on TVSum; with only five random splits, these averages could easily overlap under typical split-level variance in this benchmark. Please report per-split results and standard deviations, and ideally run paired comparisons against the strongest baseline on identical splits so the reader can judge whether the claimed improvements are statistically distinguishable from noise.","section":"Section 5.2.3 and Table 3"}],"minor_comments":[{"comment":"The per-iteration F-score labels in Figure 8(a)-(c) are identical (0.368, 0.423, 0.466, 0.487) for SumGraph, VideoGraph without language, and VideoGraph with word-level queries, even though the final F-scores differ (50.3, 62.1, 70.2). Please correct the labels, or state explicitly that these intermediate values are not per-method and explain why they are repeated.","section":"Figure 8"},{"comment":"The text says VideoGraph uses '0.8M fewer parameters' than [44], but Table 11 shows total parameters of 49.3M for VideoGraph versus 12.1M for [44]; the 0.8M reduction holds only for learnable parameters. Please disambiguate 'parameters' as 'learnable parameters' in that sentence.","section":"Section 5.4 and Table 11"},{"comment":"The word-level language queries are the class labels of the same detected objects that provide the visual node features, so the language signal is not independent of the visual detector. A short discussion of how much of the word-query gain is genuinely semantic rather than a re-encoding of detector output would strengthen the comparison, and the future-work paragraph already partially acknowledges this concern.","section":"Sections 4.2.2 and 5.2.2"},{"comment":"For the QFVS query-focused protocol, the paper says two videos are used for training, one for validation, and one for testing, but Table 6 reports results for four videos. Please state explicitly whether the 2/1/1 split is rotated over all four videos and whether Table 10's word-count selection on QFVS uses the validation video or the same test video used in the main table.","section":"Section 5.1.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is technically solid: the architecture is described at a reproducible level, the code is released, and the ablation coverage is thorough. My recommendation rests on the evaluation protocol rather than on the method itself. The test-set-adaptive selection of N and W and the absence of variance reporting are fixable within the scope of a revision, so I do not recommend rejection. I see no novelty disclosure concern relative to the authors' prior SumGraph work; the current version adds object-level spatial graph modeling and language-guided graph construction, which are substantial extensions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper to know about: VideoGraph, from Park/Lee/Sohn. It extends the authors' SumGraph line by adding object-level spatial graphs, language-guided cross-attention, and recursive refinement of both adjacency matrices. Word-level queries come from the detector's class labels, so no captioning model is needed. On SumMe/TVSum it reports 58.1/69.5 supervised standard, a few points above CLIP-It. The architecture is specified carefully, ablations are extensive, and code is released.\n\nWhat's genuinely useful: the word-level query idea is practical, the spatial/temporal recursive refinement is a clear extension of SumGraph, and the efficiency comparison is honest. This is a competently executed applied paper.\n\nThe soft spots are real but fixable. The main one: N=16 and W=8 are selected by maximizing F-score on the very test splits that produce the headline numbers (Tables 9 and 10), with no validation split and no per-split standard deviation. The N curve moves 9 points, the W curve about 3, while the margin over CLIP-It is 3.9 and 3.2. So the SOTA claim at this magnitude is not established. A proper validation protocol and variances would likely settle it. Second, the word-level query is literally the class labels of the same frozen detector that supplies visual features, so 'language guidance' is not an independent signal; the paper's own future-work section concedes pretrained biases propagate. An ablation with an independently sourced query would help. Third, Fig. 8 shows identical per-iteration F-scores across methods yet different final scores, and the text's 6.8% improvement doesn't match the 11.8-point difference. That is sloppy but not load-bearing. The QFVS number rests on one test video, so treat it as indicative.\n\nCitation and related-work handling are fair, including ERA and CLIP-It. The math is straightforward and the method is reproducible from the text.\n\nWho is this for? Anyone working on language-guided video summarization. It deserves a serious referee; a revision that reports variances, fixes the hyperparameter selection protocol, and adds an independent word-query ablation would make the contribution solid.","headline":"Competent incremental architecture with real benchmark gains, but the headline margin is fragile because N and W are tuned on the test splits and no variance is reported.","tokens_in":29855,"tokens_out":1951,"would_cite":true,"duration_ms":18782,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"VideoGraph claims that video summarization improves when frames are modeled as language-guided graphs over detected objects, reaching state-of-the-art F-scores of 58.1 on SumMe and 69.5 on TVSum.","keywords":["video summarization","language-guided summarization","query-focused summarization","spatiotemporal graph","graph convolutional network","recursive refinement","keyframe selection","object-level representation"],"falsifier":"On TVSum, set the language query representation $q$ to zero in both cross-attention modules while keeping the object graph, recursive refinement, and all losses identical; if the F-score stays near the reported 69.5 instead of falling toward the no-language 65.2, then language guidance is not what is carrying the result.","tokens_in":28524,"feed_emoji":"🎬","tokens_out":10722,"duration_ms":85855,"temperature":0.7,"pith_summary":"This paper tries to establish that video summarization should be modeled at the level of objects and their semantic relations, not just whole frames. It proposes VideoGraph, a spatiotemporal graph network in which detected objects are nodes of per-frame spatial graphs, frames are nodes of a temporal graph, and a language query derived from the video is injected into both graphs to guide node representations. The graphs are then refined recursively by estimating residual adjacency matrices, so keyframes end up connected by story-level semantic affinity rather than visual similarity. On the SumMe and TVSum benchmarks the model reports state-of-the-art F-scores (58.1 and 69.5 with word-level queries in the supervised standard setting), and on the QFVS dataset it improves both generic and query-focused summarization. A sympathetic reader would care because this offers a concrete recipe for distinguishing keyframes from visually similar background frames, and it works in supervised and unsupervised regimes.","feed_headline":"Object-level graph summarization hits 58.1 on SumMe, 69.5 on TVSum","feed_subtitle":"Language-guided recursive graphs over detected objects beat prior frame-level and query-based summarizers.","key_machinery":"The load-bearing machinery is a pair of nested graph networks with recursive residual refinement: a spatial graph whose nodes are detected objects in each frame, and a temporal graph whose nodes are averaged object representations with positional encodings. The language query representation is used as the query in multi-head cross-attention at both graph levels, so each node's features are rewritten to be semantic rather than purely visual. At every iteration, the temporal adjacency matrix is refined with a residual estimated from spatial graph outputs, and the spatial adjacency matrix is refined with a residual estimated from temporal graph outputs, so the graphs gradually converge to story-level connectivity rather than visual-similarity connectivity. Graph convolution then aggregates nodes along these refined edges.","core_discovery":"The central claim is that formulating video summarization as a language-guided spatiotemporal graph over detected objects, with recursively refined adjacency matrices, outperforms prior frame-level and language-guided approaches. In VideoGraph, each frame is a spatial graph whose nodes are object regions; those nodes are averaged and position-encoded into frame nodes of a temporal graph. A language query---either captions from a dense video captioning model or the most frequent detected object class names---acts as the query in multi-head cross-attention, rewriting object and frame features so that graph edges encode semantic affinity instead of raw visual similarity. The spatial and temporal graphs refine each other: the temporal graph estimates residuals that update the spatial adjacency matrix, and the spatial graph estimates residuals that update the temporal adjacency matrix, repeated for five iterations. At convergence the temporal graph links keyframes with high affinity, giving the reported 58.1 F-score on SumMe, 69.5 on TVSum, and 57.95 average F-score on query-focused QFVS.","pith_inferences":["The paper's own future-work paragraph concedes that pretrained-model biases can propagate into the model; because the same frozen detector feeds both node features and word queries, swapping in an open-vocabulary detector (one that can recognize objects beyond its training classes) is the most direct stress test of where the gains come from.","The optimal number of query words shifts with video length (8 words on SumMe and TVSum, 25 words on QFVS), suggesting an adaptive rule that chooses word count per video from detection statistics or scene boundaries rather than fixing it per dataset.","The recursive residual coupling between spatial and temporal graphs is a general way to impose story-level connectivity, so it could plausibly transfer to neighbouring video-language tasks such as query-focused moment retrieval or temporal grounding.","The dominance scores read off the final spatial edge weights imply a weakly supervised object-importance signal, one that could rank objects or select queries without any object-level annotations."],"forward_implications":["Word-level queries (frequent detected class names) are enough to reach the best scores on short videos: 58.1 on SumMe and 69.5 on TVSum, beating the same model without language by 3.4 and 4.3 points respectively.","Sentence-level queries become more valuable on long multi-event videos: on QFVS generic summarization the sentence variant scores 71.29 versus 68.75 for word-level, and with ground-truth captions 77.83.","Recursive refinement is not decorative: the unrefined initial graph gives 49.8 on SumMe and 59.6 on TVSum, while five iterations give 58.1 and 69.5, with convergence in three to five iterations.","Language guidance can be added without caption-generation latency: the word-level model skips the roughly 250-second captioning step and still gets a higher F-score than the sentence-level variant trained on generated captions.","The unsupervised variant, trained only with sparsity, diversity, and reconstruction losses, reaches 55.0 on SumMe and 66.8 on TVSum with word-level queries, narrowing the gap between supervised and unsupervised video summarization."],"supporting_citations":[{"why":"Supplies the recursive temporal-graph formulation that VideoGraph extends to spatiotemporal graphs and is the main comparison baseline.","marker":"[44]"},{"why":"Defines the language-guided summarization setting and the cross-attention mechanism that VideoGraph adapts with language as query.","marker":"[37]"},{"why":"Provides the frozen object detector whose region proposals become spatial graph nodes.","marker":"[46]"},{"why":"Provides the pretrained object features and class labels used as node features and as word-level language queries.","marker":"[88]"},{"why":"Generates the sentence-level caption queries used in the sentence-query variant.","marker":"[40]"},{"why":"Text-encodes the sentence and word queries into the language representation used in both graphs.","marker":"[79]"},{"why":"Defines the graph convolution used for spatial and temporal aggregation in VideoGraph.","marker":"[21]"},{"why":"Supplies the SumMe benchmark and ground-truth annotations used in training and evaluation.","marker":"[47]"},{"why":"Supplies the TVSum benchmark and ground-truth annotations used in training and evaluation.","marker":"[48]"},{"why":"Supplies the QFVS egocentric dataset with query-focused ground-truth summaries on which the method reports query-focused gains.","marker":"[6]"}],"fun_headline_variants":["Language-guided object graphs refine keyframes recursively","Recursive spatiotemporal graphs over objects for video summarization","VideoGraph: language-guided object graphs beat baseline summarizers","Object graph model with language queries scores 58.1 on SumMe","Spatiotemporal object graphs with linguistic guidance for keyframes"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the frozen object detector finds the entities that carry the video's story; if its top proposals miss or mislabel those entities, the spatial graph, the temporal graph, and the word-level language queries all degrade together because they share that single visual source.","fun_headline_variants_meta":{"raw":{"variants":["Language-guided object graphs refine keyframes recursively","Recursive spatiotemporal graphs over objects for video summarization","VideoGraph: language-guided object graphs beat baseline summarizers","Object graph model with language queries scores 58.1 on SumMe","Spatiotemporal object graphs with linguistic guidance for keyframes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000871,"raw_usage":{"total_tokens":3794,"prompt_tokens":992,"completion_tokens":2802,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":608,"completion_tokens_details":{"reasoning_tokens":2720}},"tokens_in":608,"tokens_out":2802,"duration_ms":19179,"temperature":1.0,"reasoning_tokens":2720,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:22:45.059530+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On TVSum, set the language query representation $q$ to zero in both cross-attention modules while keeping the object graph, recursive refinement, and all losses identical; if the F-score stays near the reported 69.5 instead of falling toward the no-language 65.2, then language guidance is not what is carrying the result.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the recursive temporal-graph formulation that VideoGraph extends to spatiotemporal graphs and is the main comparison baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the language-guided summarization setting and the cross-attention mechanism that VideoGraph adapts with language as query."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the frozen object detector whose region proposals become spatial graph nodes."},{"cited_title":"IEEE Conf","cited_arxiv_id":null,"evidence_quote":"Provides the pretrained object features and class labels used as node features and as word-level language queries."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Generates the sentence-level caption queries used in the sentence-query variant."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Text-encodes the sentence and word queries into the language representation used in both graphs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the graph convolution used for spatial and temporal aggregation in VideoGraph."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the SumMe benchmark and ground-truth annotations used in training and evaluation."},{"cited_title":"IEEE Conf","cited_arxiv_id":null,"evidence_quote":"Supplies the TVSum benchmark and ground-truth annotations used in training and evaluation."}],"review_version":2}