{"id":"212a22a7-3297-4399-8e23-4cfc5660a1a8","arxiv_id":"2411.18042","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"HyperGLM builds a unified hypergraph from entity scene graphs and a procedural graph, injects it into an LLM, and reports state-of-the-art results on five video scene graph tasks using a new VSGR dataset.","lead":"HyperGLM is a video-scene-graph system that feeds a unified hypergraph, built from spatial and temporal relationship graphs, into a large language model, along with a new 1.9M-frame reasoning dataset. The paper reports gains on five video understanding tasks, but some of the reported numbers do not consistently support that claim.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VQA/VC/RR gains in Tables 5-7 may be read from an oracle hypergraph: H at test time is unspecified, and VSGR items are generated from the same triplets H encodes (Sec 5.1). A predicted-vs-ground-truth H or graph-ablated evaluation is needed before the five-task SOTA claim can be credited.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing point: no statement of whether test-time H is predicted or ground truth, and a dataset construction that makes leakage possible. This is the single most important issue because the entire 'reasoning' half of the paper (VQA, VC, RR) is evaluated only through this pathway. If H is ground truth, HyperGLM is not demonstrating multimodal LLM reasoning on a scene hypergraph; it is demonstrating the ability to copy a relation from an oracle graph. The issue extends beyond experimental rigor to construct validity: the tasks are defined in Sec. 5.1 from the same triplets the graph encodes, so H is a near-complete answer key. Table 6's internal contradiction (CIDEr 54.5 vs 57.1 and MENTOR 30.7 vs 37.5 for MV-GPT) independently motivates caution, but the leakage concern alone would require rejecting the paper's central claim as stated. A concrete predicted-H ablation and a leak-rate statistic would settle it. If the authors can show the reported H is model-predicted and that removing it does not collapse accuracy, the VQA/VC/RR results would be more credible; the SGG/SGA comparisons, while still lacking error bars and code, are at least a valid comparison against prior scene-graph models. As submitted, the evidence does not support the stated conclusions, so I concur with the reader's REJECT recommendation; no verdict change is needed.","tokens_in":15409,"tokens_out":6126,"duration_ms":53681,"concrete_test":"Run one controlled evaluation on the VSGR VQA/RR test split with three variants: (1) HyperGLM with the paper's test-time H; (2) HyperGLM with H removed (video + question only); (3) HyperGLM with H constructed from the model's own predicted scene graphs using the same detector/triplet pipeline used for SGG, not from ground-truth annotations. Compare against the Table 5 and Table 7 numbers. Additionally, compute the fraction of test questions whose gold answer triplet appears verbatim in the ground-truth H supplied at test time; if that fraction is high and accuracy collapses without H or with predicted H, the reported VQA/RR gains are attributable to oracle graph access, not relational reasoning. Apply the same H-removal check to VC by generating captions with and without H.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that injecting the unified HyperGraph into LLMs 'consistently outperforms state-of-the-art methods across five tasks' (Abstract; Sec. 4.2) depends critically on the VQA, VC, and RR evaluations in Tables 5-7. These tasks are meaningful only if the graph H passed to the LLM at test time is produced by the model from the same visible video evidence available to the baselines, and does not already contain the relationships being queried. The paper never states whether H is model-predicted or ground-truth. Sec. 5.1 describes generating VQA and VC items from the same <subject, relation, object> triplets that HyperGLM encodes into H (Sec. 4.1), and Fig. 5 shows a hypergraph whose edges are exactly the relations used in the dialog. If H is ground-truth at test time, a question such as 'what is the relation between person and cup?' can be answered by reading the corresponding hyperedge, and comparison against video-only baselines (Video-ChatGPT, Video-LLaVA, MovieChat, Chat-UniVi, MA-LMM, LLaMA-VID) is not a test of reasoning. No ablation removes H, and no experiment replaces H with model-predicted scene graphs; the Limitations section does not acknowledge this ambiguity. Separately, Table 6 itself contradicts 'consistent' SOTA: HyperGLM's CIDEr/MENTOR are below MV-GPT, so the central claim is already overstated even before the leakage question is resolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HyperGLM, a multimodal LLM-based framework for video scene graph generation (SGG), scene graph anticipation (SGA), video question answering (VQA), video captioning (VC), and relation reasoning (RR). The method constructs a unified HyperGraph H that merges per-frame entity scene graphs with a procedural graph modeling relationship transitions, samples hyperedges via random walks, and injects H into an LLM (Mistral-7B with LoRA). The authors also introduce the VSGR dataset with 1.9M frames from third-person, egocentric, and drone viewpoints, supporting all five tasks. The paper claims that HyperGLM consistently outperforms state-of-the-art methods across all five tasks on PVSG, Action Genome, and VSGR. The manuscript includes algorithm pseudocode, implementation details, ablation plots for hypergraph parameters, and comparison tables for each task.","tokens_in":15737,"tokens_out":5205,"duration_ms":48075,"significance":"If validated, the central idea of representing higher-order spatial-temporal interactions as a hypergraph and injecting it into an LLM could be a useful direction for video scene graph reasoning. The VSGR dataset is a potentially valuable resource, being larger than prior video scene graph datasets and covering five tasks from multiple viewpoints. The random-walk construction in Algorithm 1 is clearly specified, and the authors attempt to address long-tail predicate bias through a procedural graph. However, the significance of the empirical claims is currently undermined by an underspecified test-time protocol for H and by evaluation results that do not uniformly support the 'consistently outperforms' statement. The paper's contributions are credible only after these issues are resolved with additional experiments or explicit clarifications.","major_comments":[{"comment":"The test-time provenance of the HyperGraph H is not specified. In Sec. 5.1, the VQA, VC, and RR items are generated from <subject, relation, object> triplets, and in Sec. 4.1, H is constructed by integrating those same entity scene graphs G_t and the procedural graph P. If H at test time is the ground-truth graph, then a question such as 'what is the relation between person and cup?' can be answered by reading the corresponding hyperedge, and the comparison against video-only baselines in Tables 5-7 does not measure reasoning. The paper never states whether H is model-predicted or ground-truth, and Fig. 5 appears to show ground-truth relations in the hypergraph. Please state explicitly how H is obtained at inference for each of the five tasks, and provide an ablation without H and with a model-predicted graph built from detected objects and predicted pairwise relations. Without this, the central five-task SOTA claim cannot be evaluated.","section":"Sec. 4.2, Eq. (7) and Sec. 5.1"},{"comment":"The abstract and conclusion claim that HyperGLM 'consistently outperforms state-of-the-art methods' across five tasks, but Table 6 directly contradicts this: HyperGLM's CIDEr of 54.5 is below MV-GPT's 57.1, and its MENTOR of 30.7 is below MV-GPT's 37.5. Similarly, Table 2 shows HyperGLM with lower mean Recall than SceneSayerSDE in multiple VSGR cells (e.g., F=0.3, R/mR@20: 16.5 vs 17.0; F=0.7, R/mR@10: 13.5 vs 14.5). The claim should be revised to reflect that HyperGLM achieves the best or near-best results on some metrics, or the authors should provide a statistical aggregation justifying 'consistently outperforms'.","section":"Table 6 and Table 2"},{"comment":"The evaluation reports only point estimates without error bars, multiple seeds, or significance tests. Many claimed improvements are small; for example, Table 5 shows VQA accuracy of 45.4% for HyperGLM versus 44.3% for Chat-UniVi-7B, and Table 7 shows RR accuracy of 47.2% versus 44.1% for LLaMA-VID-7B. The reader cannot determine whether these gaps are above run-to-run noise, especially for LLM-based systems with decoding stochasticity. Please report variance across at least three runs or provide significance tests for the key SOTA comparisons.","section":"Tables 3-7"},{"comment":"The construction of H appears to require relationship labels for the entity scene graphs G_t (Eq. (6)), but for the SGG task the model is supposed to predict those relationships from raw video frames. It is unclear where the relationships used to build H come from at test time. If ground-truth relationships are used to build H during SGG inference, then the R@20 gains in Table 3 are not meaningful. If H is instead built from a base model's predictions, that pipeline should be described explicitly and trained end-to-end or with a comparable protocol. Please clarify the inference-time construction of H for SGG and SGA.","section":"Sec. 4.1 and Sec. 6.1"}],"minor_comments":[{"comment":"The implementation details state 'We train for two epochs with a batch size of 128 over 16 iterations on 4 × GPUs' — 16 iterations is inconsistent with two epochs on a dataset of this size; this is likely a typo and should be corrected.","section":"Sec. 6.1"},{"comment":"The text refers to 'theoretical foundations and mathematical properties are detailed in the Appendices' and to additional parameter experiments in the Appendices, but the submitted version does not contain these appendices. Please include them or remove the references.","section":"Main text and Appendices"},{"comment":"The baseline name 'Video-LLaV A-7B' in Table 5 appears to be a typo for 'Video-LLaVA-7B'; please make the baseline naming consistent throughout all tables.","section":"Tables 5 and 7"},{"comment":"The ablation in Fig. 6 shows performance versus the number of hyperedges, but no variance or confidence intervals are provided; adding error bars with multiple runs would make the choice of Nw=60 and Nl=7 more convincing.","section":"Fig. 6"},{"comment":"The dataset construction states that 'we exclude questions that LLMs can answer correctly,' but it is not specified which LLM is used for this filtering or how that choice affects the difficulty and validity of the remaining questions; please provide this detail and the proportion of questions removed.","section":"Sec. 5.1"}],"recommendation":"major_revision","confidential_remarks":"The core idea is interesting and the dataset is a substantial resource, but the current manuscript requires a major revision. The biggest concern is the underspecified test-time hypergraph: if H is ground-truth, the VQA/VC/RR results are inflated by label leakage, and even the SGG/SGA results would be circular. This is fixable by clearly specifying a predicted-graph protocol and adding ablations, but without those the paper cannot be published. The Table 6 contradiction with the 'consistently outperforms' claim is also a serious overstatement that must be corrected. I would advise the editor to ask for a revised version with additional experiments and a more measured set of claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the SGG/SGA results and the VSGR dataset are worth a look, but the headline 'consistently outperforms' across five tasks doesn't survive contact with the paper's own tables, and there's a likely leakage problem in the reasoning tasks.\n\nWhat's actually new: a unified hypergraph that combines entity scene graphs with a procedural graph via random walks, injected into a Mistral-7B LLM with LoRA. That specific configuration is not in the prior work I know. The VSGR dataset (1.9M frames, five tasks, including a new Relation Reasoning task) is a real artifact and potentially useful. The SGG/SGA comparisons on PVSG, Action Genome, and VSGR against established baselines (STTran, DSGDetr, SceneSayer) are informative and mostly in their favor — for instance R@20 7.5 vs 5.8 on PVSG, and R@10 38.8 vs 37.3 on Action Genome SGA. That part is credible evidence.\n\nThe soft spots are serious. First, the VQA/VC/RR evaluations in Tables 5–7 may be reading answers from an oracle. The questions and captions are generated from the same <subject, relation, object> triplets that the hypergraph encodes (Sec 5.1), and the paper never states whether the test-time H is model-predicted or ground-truth. Figure 5 shows a hypergraph whose edges are exactly the relations used in the dialog. If H is ground truth at test time, the improvements over video-only baselines (Video-ChatGPT, Video-LLaVA, etc.) reflect reading the input, not reasoning. The Limitations section does not acknowledge this ambiguity, and there is no ablation removing H or replacing it with predicted scene graphs. That is a load-bearing flaw.\n\nSecond, the 'consistently outperforms' claim is already contradicted by Table 6: CIDEr 54.5 is below MV-GPT's 57.1, and MENTOR 30.7 is below MV-GPT and others. Some VSGR SGA cells in Table 2 also trail SceneSayerSDE. So the central claim is overstated even before the leakage question is settled.\n\nThird, there are no error bars or significance tests, and hyperparameters (Nw, Nl) appear to be tuned on the same benchmark they are evaluated on. No code or data is released.\n\nWho is this for? Researchers in video scene graph generation and multimodal LLM reasoning. The dataset could be a useful resource if cleaned and released. As submitted, the evidence supports a paper about a new hypergraph representation and dataset, not one about five-task SOTA.\n\nRecommendation: send to peer review — the core idea and dataset are substantive and deserve referee time — but expect heavy revision. I would ask for a clarification of H at test time, an ablation without H, and a corrected abstract that matches the tables.","headline":"Useful hypergraph idea and a big new dataset, but the five-task SOTA claim is undermined by an unspecified test-time graph and the paper's own captioning numbers.","tokens_in":16319,"tokens_out":2123,"would_cite":false,"duration_ms":18890,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A unified hypergraph of spatial and causal relationships, injected into an LLM, improves video scene graph generation, anticipation, and reasoning across five tasks.","keywords":["video scene graph generation","scene graph anticipation","hypergraph","multimodal large language models","video question answering","video captioning","relation reasoning","video scene graph reasoning dataset"],"falsifier":"A decisive experiment: evaluate the same trained HyperGLM on VSGR VQA and Relation Reasoning twice, once with the hypergraph injected and once with a video-only prompt. If accuracy collapses when the graph is withheld, and matches the reported numbers when the LLM is given the graph text alone without video, the reported gains reflect reading injected labels rather than video reasoning.","tokens_in":15150,"feed_emoji":"🎬","tokens_out":13165,"duration_ms":94459,"temperature":0.7,"pith_summary":"This paper claims that video scene understanding improves when object relationships are represented as a unified hypergraph—hyperedges connecting multiple entities and relationship types—and when that hypergraph is injected into a large language model. Prior video scene graph methods, the authors argue, model only pairwise relationships and miss multi-way interactions and the causal evolution of relations across frames. The authors build the VSGR dataset (1.9 million frames, three viewpoints, five tasks) and present HyperGLM as consistently outperforming state-of-the-art baselines on scene graph generation, scene graph anticipation, video question answering, video captioning, and relation reasoning. If the claim holds, higher-order graph structure would be a useful intermediate representation for LLM-based video reasoning.","feed_headline":"HyperGLM tops five video scene tasks with an injected hypergraph","feed_subtitle":"Multi-way scene hyperedges beat pairwise links on generation, anticipation, and reasoning.","key_machinery":"The central object is the unified HyperGraph $H = (V_H, E_H)$, which combines entity scene graphs $G_t$ (spatial subject–relationship–object triples per frame) with a procedural graph $P$ (weighted transitions between relationship categories, with probabilities $w(r_m, r_n)$ estimated from observed frequencies). A random-walk algorithm (Alg. 1) samples hyperedges from this combined structure—e.g., $\\{person, holding, placing, releasing\\}$—capturing multi-object and multi-step interactions. This hypergraph is injected into a Mistral-7B instruction-tuned LLM through the conditioning $p(A \\mid V, Q, H)$, and the procedural graph's transition probabilities are what allow the model to anticipate future relationships.","core_discovery":"HyperGLM's central claim is that a unified HyperGraph, merging per-frame entity scene graphs with a procedural graph of relationship transitions, enables a multimodal LLM to reason about multi-object interactions better than pairwise scene graphs alone. The hypergraph is constructed by random walks that sample hyperedges (for instance, person–sitting–holding–playing guitar), and the generation objective $p(A \\mid V, Q, H)$ conditions the language model directly on this structure. The paper reports consistent gains over transformer-based and NeuralODE/NeuralSDE baselines, with SGG Recall@20 of 7.5% on PVSG and 35.8% on VSGR, SGA Recall@10 of 35.7% on Action Genome and 25.1% on VSGR, 45.4% accuracy on VQA, and 47.2% accuracy on Relation Reasoning.","pith_inferences":["The paper does not state whether the hypergraph injected at test time for VQA, VC, and RR is model-predicted or ground truth; if it is ground truth, the reported gains over video-only baselines likely overstate the model's reasoning ability.","Because the VSGR questions and captions are generated from the same <subject, relation, object> triplets that the hypergraph encodes, a decisive test is to withhold the hypergraph at test time and measure how much accuracy survives—this separates graph-reading from genuine video reasoning.","The random-walk parameters ($N_w = 60$, $N_l = 7$) are fixed; a testable extension would make the walk count and length adaptive to scene complexity rather than constant across datasets."],"forward_implications":["If HyperGLM's claim holds, replacing pairwise scene-graph modules with hypergraph construction should improve downstream video tasks that depend on multi-object relationships, such as event forecasting and human–object interaction understanding.","The VSGR dataset gives the community a single benchmark with 1.9M frames and annotations for all five tasks across third-person, egocentric, and drone views, enabling direct comparison of generation, anticipation, and reasoning methods.","The reported gap between HyperGLM and the hypergraph-only variant indicates that injecting the structured graph into an LLM yields additional reasoning gains beyond the graph representation alone.","Because the procedural graph is built from observed transition frequencies, the method's anticipation ability should carry over to new videos whose relationship dynamics follow the same regularities as the training data."],"supporting_citations":[{"why":"Supplies the PVSG dataset and the Transformer baseline for the Scene Graph Generation comparison.","marker":"[52]"},{"why":"Supplies the Action Genome dataset and evaluation settings for Scene Graph Anticipation.","marker":"[18]"},{"why":"The SceneSayer NeuralODE/NeuralSDE baseline that HyperGLM claims to surpass in anticipation.","marker":"[36]"},{"why":"Source of third-person and egocentric videos for the VSGR dataset and the HIG SGG baseline.","marker":"[34]"},{"why":"Source of drone-view videos for VSGR and the CYCLO SGG baseline.","marker":"[35]"},{"why":"Video-LLaVA is the VQA and Relation Reasoning baseline that HyperGLM compares against.","marker":"[30]"},{"why":"Chat-UniVi provides an additional Video Question Answering baseline.","marker":"[20]"},{"why":"Mistral 7B is the LLM backbone into which the hypergraph is injected.","marker":"[19]"}],"fun_headline_variants":["HyperGLM's hypergraph tops five video scene tasks","HyperGraph injection lifts video reasoning across five tasks","One hypergraph, five scene tasks: HyperGLM's edge","HyperGLM's multi-way hyperedges beat pairwise on video tasks","Unified hypergraph fuels HyperGLM's five-task video wins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the hypergraph fed to the LLM at test time contains only information the model could legitimately infer from the video, not the ground-truth relationships used to write the VSGR questions and captions.","fun_headline_variants_meta":{"raw":{"variants":["HyperGLM's hypergraph tops five video scene tasks","HyperGraph injection lifts video reasoning across five tasks","One hypergraph, five scene tasks: HyperGLM's edge","HyperGLM's multi-way hyperedges beat pairwise on video tasks","Unified hypergraph fuels HyperGLM's five-task video wins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1763,"prompt_tokens":929,"completion_tokens":834,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":749}},"tokens_in":545,"tokens_out":834,"duration_ms":6901,"temperature":1.0,"reasoning_tokens":749,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:34:33.265468+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decisive experiment: evaluate the same trained HyperGLM on VSGR VQA and Relation Reasoning twice, once with the hypergraph injected and once with a video-only prompt. If accuracy collapses when the graph is withheld, and matches the reported numbers when the LLM is given the graph text alone without video, the reported gains reflect reading injected labels rather than video reasoning.","supporting_citations":[{"cited_title":"Panoptic video scene graph generation","cited_arxiv_id":null,"evidence_quote":"Supplies the PVSG dataset and the Transformer baseline for the Scene Graph Generation comparison."},{"cited_title":"Action genome: Actions as compositions of spatio- temporal scene graphs","cited_arxiv_id":null,"evidence_quote":"Supplies the Action Genome dataset and evaluation settings for Scene Graph Anticipation."},{"cited_title":"Towards scene graph anticipation","cited_arxiv_id":null,"evidence_quote":"The SceneSayer NeuralODE/NeuralSDE baseline that HyperGLM claims to surpass in anticipation."},{"cited_title":"Hig: Hier- archical interlacement graph approach to scene graph genera- tion in video understanding","cited_arxiv_id":null,"evidence_quote":"Source of third-person and egocentric videos for the VSGR dataset and the HIG SGG baseline."},{"cited_title":"CYCLO: Cyclic graph transformer approach to multi-object relationship modeling in aerial videos","cited_arxiv_id":null,"evidence_quote":"Source of drone-view videos for VSGR and the CYCLO SGG baseline."},{"cited_title":"Chat-univi: Unified visual representation em- powers large language models with image and video un- derstanding","cited_arxiv_id":null,"evidence_quote":"Chat-UniVi provides an additional Video Question Answering baseline."}],"review_version":1}