{"id":"0a97e6ea-3430-4078-b1e5-787bc2530455","arxiv_id":"2505.03581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A method that compresses a sequence of textual scene graphs into a small set of latent tokens, enabling an LLM to answer situated questions about dynamic scenes with state-of-the-art accuracy on STAR and AGQA2.0.","lead":"DyGEnc encodes a sequence of textual scene graphs into a few special tokens that a large language model can read, and answers questions about changing scenes with high accuracy on the STAR and AGQA benchmarks. The catch is that the high scores come from using hand-labeled scene graphs, not raw video, so the claimed 15-25% margin over visual methods does not yet transfer to real robots.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline gain over visual baselines rests on oracle scene-graph inputs; DRobot shows accuracy <0.35 with auto-extracted graphs, so the claimed 15–25% margin is not demonstrated for video input.","rationale":"The reader's weakest assumption—that ground-truth graph annotations are representative of automatically extractable graphs—is the same load-bearing concern I identify. The DRobot experiment (Sec. IV-F, Table VI) is direct evidence within the paper itself: with foundation-model-extracted graphs, accuracy drops to 0.30–0.34, far below the 0.89 on STAR and below the visual baselines in Table IV. This shows the headline margin is an artifact of oracle input, not a property of the method when applied to raw video. I considered whether the permissive evaluation metric is a more fundamental flaw, but it is secondary: even with a strict metric, the oracle-graph advantage would remain. Conversely, if DyGEnc were evaluated with predicted graphs and still outperformed visual baselines, the metric looseness might be fixable by re-scoring. The method has genuine contributions: a well-defined architecture, thorough ablations (Tables I–III), and honest acknowledgment in Sec. IV-F that real-world performance is much lower. The paper is not internally inconsistent; rather, the abstract overclaims relative to the experiments. A conditional acceptance requiring the authors to (a) reframe the claim to 'given scene graphs' or (b) provide a matched baseline with predicted graphs is appropriate. Since my analysis aligns with the reader's, I recommend no change to the CONDITIONAL verdict.","tokens_in":13244,"tokens_out":4936,"duration_ms":48721,"concrete_test":"Reproduce the STAR validation evaluation of Table IV using DyGEnc-3B with automatically predicted scene graphs (e.g., from a public video scene-graph generator such as PVSG trained on Action Genome, or from the GPT-based pipeline used in Table VI) instead of ground-truth annotations, keeping the same LoRA fine-tuning and substring-match metric. If the average accuracy falls below 0.71 (the LRR visual baseline), the claimed 15–25% margin over visual methods is not achieved under realistic input conditions. As a secondary check, recompute all baselines in Table IV under the paper's substring-match metric to quantify the metric's contribution to the reported margin.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that DyGEnc outperforms existing visual methods by 15–25%—is supported only by evaluations where DyGEnc receives ground-truth scene graphs (STAR/AGQA annotations) while the visual baselines receive raw video. This is a protocol mismatch: DyGEnc is given the correct symbolic structure (objects, relations, actions) at test time, whereas video baselines must infer structure from pixels. Section IV-F and Table VI show the consequence: when graphs are extracted automatically with foundation models on the DRobot benchmark, DyGEnc accuracy falls to 0.30–0.34, below every visual baseline in Table IV (e.g., LRR at 0.71). A second compounding issue is the evaluation metric in Sec. IV-B: a prediction is correct if the ground-truth answer 'contains' the model's response, a more permissive criterion than the exact-match accuracy typically reported by the baselines in Tables IV–V. This can further inflate the reported margin. The method itself may be a sound graph-sequence encoder, but the abstract's claim as stated is not supported without a same-input comparison (predicted graphs for DyGEnc) and a matched evaluation metric.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DyGEnc, a method for encoding a sequence of textual scene graphs into a small number of latent tokens that are injected into a frozen, LoRA-tuned large language model for visual question answering. The architecture combines a text encoder (ModernBERT), a graph encoder (GraphTransformer), a sequence encoder (Q-Former with a fixed number of query tokens), and an MLP projector. The authors evaluate on the STAR and AGQA2.0 benchmarks, reporting average accuracies of 0.89 and 0.73 respectively, and additionally describe a robotics experiment (DRobot) in which scene graphs are extracted automatically from video via foundation models. The central claim, stated in the abstract and Section I, is that DyGEnc outperforms existing visual methods by 15–25% on questions about the history of human-to-object interactions.","tokens_in":13358,"tokens_out":2185,"duration_ms":22336,"significance":"If the headline comparison were fair, the contribution would be significant: the paper demonstrates that a dynamic scene can be compressed into a single latent graph token while retaining enough temporal information for an LLM to answer interaction and sequence questions with high accuracy, and it provides a thorough ablation of the encoder components. The architecture is clearly specified and the ablations are sensible. However, the practical significance is heavily contingent on the quality of automatically extracted scene graphs; the DRobot experiment (Section IV-F) shows accuracy of only 0.30–0.34 with foundation-model-extracted graphs, far below the 0.89 reported on STAR. This discrepancy substantially weakens the real-world applicability claim. The method may still be valuable as a graph-sequence encoder, but the paper's central 'large margin over visual methods' claim is not supported under a protocol where DyGEnc receives oracle annotations while baselines receive raw video.","major_comments":[{"comment":"The comparison with visual baselines is unfair: DyGEnc receives ground-truth scene graph annotations for STAR and AGQA2.0, while all baselines (STEP, Q-ViD, MIST, SeViLA, ViLA, VidF4, LRR) receive raw video or video features. The 15–25% margin over visual methods claimed in the abstract is therefore a comparison of oracle-augmented reasoning against end-to-end visual perception, not a head-to-head test of the same input modality. Section IV-F and Table VI reinforce this concern: when graphs are extracted automatically with foundation models on the DRobot benchmark, DyGEnc accuracy falls to 0.30–0.34, below the 0.71 reported for LRR in Table IV. To support the headline claim, the authors should either (a) compare against visual baselines using predicted scene graphs as input to DyGEnc, or (b) explicitly reframe the claim as 'DyGEnc outperforms visual methods when given ground-truth scene graphs', and move the video-input results to a prominent position.","section":"Sec. IV-D.4 and Table IV"},{"comment":"The evaluation metric is more permissive than the exact-match accuracy typically used by the baselines: a prediction is counted correct if the ground-truth answer 'contains' the generated response. This 'contains' criterion can inflate scores relative to exact-match or normalized-answer evaluation. Since Tables IV and V compare against numbers reported by prior papers, the paper must verify that the baselines used the same evaluation protocol; if not, the margins are not directly comparable. At minimum, the authors should report both 'contains' and exact-match accuracy for DyGEnc on STAR and AGQA2.0 to quantify the inflation.","section":"Sec. IV-B"},{"comment":"The DRobot experiment is presented as evidence that DyGEnc can be 'seamlessly extended to process raw input images', but the reported accuracy (0.30–0.34) is barely above random for a multi-class setting and far below the benchmark numbers. The paper does not compare DRobot results against any visual baseline, so it is unclear whether DyGEnc with extracted graphs is competitive with even simple video QA methods on this data. The limitations section acknowledges the keyframe extraction issue but does not address the graph-construction quality gap. This is a load-bearing issue for the real-world applicability claim; the paper should either provide a baseline comparison on DRobot or temper the claim to 'the full pipeline is feasible but currently underperforms on automatic graphs'.","section":"Sec. IV-F (DRobot)"}],"minor_comments":[{"comment":"There are numerous typos and grammatical slips, e.g., 'positivly skewed' (Sec. IV-A.2), 'appliend' (Sec. III-B), 'unconstraned' (Sec. III-B), 'effietefly' (Sec. IV-D.2), 'Recall metric' where accuracy is used (Sec. IV-D.1, Sec. IV-D.2), and 'AQGA' instead of AGQA in Sec. IV-E heading. These should be corrected.","section":"Throughout"},{"comment":"The column abbreviations in Table V (Obj.-Rel., Rel.-Act., etc.) are not fully expanded in the table; the expansion is only partially given in the text. Please add a caption or footnote that spells out all abbreviations (e.g., Obj.-Rel. = Object-Relationship, Sup. = Superlative, E. = Exists, Act.-Rec. = Activity Recognition, B. = Binary, O. = Open, A. = All).","section":"Sec. IV-E / Table V"},{"comment":"The STAR benchmark reference appears in the related work as [10] but is cited as a 2024 arXiv preprint; the published version (if any) should be cited instead. Also, the paper inconsistently refers to 'STAR' as both a dataset and a benchmark; please standardize.","section":"Sec. II-B / Reference [10]"},{"comment":"The preprocessing step that 'retains unique graphs' is described briefly; it would be helpful to state whether duplicates are removed globally or within each sequence, and how this affects the temporal index t for duration reasoning.","section":"Sec. IV-C.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core architectural idea is interesting and the ablations are well-executed, but the headline claim is overstated relative to the evidence. The authors should be asked to either provide a same-input comparison (predicted graphs for DyGEnc) or substantially reframe the contribution as an oracle-graph reasoning method. The DRobot results should be included in the abstract's discussion of limitations, rather than presented as evidence of seamless extension. I would not reject the paper outright, because the graph-sequence encoder itself appears sound and the claimed compression (0.03x) is a useful property; however, the current framing risks misleading readers about the method's practical readiness."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read: the paper is a legitimate but modest extension of G-Retriever to dynamic scene graphs. The new piece is the Q-Former temporal encoder that compresses a variable-length sequence of textual scene graphs into one (or a few) latent tokens with rotary positional encoding, and the ablations show that compression to a single token costs little on Interaction/Sequence questions. That is a real, useful result for graph-based QA and robotic memory.\n\nThat said, the headline claim — outperforming visual methods by 15-25% — is not supported as stated. On STAR and AGQA, DyGEnc is given ground-truth scene graph annotations, while the visual baselines see only raw video. That is an apples-to-oranges comparison. The paper's own DRobot experiment (Table VI) shows accuracy falls to 0.30-0.34 when graphs are extracted automatically from images with foundation models, which is below every visual baseline in Table IV. So the 15-25% margin is only for oracle graph input, not for video input. The abstract needs to be reframed or the evaluation needs a matched baseline (e.g., running DyGEnc on predicted graphs on STAR/AGQA, or comparing against visual baselines on the same graph input).\n\nThere is also a secondary metric concern: the paper counts a prediction as correct if the ground-truth answer 'contains' the model's response. This is more permissive than exact-match accuracy. The authors say they borrow metrics from prior work, and if the baselines they compare against used the same 'contains' protocol, then it's fair. But they should state that explicitly and ideally report exact-match as well, because the baselines in Table IV likely report a stricter metric.\n\nThe paper is otherwise well done: the architecture is clearly specified, the ablations are sensible (Tables I-III), and the discussion of limitations is honest — they acknowledge the keyframe-extraction problem and that they are not yet a 'foundational' graph encoder. The code is available.\n\nNet: this is a solid graph-sequence encoder, but the central claim in its current form overstates what is demonstrated. A serious referee should ask for a matched comparison or a revised claim before acceptance. With that revision, it would be a useful contribution to the graph-based QA / robotic memory community.","headline":"Solid graph-sequence encoder, but the headline 15-25% margin over visual methods rests on oracle scene-graph input and does not hold for automatically extracted graphs.","tokens_in":14003,"tokens_out":3172,"would_cite":false,"duration_ms":30664,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DyGEnc compresses a sequence of textual scene graphs into a single latent token and shows that a frozen LLM, tuned with LoRA, can then answer interaction and sequencing questions about the scene at high accuracy, outperforming visual…","keywords":["dynamic scene graphs","video question answering","textual scene graphs","graph encoding","large language models","LoRA fine-tuning","Q-Former","robotic memory"],"falsifier":"Run DyGEnc on the STAR validation videos using the same foundation-model graph extraction pipeline as the DRobot experiment instead of the provided annotations. If average accuracy falls to the 0.30–0.34 range rather than remaining near 0.89, then the method's advantage over visual baselines does not survive real-world graph extraction.","tokens_in":12956,"feed_emoji":"🧠","tokens_out":5385,"duration_ms":49209,"temperature":0.7,"pith_summary":"DyGEnc aims to show that a long sequence of textual scene graphs describing a changing environment can be compressed into a fixed handful of latent tokens—one in the default configuration—and still give a large language model enough temporal information to answer questions about what happened, in what order, and to which objects. The method fuses a graph encoder with a Q-Former sequence encoder and injects the compressed tokens as a soft prompt into a LoRA-tuned frozen LLM. On the STAR benchmark this reaches 0.89 average accuracy with Llama3.2-3B and 0.73 on AGQA2.0, beating video-based baselines by 15–25 percent on human–object interaction and sequence queries. The same pipeline can be run on raw video when scene graphs are extracted by foundation models, which the authors demonstrate on a wheeled-manipulator robot, though accuracy on that out-of-distribution setup drops to 0.30–0.34.","feed_headline":"One token per dynamic scene still yields 0.89 QA accuracy","feed_subtitle":"By encoding a sequence of scene graphs into a single latent token, an LLM answers interaction questions at 0.97 on STAR.","key_machinery":"The load-bearing mechanism is the Q-Former sequence encoder acting as a learned compressor. For each graph in the sequence, a graph encoder produces a token enriched with rotary positional encoding; then a fixed set of learnable query tokens attends across all graph tokens via cross-attention, collapsing an arbitrarily long graph sequence into a constant number of latent tokens. Because the number of output tokens does not depend on sequence length, the LLM context stays fixed regardless of how many scene graphs are observed. The graph encoder and Q-Former are trained together with a LoRA-tuned LLM, so the compression is optimized for the downstream question-answering objective rather than for reconstruction.","core_discovery":"The paper's central discovery is that the temporal content of a dynamic scene—who touched what, in what order—survives aggressive compression into a single learned latent token, provided the token is produced by a graph-aware encoder trained end-to-end with a language model. Given a sequence of textual scene graphs, DyGEnc encodes node and edge text with ModernBert, aggregates each graph with a GraphTransformer into one graph token, adds rotary positional encoding, and lets a Q-Former compress the whole sequence into $k$ query tokens (by default $k=1$). These tokens are projected into the LLM embedding space between <graph> and </graph> markers, and only a LoRA adapter plus the encoders are trained. With this setup DyGEnc reports 0.97 interaction and 0.90 sequence accuracy on STAR (0.89 average) and 0.73 average on AGQA2.0, outperforming the visual baselines it compares against by 15–25 percent on the interaction and sequence question types. Cross-attention maps show the single token attending to the frames that answer the query, evidence that reasoning is driven by relevant frames rather than memorized answers.","pith_inferences":["The 0.30–0.34 accuracy on automatically extracted graphs versus 0.89 on gold annotations suggests the practical bottleneck is graph extraction quality, not the encoder; improving automatic textual scene graphs would likely transfer directly to DyGEnc accuracy.","A single latent token appears to preserve enough information for interaction and sequence questions, but the ablation shows prediction and feasibility questions improve markedly with more tokens (16 tokens raise prediction from 0.77 to 0.91), implying different question types need different compression budgets.","If graph extraction improves to near-gold quality, graph-based memory could replace implicit video representations in embodied QA, because the reasoning component already reaches near-ceiling accuracy on curated graphs.","A direct test of the deployment claim would be to run DyGEnc on raw video from STAR rather than gold graphs, measuring the true end-to-end margin over visual baselines."],"forward_implications":["With gold scene-graph annotations, DyGEnc answers interaction questions at 0.97 and sequence questions at 0.90 on STAR, so temporal relational reasoning does not require per-frame pixel features.","Constant token count means graph sequences of arbitrary length fit into a fixed LLM context, which is a prerequisite for long-horizon robotic memory.","The 3B Llama model matches or beats the 8B model in most DyGEnc configurations, so the approach's accuracy is not dependent on the largest available LLM.","On AGQA2.0 open questions, DyGEnc outperforms prior methods by a wide margin, indicating the method distinguishes unique textual graph features rather than exploiting binary answer biases.","Using foundation-model graph extraction, the same DyGEnc model can answer questions from raw video on a robot, showing the architecture transfers beyond curated graph annotations."],"supporting_citations":[{"why":"Supplies the STAR benchmark with ground-truth scene graphs and situated QA pairs used for the main evaluation and ablations.","marker":"[10]"},{"why":"Supplies the AGQA2.0 benchmark, the second evaluation dataset with 2.27M balanced QA pairs and graph annotations.","marker":"[11]"},{"why":"Prior work encoding a single graph into latent tokens, which DyGEnc extends to sequences of graphs.","marker":"[7]"},{"why":"Q-Former architecture, reused as the sequence encoder with learnable query tokens for temporal compression.","marker":"[41]"},{"why":"LoRA, the parameter-efficient fine-tuning method applied to the frozen LLM's attention projections.","marker":"[43]"},{"why":"Llama 3 model family, the base LLMs (3.2-3B and 3.1-8B) that receive the latent graph tokens.","marker":"[44]"},{"why":"GraphTransformer, the graph neural encoder that aggregates each textual scene graph into a graph token.","marker":"[39]"},{"why":"ModernBert, the text encoder used to embed node and edge attributes before graph aggregation.","marker":"[37]"},{"why":"G-Retriever subgraph retrieval based on the Prize-Collecting Steiner Tree, used to reduce context in the robotic deployment.","marker":"[57]"}],"fun_headline_variants":["Single-token scene encoding beats visual models by 15-25%","One learned token condenses a whole dynamic scene for QA","Graph-aware encoder packs temporal scene into single token","LLM reads one token per scene, hits 0.97 on interaction QA","From scene graphs to one token: interpretable dynamic QA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that the ground-truth scene graph annotations used in the STAR and AGQA evaluations are similar in quality to the graphs the system would receive from automatic extraction in a real deployment; the 15–25 percent margin is measured against visual methods under gold annotations, and drops to 0.30–0.34 accuracy when graphs are extracted by foundation models on the robot benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Single-token scene encoding beats visual models by 15-25%","One learned token condenses a whole dynamic scene for QA","Graph-aware encoder packs temporal scene into single token","LLM reads one token per scene, hits 0.97 on interaction QA","From scene graphs to one token: interpretable dynamic QA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1815,"prompt_tokens":1000,"completion_tokens":815,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":728}},"tokens_in":616,"tokens_out":815,"duration_ms":7039,"temperature":1.0,"reasoning_tokens":728,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:47:25.572204+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run DyGEnc on the STAR validation videos using the same foundation-model graph extraction pipeline as the DRobot experiment instead of the provided annotations. If average accuracy falls to the 0.30–0.34 range rather than remaining near 0.89, then the method's advantage over visual baselines does not survive real-world graph extraction.","supporting_citations":[{"cited_title":"Agqa 2.0: An updated benchmark for compositional spatio-temporal reasoning,","cited_arxiv_id":null,"evidence_quote":"Supplies the AGQA2.0 benchmark, the second evaluation dataset with 2.27M balanced QA pairs and graph annotations."},{"cited_title":"G-retriever: Retrieval-augmented generation for textual graph understanding and question answering,","cited_arxiv_id":null,"evidence_quote":"Prior work encoding a single graph into latent tokens, which DyGEnc extends to sequences of graphs."}],"review_version":1}