{"id":"dde9cff4-14bd-4cf3-8431-be18826bd11e","arxiv_id":"2608.11017","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"R4DSG builds an anchor-relative 4D scene graph memory from RGB egocentric video and shows improved object-centric QA accuracy over text retrieval.","lead":"This paper introduces R4DSG, a memory system that turns long egocentric video into a compact scene graph by tracking objects relative to stable anchors like tables and fridges. It reports better answers to object-centric questions, particularly 'when' questions, than a text-based retrieval baseline.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported QA gains rest on an unvalidated perception/association stack: masks, persistent tracks, and Eq. 5 anchor-change events are never checked against ground truth, so the memory documents that drive the gains may not faithfully encode object state.","rationale":"The reader's weakest_assumption matches the single most load-bearing risk I see: the entire empirical case depends on the automatic memory being faithful, but no component-level evidence is supplied. I do not find an internal contradiction in the method or a fatal flaw in the QA protocol. The paper is appropriately cautious about the single-subject split and the exploratory why result, and the No-Transition control is a reasonable ablation. The lack of a ground-truth perception/tracking check is nevertheless decisive for interpreting the central claim: until the memory content is validated, the 6.7/12.5-point gains could in principle be driven by plausible-sounding but unfaithful documents. This is exactly the kind of condition that should keep the paper conditional rather than accepted, and it does not soften the reader's verdict.","tokens_in":14436,"tokens_out":7109,"duration_ms":65864,"concrete_test":"On a random sample of Day1-A1-JAKE clips, manually annotate ground-truth object tracks, static/dynamic roles, and anchor-change event intervals. Run the R4DSG pipeline to produce predicted masks, persistent IDs, and Eq. 5 events; compute mask IoU, track IDF1, and event precision/recall. Then build two QA memories, one from predicted outputs and one from the ground-truth annotations, and compare retrieval-plus QA accuracy. If accuracy on the ground-truth memory is substantially higher, the reported gains partly reflect compensation for perception errors; if the two accuracies match, the upstream concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The method's claimed contribution is the construction of a queryable object memory, and the QA model only sees memory documents generated by that construction. Yet the fidelity of the construction is never measured. In Sec. 3.4.3, persistent identity is decided by hand-defined association rules (semantic compatibility, coarse 3D size continuity, relative 3D position continuity, neighborhood consistency), and Eq. 5 emits an anchor-change event only when the new anchor-relative state 'remains stable across time'; no thresholds or validation of these criteria are given. The evaluation in Sec. 4 reports end-to-end multiple-choice accuracy on 255 questions, with no ground-truth comparison for masks, 3D lifts, persistent IDs, static/dynamic roles, or anchor-change events. If the segmentation or lifting is noisy, or if the association policy splits/merges tracks or misassigns anchors, the retrieval documents can contain fluent but false transitions. An LLM answering from such documents can then appear to succeed while the memory itself is not object-grounded. This is load-bearing because the claim is not merely that R4DSG gets higher QA accuracy, but that a relative 4D object memory is the reason; without validating the memory, the mechanism is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents R4DSG, a pipeline that converts long egocentric RGB video into a queryable memory of relative 4D scene graphs. It separates static anchors from dynamic objects, maintains persistent object tracks via hand-defined association rules, writes anchor-relative temporal events (Eq. 5) and segment-level retrieval documents (Eq. 7), and retrieves top-8 documents for a Qwen-based multiple-choice QA system. Evaluated on 255 object-centric questions from EgoLifeQA A1_JAKE, it reports a 6.7-point overall gain and a 12.5-point gain on when questions over EgoRAG-Text under question-only retrieval, plus ablations and an exploratory why-memory extension. The authors explicitly scope all claims to this single-subject split and acknowledge that the outputs are not a human-verified perception benchmark.","tokens_in":14604,"tokens_out":3917,"duration_ms":34149,"significance":"If the result holds, the paper would demonstrate a practical way to build object-centric memory from monocular wearable video without global alignment or depth input, compressing 828 clips into a 0.58-MB JSON memory. The contribution is a representation and a deterministic memory-writing operator rather than a trained model, which makes the pipeline reproducible in principle and separates the method from end-to-end learned approaches. The paper is honest about its limitations: it explicitly limits empirical claims to a single-subject split, describes the why-memory result as exploratory, and states that outputs are not a human-verified perception benchmark. The No-Transition control also provides a useful first attempt at isolating the effect of persistent identity and anchor transitions. However, the strength of the evidence is not yet commensurate with the central claim: the evaluation is small, the adapted baselines are not official reproductions, and the upstream perception stack that generates the memory documents is never validated against ground truth. The significance of the mechanism is therefore plausible but not established.","major_comments":[{"comment":"The load-bearing component of the pipeline is the automatic construction of persistent object tracks and anchor-change events, yet the manuscript never evaluates masks, 3D lifts, persistent IDs, static/dynamic role assignments, or the Eq. (5) anchor-change criterion against ground truth or even a manual inspection sample at scale. The QA model consumes only the memory documents written by this stack, so end-to-end accuracy cannot distinguish a faithful memory from one that produces fluent but false transitions. This is exactly the correctness-risk concern raised by the stress-test note, and it lands. Please add a quantitative validation of the perception/association modules—at minimum ID precision/recall and anchor-change precision/recall on a labeled subset of the released Day1 stream—and provide the operational thresholds used for the 'stable across time' criterion in Eq. (5) and for the association rules in Sec. 3.4.3.","section":"Sec. 3.4.2–3.4.4, Eq. (5)"},{"comment":"The evaluation uses 255 questions from a single subject with no error bars, confidence intervals, or significance tests. A 6.7-point overall gain is roughly 17 questions, and the 12.5-point gain on the 72-question when subset is roughly 9 questions; these differences may be within chance variation. Report statistical significance, confidence intervals, or a per-day breakdown to establish that the gains are not noise. In addition, the adapted EMQA-style and AMEGO-inspired baselines are not official reproductions, and the implementation details are not given in the paper; please provide the exact adaptation procedure (or use official checkpoints) so that the comparison is verifiable and fair.","section":"Sec. 4.1, Table 2"},{"comment":"The No-Transition control is the only direct test of the paper's central mechanism, but its specification is incomplete. The text says it removes 'persistent cross-segment identity and anchor-transition writing' while sharing 'cached visual evidence and local relations,' but it is not clear how the control's retrieval documents are constructed without persistent identity or whether the retrieval protocol is identical to the main method. The memory-granularity ablation in Table 4 is explicitly described as not isolating the causal mechanism, which further weakens the connection between representation and gain. Please specify the No-Transition control in full—document schema, retrieval procedure, and how identity removal is implemented—so that the reader can verify that the comparison isolates anchor-transition memory from other differences.","section":"Sec. 4.2.3, Table 4; Sec. 4.1 No-Transition control"}],"minor_comments":[{"comment":"The text alternates between 'SAM 3-style'/'SAM 3D-style' and 'SAM 3'/'SAM 3D'; make the naming consistent and clarify whether the actual released models are used or only style-inspired reimplementations, since the reproducibility section promises a concrete pipeline.","section":"Figure 2 and Sec. 3.4.2"},{"comment":"State the version/release date of Qwen3.5-27B and confirm that the same prompt template and decoding settings are used across all methods; otherwise the comparison may be sensitive to prompt-formatting differences rather than memory design.","section":"Sec. 4.1"},{"comment":"The Y_k field is described as optional explanation-oriented, but the main Retrieval+ results in Table 2 presumably have Y_k empty; state this explicitly in the main text or in the table caption so that the reader knows which fields were active for the headline numbers.","section":"Sec. 3.3, Eq. (6)"},{"comment":"The Event-only row reports 27.8 on When, which is close to the Plain RAG number; add a sentence interpreting why Event-only underperforms Episodic-only, as this seems to undercut the intuitive claim that anchor-change events are the key to when questions.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a borderline case. The authors are appropriately transparent about the evaluation scale and the lack of perception validation, but the central mechanism—that a relative 4D object memory, rather than the perception stack or the retrieval protocol, drives the observed QA gains—remains unverified. If the authors can add even a small perception validation study on a labeled subset of the released data and provide significance testing or additional subjects, the paper could become acceptable. The adapted baselines need more implementation detail to rule out unfair comparison. The work fits the multimedia memory/QA scope, although the novelty over AMEGO and existing scene-graph memory approaches is incremental rather than dramatic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look. The idea is genuinely different from the 3D scene-graph line: instead of building a globally aligned map, R4DSG keeps stable anchors and tracks dynamic objects relative to them, then writes segment-level retrieval documents that preserve object identity and anchor-relative transitions. That is a sensible way to build a queryable memory from free-motion monocular RGB video, and it is not in the prior work they cite. The staged pipeline—semantic episodes, promptable video segmentation, RGB-only 3D lifting, conservative association, memory writing—is clearly described, and the bag-transfer example makes the abstraction concrete. The paper is also honest: they state the empirical claims are limited to a single-subject public split and say the outputs are not a human-verified perception benchmark.\n\nThe soft spots are real. The evaluation is 255 questions from one subject, with no error bars or significance tests, and the adapted baselines are not official reproductions. The why result sits on 15 questions and is correctly labeled exploratory. The bigger issue is the one the stress-test flags: the memory construction itself is never validated. Masks, 3D lifts, persistent tracks, and the Eq. 5 anchor-change events are all produced by hand-defined rules without reported thresholds, and the paper reports only end-to-end QA accuracy. If those components are noisy, the retrieval documents can contain fluent but false transitions, and the QA gain could be an artifact of the LLM papering over bad memory. The No-Transition control does some work—it shares cached visual evidence and local relations, so the gap suggests persistent identity and anchor transitions matter—but it does not validate fidelity, only that the extra machinery changes the score.\n\nWho is this for? People working on egocentric video memory, long-video QA, or structured scene representations. They will find the formulation useful and the limitations instructive. I would send it to reviewers rather than desk reject: the idea is novel enough and the write-up clear enough that a serious referee can push for the right validation. I would not take the QA numbers at face value until the perception stack is checked against ground truth and at least one more subject or day split is added.","headline":"A genuinely new memory representation for egocentric video QA, clearly described and honestly limited, but the evidence is too thin to prove the memory mechanism works.","tokens_in":15230,"tokens_out":2402,"would_cite":true,"duration_ms":21887,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Object memory beats text memory for egocentric video Q&A.","keywords":["egocentric video","3D scene graph","temporal memory","object-centric question answering","retrieval-augmented generation","RGB-only 3D lifting","persistent object tracking","anchor-relative representation"],"falsifier":"Manually label persistent object identities and anchor-change events on a subset of the same 828-clip Day1-A1-JAKE stream, then compare the predicted tracks and memory entries to those labels; if the tracks are frequently split or merged and the anchor-relative states are wrong, yet end-to-end QA accuracy stays at 39.6 percent, then the claimed mechanism of persistent identity plus anchor-relative transitions is not what drives the reported gains.","tokens_in":14165,"feed_emoji":"🕶️","tokens_out":7631,"duration_ms":119570,"temperature":0.7,"pith_summary":"This paper tries to establish that a long egocentric RGB video can be converted into a compact, queryable memory organized around persistent objects and their movements relative to stable scene anchors, and that this memory supports object-centric question answering better than text- or caption-based retrieval. The proposed method, R4DSG, never builds a globally aligned 3D map; it only needs monocular, unposed RGB video, and it records where an object is relative to stable references such as a table or fridge, when that relation changed, and what local context explains the change. On 255 object-related questions from EgoLifeQA, the memory outperforms the text-based EgoRAG-Text baseline by 6.7 points overall and 12.5 points on when questions under question-only retrieval. A reader should care because wearable assistants and AR systems need exactly this kind of long-horizon object memory, and the result suggests scene-graph structure can be made practical without depth or posed camera input.","feed_headline":"Object memory beats text memory for egocentric video Q&A","feed_subtitle":"A relative 4D scene graph lifts object-question accuracy 6.7 points and when-questions 12.5 points on EgoLifeQA.","key_machinery":"The central object is the relative 4D scene graph memory: a time-indexed collection of anchor-relative object states rather than a globally consistent 4D reconstruction. Within it, the load-bearing identity is the temporal event $e_i=(p,[t_s,t_e],a^-,a^+,r_i)$, written when persistent object $p$ moves from source anchor $a^-$ to destination anchor $a^+$ over span $[t_s,t_e]$ with a local rationale $r_i$. The machinery that carries the argument is the combination of episode-conditioned promptable video segmentation (SAM 3-style), RGB-only 3D lifting (SAM 3D-style) to get coarse relative location and extent, conservative persistent-track association across frames, static-anchor inference, and the deterministic memory-writing operator $h$ that aggregates frame graphs and events into segment-level retrieval documents. This mechanism keeps errors local, avoids global alignment, and yields a memory that is compact enough to retrieve (0.58 MB for roughly seven hours of footage) and structured enough to answer object-state questions.","core_discovery":"R4DSG's central discovery is that the right unit for long-horizon egocentric memory is not a clip, caption, or global 3D reconstruction but a temporal event recording a persistent object's transition from one stable anchor to another. Each frame contributes a small graph of semantic and relative 3D attributes; the system links these into persistent tracks using semantic compatibility, size and position continuity, and neighborhood consistency; and it writes a memory entry whenever a dynamic object's dominant anchor changes and the new state remains stable over time. The resulting memory stores time spans, places, activities, object states, and interaction cues in segment-level documents that can be retrieved directly for question answering. The authors argue that this anchor-relative, identity-preserving design is what lets the system answer where, when, and why questions that captions and transcripts do not preserve.","pith_inferences":["A testable extension: benchmark the upstream perception stack (mask quality, 3D lift error, track precision and recall) against manual persistent-object annotations on the same footage; the argument would be strengthened if QA accuracy tracks perception accuracy rather than staying flat when tracks are corrupted.","The anchor-relative design implies that wearable assistants do not need to solve simultaneous localization and mapping to answer where and when questions, which could make memory construction feasible on-device; a direct stress test would be running the pipeline on multi-day continuous capture and measuring cross-day identity continuity, which the paper lists as future work.","If the gains replicate beyond the single A1_JAKE subject, the memory format could also serve embodied multimedia agents that pool experiences across users, since the anchor-relative state is defined locally and does not require a shared world coordinate system.","One implicit reading is that caption-centric memory loses exactly the relational change signal that object-centric questions rely on; an even stronger test would compare R4DSG against an oracle that retrieves the correct clip, isolating whether the gains come from retrieval precision or from answer synthesis over structured evidence."],"forward_implications":["Under question-only retrieval, R4DSG scores 39.6 percent overall versus 32.9 percent for EgoRAG-Text on the 255-question object-centric subset, a 6.7-point gain, and 43.1 percent versus 30.6 percent on the 72 when questions, a 12.5-point gain.","Removing persistent cross-segment identity and anchor-transition writing (the No-Transition control) drops accuracy to 34.9 percent overall and 34.7 percent on when, indicating the graph transitions themselves carry the benefit rather than object/relation serialization alone.","The memory compresses 828 clips into 134 retrieval documents totaling 0.58 MB, making the representation practical as a persistent memory substrate for wearable assistants.","Option-blind retrieval with no explanation fields still outperforms the AMEGO-inspired active-memory baseline by 1.2 points overall and 5.6 points on when, showing the benefit does not depend on exposing answer options.","Adding explanation-oriented fields raises why-subset accuracy from 33.3 percent to 40.0 percent, though the authors flag this result as exploratory due to the 15-question subset size."],"supporting_citations":[{"why":"Supplies the SAM 3-style promptable video segmentation and temporal propagation that produces episode-scoped masks and tracklets.","marker":"[2]"},{"why":"Supplies the SAM 3D-style RGB-only relative 3D lifting that yields coarse 3D location, extent, and pairwise spatial cues without depth or point clouds.","marker":"[4]"},{"why":"Provides the EgoLifeQA benchmark and the EgoRAG-Text baseline that define the evaluation set and the main comparison point.","marker":"[38]"},{"why":"Supplies the EMQA-style episodic memory baseline that the paper adapts as an option-blind retrieval comparison.","marker":"[7]"},{"why":"Supplies the AMEGO-inspired active-memory baseline that the paper adapts and outperforms under conservative protocols.","marker":"[11]"}],"fun_headline_variants":["R4DSG memory lifts object-question scores 6.7 points in egocentric video","Relative 4D scene graphs outscore captions for egocentric object QA","Object memory gains 6.7 points, when-questions 12.5, in long video QA","Egocentric video QA improved by anchor-relative 4D scene graph memory","Scene graph memory answers object questions captions miss in egocentric video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes its automatic perception stack, video segmentation, RGB-only 3D lifting, and the hand-defined association rules for linking observations into persistent tracks, produces accurate enough object identities and anchor-relative states, since the paper never evaluates these modules against ground truth and only reports end-to-end QA accuracy.","fun_headline_variants_meta":{"raw":{"variants":["R4DSG memory lifts object-question scores 6.7 points in egocentric video","Relative 4D scene graphs outscore captions for egocentric object QA","Object memory gains 6.7 points, when-questions 12.5, in long video QA","Egocentric video QA improved by anchor-relative 4D scene graph memory","Scene graph memory answers object questions captions miss in egocentric video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00057,"raw_usage":{"total_tokens":2747,"prompt_tokens":1049,"completion_tokens":1698,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":665,"completion_tokens_details":{"reasoning_tokens":1589}},"tokens_in":665,"tokens_out":1698,"duration_ms":10958,"temperature":1.0,"reasoning_tokens":1589,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:57:07.040451+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually label persistent object identities and anchor-change events on a subset of the same 828-clip Day1-A1-JAKE stream, then compare the predicted tracks and memory entries to those labels; if the tracks are frequently split or merged and the anchor-relative states are wrong, yet end-to-end QA accuracy stays at 39.6 percent, then the claimed mechanism of persistent identity plus anchor-relative transitions is not what drives the reported gains.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the EgoLifeQA benchmark and the EgoRAG-Text baseline that define the evaluation set and the main comparison point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the EMQA-style episodic memory baseline that the paper adapts as an option-blind retrieval comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the AMEGO-inspired active-memory baseline that the paper adapts and outperforms under conservative protocols."}],"review_version":1}