{"id":"c9464595-aaa3-43df-88a4-7c02ca77cd7b","arxiv_id":"2501.00358","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Embodied VideoAgent augments an LLM-based video agent with persistent object memory built from egocentric video, depth, and pose, plus VLM-based memory updates, reporting gains on Ego4D-VQ3D, OpenEQA, and EnvQA.","lead":"Researchers built an AI agent that remembers objects in 3D scenes from first-person video plus depth and position data, updating its memory when actions occur. It reports improved accuracy on three embodied scene understanding benchmarks and shows uses in robot manipulation and simulated interaction.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OpenEQA headline gain is not established: the one-fifth subset is compared against full-benchmark baselines, and no ablation isolates the persistent-memory contribution from the stronger GPT-4o VQA backbone.","rationale":"The reader's weakest_assumption and my load-bearing concern are the same: the OpenEQA comparison mixes a one-fifth subset with full-benchmark baselines. That directly undermines the abstract's headline gain of 5.8%, which is a central quantitative claim. I agree fully with that diagnosis. My second point, lack of ablations to isolate the memory contribution, is related but secondary: even with a fair subset, the attribution of the gain to the new persistent memory architecture versus the stronger VQA backbone (GPT-4o) is not established. The reader also noted the missing ablations. I do not see an internal logical inconsistency in the system design; the VQ3D comparison is fairer because all methods use the same precomputed poses and depth, and the EnvQA comparison uses the same subset protocol for all tested methods. The central claim is plausible but currently conditional on releasing the subset and rerunning baselines. That is exactly the reader's verdict, so I recommend UNCHANGED rather than a harsher verdict. No independent mechanical check is possible from the text alone; the concrete test requires the authors' subset selection and baselines.","tokens_in":23440,"tokens_out":2388,"duration_ms":20218,"concrete_test":"Release the exact one-fifth OpenEQA subset (question IDs and scene splits) and rerun the borrowed full-set baselines (GPT-4 w/LLaVA-1.5, GPT-4 w/CG, Video-LLaVA, LLaMA-VID) on that identical subset. If the baselines' subset scores are within or above the reported margins, the 5.8% claim is not established. Additionally, run an ablation on at least OpenEQA and EnvQA that replaces the persistent object memory and VLM update with VideoAgent's original memory while keeping the identical GPT-4o VQA tool and LLM controller; if the gain over VideoAgent shrinks to near zero, the memory architecture is not the cause of the claimed improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of an OpenEQA gain of 5.8% is not supported by the presented evidence. Section 3.2.1 states that Embodied VideoAgent is tested on a randomly selected one-fifth subset of OpenEQA due to cost, while Table 2 lists GPT-4 w/LLaVA-1.5 (45.4/40.0/43.6) and GPT-4 w/CG (37.8/34.0/36.5) as borrowed from the original benchmark paper on the full set. Comparing a one-fifth subset score against full-set baselines assumes the subset is representative; the paper argues the subset is harder based on Video-LLaVA/LLaMA-VID drops, but it does not report the subset's composition, the borrowed baselines' performance on that exact subset, or equivalent per-split counts. If the subset skews toward harder scenes, the 47.0 ALL score does not entail a 5.8% gain over GPT-4 w/LLaVA-1.5's 43.6 on the full set. A secondary but load-bearing weakness is attribution: the paper provides no ablation that removes the new persistent object memory, VLM-based update, or history buffers while holding the VQA backbone (GPT-4o vs InternVL2-8B) fixed. Since Embodied VideoAgent (GPT-4o) uses a much stronger VQA backbone than VideoAgent, part or all of the OpenEQA/EnvQA gain over VideoAgent could be due to the backbone instead of the memory design. EnvQA reports only 200 questions per type with no variance or significance testing, and VQ3D gains are on one validation set without repeated-seed or statistical analysis. These issues weaken the causal claim that the memory architecture, not the underlying LLM/VLM, is responsible for the headline gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Embodied VideoAgent, an LLM-based multimodal agent that extends the prior VideoAgent system by constructing a persistent object memory from egocentric video, depth maps, and camera poses, and by updating that memory with a VLM when actions or activities are perceived. The agent also maintains action and visible-object history buffers and uses four perception tools plus seven embodied action primitives. It is evaluated on Ego4D-VQ3D object localization, OpenEQA embodied question answering, and EnvQA video question answering, with reported gains of 4.9%, 5.8%, and 11.7% respectively, and is further demonstrated on synthetic embodied interaction generation and a robot manipulation scenario. The central claim is that the persistent memory architecture, rather than the underlying LLM/VLM backbone, is responsible for the improvements.","tokens_in":23829,"tokens_out":9084,"duration_ms":73282,"significance":"If the empirical claims hold, the paper would demonstrate a practical and cost-effective way to handle dynamic 3D scene understanding by augmenting an LLM agent with structured persistent memory and VLM-based memory updates. The manuscript has several strengths: the VQ3D evaluation shares precomputed camera poses and depth maps across methods, making that comparison clean; the appendix provides detailed algorithms (object re-ID, memory update, tool prompts) and honest failure cases; and the application demonstrations, while qualitative, are suggestive of downstream utility. However, the headline OpenEQA number is not established because it compares a one-fifth subset score against full-benchmark baselines, and no ablation isolates the memory contribution from the stronger GPT-4o/InternVL2-8B backbone. The EnvQA and VQ3D results also lack statistical grounding. These issues are load-bearing for the paper's central claims, though they appear fixable with additional experiments.","major_comments":[{"comment":"The abstract's claim of a 5.8% gain on OpenEQA is not supported by the reported experiments. Embodied VideoAgent is evaluated on a randomly selected one-fifth subset of OpenEQA, while the GPT-4 w/LLaVA-1.5 (43.6 ALL) and GPT-4 w/CG (36.5 ALL) rows are borrowed from full-benchmark evaluations. The paper argues the subset is harder based on Video-LLaVA and LLaMA-VID drops, but this does not make a subset score comparable to a full-set score for different baselines. If the intended comparison is 47.0 (subset) against 43.6 (full set), the claimed 5.8% gain is not established; if the 5.8% refers to the gap between the GPT-4o and InternVL2-8B variants (47.0 vs 41.2), then it is not a gain over an external baseline. Please evaluate all baselines on the exact same subset, or run Embodied VideoAgent on the full OpenEQA set, and report per-split counts and standard errors.","section":"Section 3.2.1, Table 2"},{"comment":"No ablation isolates the contribution of the persistent object memory, VLM-based memory update, and history buffers from the choice of backbone models. Embodied VideoAgent (GPT-4o) differs from VideoAgent not only in the memory design but also in using GPT-4o as the LLM and InternVL2-8B for VQA, so the observed gains over VideoAgent could be due in part to the stronger backbone. Please provide an ablation that either runs VideoAgent with the same GPT-4o/InternVL2-8B tools or removes the persistent memory and update mechanism from Embodied VideoAgent while keeping all other components fixed. This is necessary to support the paper's attribution that 'the strong performance of Embodied VideoAgent mainly attributed to the precise frame localization using consistent object memory and history buffer.'","section":"Section 3.2.2, Table 2"},{"comment":"The EnvQA results are reported as per-category accuracies on 200 questions per type with no variance or significance testing, and the abstract's 11.7% gain is not derivable from the table. The largest gain is in Events (25.91 vs 10.19 for Video-LLaVA), but the Orders gain over VideoAgent is only 2.5 points (68.0 vs 65.5), which is within likely noise for 200 questions. Please state the aggregation rule used to compute the 11.7% (e.g., macro-average over the three categories) and provide confidence intervals, a paired test, or a bootstrap estimate. Without this, the 'significant performance gain' is not quantitatively supported.","section":"Section 3.2.3, Table 3"},{"comment":"The VQ3D headline is a 4.88-point absolute gain in Succ% (85.37 vs 80.49) on a single validation set. Given that the validation set is small (QwP% values imply roughly 80-90 queries), this difference could correspond to only a few queries. Please report the number of queries and provide a bootstrap confidence interval or a McNemar test to establish that the gain is not due to chance. As it stands, the '5%' improvement is not statistically grounded.","section":"Section 3.1, Table 1"}],"minor_comments":[{"comment":"The text says Embodied VideoAgent (image) surpasses EgoLoc by 5%, but Table 1 shows 85.37 vs 80.49, which is 4.88%; please use the exact value or say 'approximately 5%'.","section":"Section 3.1.2"},{"comment":"The note states that camera poses for Ego4D-VQ3D are estimated using COLMAP, but Section 3.1.1 says all VQ3D methods use the same precomputed poses and depth images provided by EgoLoc. Please clarify whether EgoLoc's poses are the COLMAP poses, or whether the COLMAP remark applies only to a different experiment.","section":"Section 2.2, Note on camera poses"},{"comment":"The sentence 'validated by the better performance of Embodied VideoAgent (+16.4% over Video-LLaVA on subset) over GPT-4 w/CG (+4.4% over Video-LLaVA on full set)' appears to reference GPT-4 w/CG, but the +4.4% value corresponds to GPT-4 w/LLaVA-1.5; GPT-4 w/CG actually scores below Video-LLaVA. Please correct this attribution.","section":"Section 3.2.2"},{"comment":"The caption says 'placing the lamb on a table' in the context of a lamp object; this should be 'lamp'.","section":"Figure 12 caption"},{"comment":"The paper uses both 'Embodied VideoAgent' and 'E-VideoAgent' interchangeably. Define the abbreviation at first use and use it consistently.","section":"Throughout"},{"comment":"The abstract's numerical gains (4.9%, 5.8%, 11.7%) are not all directly traceable to the tables; please provide the exact formulas or comparisons used for each number.","section":"Abstract and Tables 1-3"}],"recommendation":"major_revision","confidential_remarks":"The OpenEQA subset-vs-full comparison and the missing ablation are the central obstacles. They are fixable with additional experiments, so I do not recommend rejection. The paper's scope and contributions fit a computer vision / embodied AI venue, but the abstract currently overstates what is demonstrated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core design is a genuine extension of VideoAgent: persistent 3D object memory built from egocentric video plus depth and pose, a VLM-driven update that associates actions with object entries through visual prompting, and two history buffers. This is new relative to the cited prior work. The VQ3D evaluation is the cleanest part—all methods share EgoLoc's precomputed poses and depth, and Embodied VideoAgent (image) beats EgoLoc on Succ% (85.37 vs 80.49) by answering more queries, with a modest trade-off in Succ* and L2. That result stands on its own.\n\nThe soft spots are real. The OpenEQA headline gain of 5.8% is not established as stated. The paper tests on a random one-fifth subset and compares against GPT-4 w/LLaVA-1.5 and GPT-4 w/CG scores borrowed from the full benchmark. The paper argues the subset is harder because Video-LLaVA and LLaMA-VID drop on it, but that does not tell us what the borrowed baselines would score on that exact subset. If the subset skews hard, the 47.0 ALL score does not imply a 5.8% gain over 43.6. This is load-bearing for the abstract claim.\n\nSecond, there is no ablation that isolates the memory contribution. Embodied VideoAgent (GPT-4o) uses a much stronger VQA backbone than VideoAgent, so part or all of the OpenEQA/EnvQA gains over VideoAgent could come from the backbone. The InternVL2-8B variant is closer, but it is still a different backbone and no component ablation is reported. EnvQA is 200 questions per type with no variance or significance testing, and the VQ3D results are on one validation set. None of this kills the paper—the mechanism is plausible and the VQ3D comparison is fair—but the causal claim that the memory architecture drives the gains is not yet supported.\n\nThis is a solid system paper for people building embodied agents with tool-use memory. It deserves a serious referee: the design is coherent, the novelty is real, and the flaws are fixable with a properly matched subset evaluation and ablations. I would send it to review, but I would not take the OpenEQA number at face value until the authors rerun the baselines on their subset or release the subset and code.","headline":"Solid extension of VideoAgent with a fair VQ3D result, but the OpenEQA headline gain is undermined by a subset-vs-full comparison and missing ablations.","tokens_in":24387,"tokens_out":2855,"would_cite":false,"duration_ms":22872,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A memory agent that fuses egocentric video with depth and pose into a persistently updated object memory beats video-only models on dynamic 3D scene understanding (gains up to 11.7%).","keywords":["egocentric video","persistent object memory","embodied scene understanding","multimodal agent","depth and camera pose","VLM memory update","3D object localization","embodied question answering"],"falsifier":"Running Embodied VideoAgent and the exact same baselines on the full OpenEQA validation set, or ablating the VLM-based memory update and checking whether EnvQA Event and Order accuracies drop, would settle whether the reported gains come from the persistent memory and update mechanism.","tokens_in":23212,"feed_emoji":"🤖","tokens_out":10331,"duration_ms":86037,"temperature":0.7,"pith_summary":"The paper proposes Embodied VideoAgent, an LLM-based agent that answers questions about dynamic 3D scenes seen from an egocentric camera. Its central claim is that a persistent object memory built from video plus depth maps and camera poses, and updated automatically by a vision-language model whenever an action changes an object, gives large gains over both end-to-end video-language models and earlier video-memory agents. On three benchmarks the agent reports gains of 4.9% on Ego4D-VQ3D 3D localization, 5.8% on OpenEQA embodied question answering, and 11.7% on EnvQA long-form interaction question answering. A sympathetic reader should care because the result would mean that structured, updatable memory, not just a larger model, is what makes embodied scene understanding work in changing scenes.","feed_headline":"Persistent memory lifts egocentric 3D scene understanding up to 11.7%","feed_subtitle":"Combining video with depth and pose lets memory update as actions occur, beating three benchmarks.","key_machinery":"The load-bearing mechanism is the persistent object memory: a table of object entries, each holding an object id and category, a state description, relations to nearby objects computed from 3D bounding boxes, a 3D bounding box obtained by 2D-3D lifting using depth and camera pose, and visual features of both the object crop and its surrounding frame. Construction pipelines an open-vocabulary detector into 2D detections, lifts them to 3D boxes, and merges duplicates through a two-stage re-identification procedure that separates static from dynamic objects, using intersection and volume-similarity scores for the former and a visual-similarity threshold for the latter. The companion mechanism is the vision-language-model-based memory update: when the agent perceives an action, it renders each candidate object's 3D box onto the current frame and prompts a vision-language model to decide which memory entry is the action's target, then changes that entry's state and logs the action in a history buffer. This combined memory-plus-update structure is what the paper credits for precise temporal and spatial localization and for answering event, order, and state questions in dynamic scenes.","core_discovery":"Embodied VideoAgent's central discovery is that fusing egocentric RGB with embodied sensor readings (depth and camera pose) into a persistent per-object memory, and refreshing that memory when actions occur, enables a tool-calling agent to track objects in dynamic scenes more accurately than either end-to-end multimodal LLMs or the video-only memory baseline. Each memory entry holds an object id and category, a state label (\"normal\", \"open\", \"close\", \"in hand\"), related objects derived from 3D bounding boxes, a 3D bounding box lifted from 2D detections via depth and pose, and visual features of the object crop and its surrounding context. Re-identification merges new detections into existing entries using visual and spatial similarity scores, with separate handling for static and dynamic objects. When an action annotation indicates an activity over an object, the agent visually prompts the vision-language model with rendered 3D boxes to identify the target entry and programmatically updates its state, logging the action in a history buffer. In evaluation, this design yields the reported gains: 85.37% success on Ego4D-VQ3D, 47.0% on the OpenEQA subset, and 25.91, 68.00, and 35.50 on EnvQA event, order, and state categories.","pith_inferences":["Editorial extension: the claimed 5.8% OpenEQA gain would be on firmer ground if the full benchmark were scored with identical baselines; the paper's random one-fifth subset makes the comparison with full-set borrowed baselines an assumption about representativeness.","Editorial extension: if the memory update is the true cause of the EnvQA gains, then disabling the VLM-based update while keeping all other components should substantially reduce Event and Order accuracy; this ablation is not reported in the paper.","Editorial extension: the method's robustness to noisy poses suggests a natural testable extension of replacing ground-truth sensor inputs with monocular depth and pose estimators end-to-end, then measuring the same three benchmarks.","Editorial extension: the persistent memory design could be carried into long-horizon robot deployments where the same object is revisited over hours, a use case the paper illustrates only in short manipulation scenes."],"forward_implications":["Dynamic 3D scene understanding can be improved by explicit persistent memory rather than by scaling end-to-end video models, because the agent outperforms both on all three benchmarks.","Egocentric video alone is not sufficient: depth and camera poses contribute the geometry needed for reliable object tracking and 3D localization.","Associating perceived actions with target objects through vision-language-model visual prompting is a key ingredient for event and order understanding, where the paper reports the largest relative gains.","The same memory supports embodied behavior beyond question answering: persistent entries let a robot retrieve an occluded object, such as an apple hidden behind a box, and complete a pick-and-place task.","The agent architecture can generate synthetic user-assistant interaction episodes by having one LLM propose tasks and the memory-augmented agent execute them in a simulator."],"supporting_citations":[{"why":"The base agent whose temporal and object memory pipeline Embodied VideoAgent extends, providing the tool-usage and re-identification skeleton.","marker":"[7]"},{"why":"Defines the Ego4D dataset and the VQ3D 3D localization task that supplies one of the three headline benchmarks.","marker":"[11]"},{"why":"Supplies the EgoLoc baseline and the precomputed camera poses and depth images used for the VQ3D evaluation.","marker":"[31]"},{"why":"Provides the OpenEQA benchmark and the caption-based and scene-graph-based question-answering baselines used for comparison.","marker":"[32]"},{"why":"Defines the EnvQA benchmark of long-form embodied interaction question answering, including the Events, Orders, and States splits.","marker":"[8]"},{"why":"The open-vocabulary detector used to extract objects and categories when constructing the persistent object memory.","marker":"[4]"},{"why":"Supplies the two-second action annotations of the camera wearer that trigger the VLM-based memory update.","marker":"[63]"},{"why":"The visual prompting technique used by the VLM to associate an action with the correct object entry through rendered 3D boxes.","marker":"[58]"},{"why":"Provides the 2D-3D lifting procedure that converts 2D detections, depth, and pose into the 3D bounding boxes stored in memory.","marker":"[13]"}],"fun_headline_variants":["Persistent memory from video and sensors lifts 3D understanding up to 11.7%","Agent's persistent memory from video and sensors yields up to 11.7% gains","Fusing video with depth and pose builds memory that improves 3D understanding","Video+embodied sensors create persistent memory that updates on action"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline OpenEQA gain assumes that the randomly selected one-fifth subset of the benchmark behaves like the full set, so that baseline scores borrowed from the full benchmark remain valid for comparison.","fun_headline_variants_meta":{"raw":{"variants":["Persistent memory from video and sensors lifts 3D understanding up to 11.7%","Agent's persistent memory from video and sensors yields up to 11.7% gains","Fusing video with depth and pose builds memory that improves 3D understanding","Video+embodied sensors create persistent memory that updates on action"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001502,"raw_usage":{"total_tokens":6043,"prompt_tokens":984,"completion_tokens":5059,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":600,"completion_tokens_details":{"reasoning_tokens":4973}},"tokens_in":600,"tokens_out":5059,"duration_ms":34113,"temperature":1.0,"reasoning_tokens":4973,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:52:28.605396+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Running Embodied VideoAgent and the exact same baselines on the full OpenEQA validation set, or ablating the VLM-based memory update and checking whether EnvQA Event and Order accuracies drop, would settle whether the reported gains come from the persistent memory and update mechanism.","supporting_citations":[{"cited_title":"Egoloc: Revisiting 3d object localization from egocentric videos with visual queries","cited_arxiv_id":null,"evidence_quote":"Supplies the EgoLoc baseline and the precomputed camera poses and depth images used for the VQ3D evaluation."},{"cited_title":"Openeqa: Embodied question answering in the era of foundation models","cited_arxiv_id":null,"evidence_quote":"Provides the OpenEQA benchmark and the caption-based and scene-graph-based question-answering baselines used for comparison."},{"cited_title":"Learning video representations from large language models","cited_arxiv_id":null,"evidence_quote":"Supplies the two-second action annotations of the camera wearer that trigger the VLM-based memory update."}],"review_version":1}