{"id":"9d9db3b3-6a1d-4e1a-9cb6-5cb595b34f26","arxiv_id":"2505.24257","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new egocentric benchmark shows vision-language models fail at spatial reasoning across disjoint frames, falling 28 points behind humans and only improving sharply when handed ground-truth 3D coordinates.","lead":"This paper introduces Disjoint-3DQA, a benchmark that tests whether vision-language models can judge spatial relations between objects that are never seen together in the same video frame. It finds that even the best models lag humans by 28 points and gain 20 points when given exact 3D coordinates, pointing to a core weakness in building 3D scene memory from video.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark construct validity hinges on untested shortcuts; a text-only baseline and label-distribution report would settle whether DISJOINT-3DQA actually requires multi-frame 3D integration.","rationale":"The reader's weakest assumption identifies exactly the load-bearing premise: DISJOINT-3DQA must genuinely require cross-frame 3D integration for its performance numbers to support the bottleneck conclusion. My analysis of the paper's construction and results supports this: the per-category plot shows priors already explain the easiest categories, the harder categories are where the conclusions must be earned, and no control for priors, label imbalance, or single-frame utility is reported. This is the single most load-bearing concern because if it lands, both headline numbers (28-point human gap and 18-point oracle gain) lose their intended interpretation—the findings would still be interesting as a general VLM weakness on spatial questions, but not as evidence about constructing 3D representations from disjoint observations. The proposed concrete test is decisive: a text-only baseline directly measures how much of the task is solvable without any visual 3D integration, while label-distribution reporting checks whether raw accuracy numbers are even meaningful. I agree with the reader's conditional verdict: the benchmark has clear value and the core observation is reasonable, but these unaddressed confounds make the strong interpretation premature. The reader already flagged this premise and the missing chance baseline, so no verdict change is needed; the conditional status already encodes the need for these checks.","tokens_in":14370,"tokens_out":8711,"duration_ms":113385,"concrete_test":"Run a text-only condition: present the same DISJOINT-3DQA questions (object names and question text only, no frames, no markers) to the same set of models, and compare against the Marked video baseline separately for the Spatial Relationship and Size and Fit categories. Also report per-category label distributions (e.g., fraction of 'left' vs 'right', fraction where object A is larger) and compute the majority-class baseline accuracy. If text-only performance is within a few points of the video baseline on the hard categories, the benchmark is not actually requiring multi-frame visual integration and the central claim is undermined; if text-only is substantially lower and label distributions are balanced, the construct-validity concern is mitigated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the 18-point oracle gain and the 28-point human gap expose a bottleneck in VLMs' construction and maintenance of 3D scene representations over time—depends on the premise that DISJOINT-3DQA questions cannot be answered from language priors, label statistics, or single-frame cues. The paper's own per-category results (Figure 3b) show strong performance on Relative Distance and Relative Size, categories where commonsense priors such as 'a vase is smaller than a couch' suffice. The harder Spatial Relationship and Size and Fit categories must therefore carry the headline gap and the oracle gain, yet no chance baseline, label distribution, or single-frame ablation is reported. If, for example, the left/right directions in Figure 9 are imbalanced, a majority-class predictor could inflate the baseline; if a text-only prompt containing object names but no frames already achieves most of the multi-frame performance, the benchmark would be measuring general commonsense rather than 3D integration. Either outcome would change the interpretation of the oracle gain from '3D representation construction is the bottleneck' to 'models can compare numbers in text once given them, but fail at visual grounding for other reasons.' The 18-point oracle effect is real, but its attribution to missing 3D memory is only as strong as this untested premise.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DISJOINT-3DQA, a benchmark for egocentric spatial reasoning in which the two relevant objects are never co-visible in the same frame. The benchmark is built from Aria Synthetic Environments and contains 5,399 QA pairs across 1,668 scenes. The authors evaluate seven VLMs under three conditions: 2D-only with and without visual object marking, sparse 3D cues (textual camera trajectories and bird's-eye-view renderings), and a full 3D context oracle that provides ground-truth object centers, volumes, and trajectories. They report that humans reach 93.96 LLM-Match score, while the best model, GPT-4o with markers, reaches 65.60, leaving a gap of about 28 points. Sparse 3D cues give only 2-4 point gains, while the full 3D context gives an 18-point gain for GPT-4o. The paper concludes that current VLMs fail primarily in constructing and maintaining 3D scene representations over time from visual signals, rather than in reasoning over explicit 3D information.","tokens_in":14577,"tokens_out":4997,"duration_ms":62840,"significance":"If the central interpretation is sustained, this is a useful and timely benchmark contribution: it targets a genuinely under-tested capability, disjoint-frame spatial reasoning, and its controlled construction, ground-truth metadata, and planned code release are strengths. The oracle condition is transparently an upper bound, as the paper states, and the candidate is not circular in a harmful sense. The key claims, however, rest on a construct-validity premise that is not yet verified: that DISJOINT-3DQA questions cannot be answered through language priors, label statistics, or single-frame cues. The per-category results in Figure 3b already show strong performance on Relative Distance and Relative Size, categories where commonsense priors likely suffice. The current evidence also has reporting gaps, including an oracle condition tested on only one model and headline numbers in the abstract that do not match the tables. These issues are fixable within the manuscript's scope, and the benchmark itself appears to be a solid diagnostic artifact that could become important with the requested control experiments and corrected reporting.","major_comments":[{"comment":"The headline oracle gain is reported inconsistently. Table 4a reports 65.60 for GPT-4o with visual marking and 83.2 for the full 3D context, an increase of 17.6 percentage points; §5.2 calls this an '18% jump'; and the abstract claims a '20% performance increase'. These numbers should be reconciled, and all gains should be stated as absolute percentage points relative to an explicitly named baseline. As written, the abstract's 20% figure matches neither the absolute-point difference nor the relative increase from the marked baseline.","section":"Abstract and §5.2, Table 4a"},{"comment":"The full 3D context is evaluated only on GPT-4o. The paper's central attribution—that the bottleneck lies in constructing 3D representations rather than in reasoning over spatial metadata—cannot be established from one model. At minimum, the full 3D condition should be run on the strongest open-source model, Qwen 2.5-72B, which is already present in the sparse-cue rows of Table 4a, or the claim should be restricted to GPT-4o and reframed as a case study.","section":"Table 4a / §5.2"},{"comment":"The construct validity of DISJOINT-3DQA as a test of multi-frame 3D integration is not yet established. The paper's per-category results show high performance on Relative Distance and Relative Size, where commonsense priors such as 'a vase is smaller than a couch' may suffice. The harder Spatial Relationship and Size and Fit categories carry the headline gap and the oracle gain, yet no text-only baseline, no label-distribution baseline, and no single-frame baseline are reported. A text-only prompt with object names and the question, a majority-class predictor using the answer distribution, and a condition showing only one of the two object frames would settle whether the benchmark measures 3D integration or general commonsense and visual grounding. This is load-bearing because the interpretation of the 28-point human gap and the 18-point oracle gain depends on it.","section":"§5.1, §5.2, and Figure 3b"},{"comment":"The abstract's claim of a decline in accuracy from 60% to 30% as the temporal gap widens is not reported anywhere in the main text or appendix. Figure 5 shows performance declining with Euclidean distance between objects, not with temporal gap, and no analysis of accuracy versus number of frames or elapsed time is provided. The claim should either be supported with a dedicated temporal-gap analysis or removed from the abstract. If distance and temporal gap are correlated in this dataset, the distinction and the specific numbers should be made explicit.","section":"Abstract and §5.3, Figure 5"},{"comment":"All reported scores are single LLM-Match judgments without error bars, confidence intervals, or significance tests. Several headline differences are small, for example the 2.72-point gain from marking for GPT-4o in Table 3a and the 2.63-point gain from the BEV cue in Table 4a. Without variance estimates or repeated judge sampling, these differences may be within noise. The paper should report bootstrap confidence intervals or comparable measures, and ideally multiple LLM-Match judge runs, for the key comparisons that support the qualitative conclusions.","section":"Tables 3a and 4a / Appendix A.7"}],"minor_comments":[{"comment":"There are typos that should be fixed: 'accors' should be 'across' in Section 7, and 'Sythetic' should be 'Synthetic' in the Limitations section.","section":"Conclusion and Limitations"},{"comment":"The text refers to 'Table 3a' and 'Figure 2' in ways that do not match the displayed figure and table labels. For instance, the human performance comparison is in Figure 3a, and the per-category analysis is in Figure 3b, not 'Figure 2'. The numbering should be harmonized throughout.","section":"§5.1 and figure/table numbering"},{"comment":"The prompt section for marked objects is introduced with the heading 'Unmarked Objects', which is clearly a copy-paste error; it should be 'Marked Objects'. This makes the experimental protocol harder to follow.","section":"Appendix A.8"},{"comment":"The text states the dataset contains 1,668 scenes and an average of four questions per scene, but 5,399 QA pairs over 1,668 scenes gives an average of about 3.2 questions per scene. Also, the main text says 1,688 scenes are sampled, while Table 2 says 1,668; these numbers should be reconciled.","section":"§2 and Table 2"},{"comment":"Table 1 lists OpenEQA's task format as MCQ, but the paper's own §4 describes DISJOINT-3DQA as following OpenEQA's open-vocabulary, natural-language-answer format. This inconsistency should be corrected.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is a potentially valuable diagnostic, and the core idea is sound. The main risk is that the headline interpretation—that the gap and oracle gain show a specific 3D-representation-construction bottleneck—currently outruns the evidence. The missing control baselines (text-only, label distribution, single-frame) and the single-model oracle are the key issues; the abstract/table inconsistencies are smaller but should still be fixed. I would be comfortable with acceptance after these revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Disjoint-3DQA is a genuine new benchmark: it forces object pairs to never co-occur in any frame, which is a real gap in the existing egocentric spatial QA sets. The oracle 3D coordinate condition and the sparse-cue comparisons are a sensible design for isolating whether failures come from perception or from reasoning over given geometry. The strong oracle effect (18 percentage points on GPT-4o) and the 28-point human gap are worth taking seriously.\n\nThe paper is less careful than the idea deserves. The abstract says \"seven state-of-the-art VLMs\" but the evaluation table lists six; the abstract's 60% to 30% temporal-gap decline does not appear in the main text or appendix; and the \"substantial 20% performance increase\" is actually 18 points over the marked baseline in Table 4a. Those are fixable but they undermine trust in the headline numbers. There are also no error bars or multiple-run variance, and no chance baseline.\n\nThe bigger concern, which the stress-test note raises, is construct validity. The per-category results show models doing fine on Relative Distance and Relative Size, where a commonsense prior like \"a vase is smaller than a couch\" can often answer the question. The harder categories carry the gap and the oracle gain, but without a text-only baseline (object names and numbers, no frames) or a label-distribution report for each category, we can't confirm that the task is actually forcing multi-frame 3D integration. A text-only baseline that already hits most of the multi-frame performance would change the story from \"VLMs can't construct 3D representations\" to \"VLMs can compare numbers once the numbers are given.\" The appendix has a direction-label histogram for spatial relationship questions, but not a full breakdown by category, and no chance accuracy.\n\nTo be clear, showing a text-only baseline is cheap and I don't think it would kill the result; the 18-point oracle gain is real and the BEV/error analyses show some genuine spatial reasoning failures. But the paper needs that control before the bottleneck claim is airtight.\n\nOverall: this is a solid benchmark paper with a useful new constraint and a clear result, wrapped in an abstract that overstates some numbers. It deserves a serious referee, with requested revisions: fix the counts, report the temporal-gap analysis, add a text-only baseline and per-category chance levels, and release data/code. I'd take it to our reading group.","headline":"Disjoint-3DQA is a genuinely new benchmark that cleanly isolates disjoint-frame spatial reasoning, but the paper's headline numbers need a cleanup pass and the benchmark needs a few control conditions before the central bottleneck claim is airtight.","tokens_in":15174,"tokens_out":2134,"would_cite":true,"duration_ms":25631,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Current vision-language models lag humans by 28 points on spatial questions about objects that never share a frame, and only oracle 3D coordinates—not realistic sparse cues—close most of that gap.","keywords":["egocentric video","spatial reasoning","vision-language models","disjoint frames","3D scene representation","benchmark","video question answering","embodied AI"],"falsifier":"Scramble the temporal order of the frames while keeping the same ground-truth 3D coordinates in the input; if a VLM still reaches the same near-83% score, the model is solving from coordinates alone and the disjoint-frame temporal integration is not what the benchmark is testing.","tokens_in":14144,"feed_emoji":"🧭","tokens_out":8361,"duration_ms":87720,"temperature":0.7,"pith_summary":"The paper introduces DISJOINT-3DQA, a 5,399-question benchmark built from egocentric RGB-D video in which every question concerns two objects that are never visible in the same frame. Across seven vision-language models, the best system (GPT-4o with visual object markers) scores 65.6% on an LLM-judged accuracy metric while humans score 94.0%, a 28-point gap that widens as the time between the two object sightings grows. Feeding models realistic sparse cues—text camera trajectories or bird's-eye-view renderings—raises scores only 2 to 4 points. But feeding them ground-truth 3D coordinates and volumes lifts GPT-4o to 83.2%, an 18-point jump over the marked baseline. The authors read this contrast as evidence that the bottleneck is not reasoning over geometry once it is given, but constructing and maintaining a 3D scene representation from temporally disjoint visual signals.","feed_headline":"True 3D coords lift VLM spatial reasoning by 20 points","feed_subtitle":"Sparse cues barely help, showing VLMs lack persistent 3D scene memory—humans lead by 28 points.","key_machinery":"The machinery is the benchmark's controlled input contrast. DISJOINT-3DQA enforces disjointness by including an object pair (A, B) only when their visibility spans TA and TB are disjoint, so no single frame shows both objects. For each pair, a world-to-camera transform TB places object B at the origin of its own egocentric frame, and the relative offset dAB = T_B(cA − cB) defines the ground-truth relation used to generate questions. Evaluation runs the same questions under four input conditions: raw frames, frames with visual markers (Set-of-Mark style), frames plus sparse 3D cues (text trajectory or BEV), and frames plus oracle full 3D metadata. The performance difference between the sparse and oracle conditions is what carries the argument that the missing capacity is 3D representation construction, not geometric reasoning.","core_discovery":"The central discovery is a diagnostic contrast. On questions that require integrating spatial information across frames where the two objects never co-occur, current VLMs solve the easy perceptual cases but cannot recover global 3D layout: accuracy falls from about 60% to near 30% as the temporal gap between sightings grows, and the best model still sits 28 points below human performance. Adding realistic sparse 3D cues—camera trajectories in text or top-down bird's-eye-view images—moves scores by only a few points. Adding ground-truth 3D centers and volumes in text produces a jump of about 18 to 20 points, bringing GPT-4o to 83.2%. Because the oracle cue makes the needed geometry explicit, the remaining failure is upstream: models are not building, updating, or holding onto an internal metric representation of the scene from partial egocentric observations.","pith_inferences":["The authors leave implicit that the oracle-condition gain may overstate the benefit of a real 3D detector, since ground-truth coordinates also remove perception errors; a noisy or partially occluded detector would recover only part of the 20 points.","A testable extension would be to replace the static-scene assumption with moving objects, asking whether models can update and maintain a 3D representation when the geometry itself changes over time.","The large drop on 'Spatial Relationship' questions suggests that part of the human advantage is not 3D geometry but the ability to read egocentric language against a remembered layout; a probe with allocentric wording could separate these."],"forward_implications":["Improving multi-frame VLMs will require architectural or training changes that build persistent 3D scene representations, not just better 2D video understanding.","The 18-to-20-point oracle gain sets an upper bound for what a perfect egocentric 3D perception module could contribute to spatial question answering.","Sparse, realistic 3D cues such as BEV maps are not sufficient as-is; models need aligned cross-view reasoning to use them.","Accuracy degradation with object distance and temporal gap means long-horizon spatial memory is a distinct weakness of current models.","The benchmark can serve as a measurable challenge for embodied AI assistants operating from first-person video."],"supporting_citations":[{"why":"Supplies the LLM-Match evaluation metric and the embodied-QA task format that DISJOINT-3DQA adopts for open-ended answers.","marker":"(Majumdar et al., 2024)"},{"why":"Provides Set-of-Mark visual grounding, whose red-circle object markers the paper shows improve VLM performance.","marker":"(Yang et al., 2023)"},{"why":"SpatialRGPT-Bench, a prior spatial benchmark that does not enforce object non-co-visibility, serving as the comparison point for the new disjointness constraint.","marker":"(Cheng et al., 2024)"},{"why":"VSI-Bench, the closest egocentric spatial benchmark, which lacks explicit 3D priors and motivates DISJOINT-3DQA's controlled 3D cues.","marker":"(Yang et al., 2024b)"},{"why":"Object permanence tracking, the related setting that DISJOINT-3DQA deliberately hardens by requiring reasoning over never-co-visible objects.","marker":"(Tokmakov et al., 2021)"},{"why":"Video-MME, the multi-frame video benchmark used to select the evaluated VLMs for strong video understanding.","marker":"(Fu et al., 2024)"},{"why":"Foundational spatial-cognition work that motivates the benchmark's question taxonomy (direction, distance, size, fit, scene description).","marker":"(Landau and Jackendoff, 1993)"}],"fun_headline_variants":["VLMs lack persistent 3D memory: oracle coords gain 20 pts","Spatial reasoning across frames: VLMs lag humans by 28 points","3D coords rescue VLM reasoning, sparse cues don't","Cross-frame VLM IQ drops 30% without oracle coords","VLMs can't track layouts over time; humans lead by 28"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark measures 3D integration only if the questions cannot be answered from language priors or single-frame cues; the paper's own category results show models already do well on relative size and distance, where priors like 'a vase is smaller than a couch' may suffice.","fun_headline_variants_meta":{"raw":{"variants":["VLMs lack persistent 3D memory: oracle coords gain 20 pts","Spatial reasoning across frames: VLMs lag humans by 28 points","3D coords rescue VLM reasoning, sparse cues don't","Cross-frame VLM IQ drops 30% without oracle coords","VLMs can't track layouts over time; humans lead by 28"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000181,"raw_usage":{"total_tokens":1298,"prompt_tokens":926,"completion_tokens":372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":542,"completion_tokens_details":{"reasoning_tokens":275}},"tokens_in":542,"tokens_out":372,"duration_ms":4685,"temperature":1.0,"reasoning_tokens":275,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:28:16.543123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Scramble the temporal order of the frames while keeping the same ground-truth 3D coordinates in the input; if a VLM still reaches the same near-83% score, the model is solving from coordinates alone and the disjoint-frame temporal integration is not what the benchmark is testing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-Match evaluation metric and the embodied-QA task format that DISJOINT-3DQA adopts for open-ended answers."},{"cited_title":"what” and “where","cited_arxiv_id":null,"evidence_quote":"Foundational spatial-cognition work that motivates the benchmark's question taxonomy (direction, distance, size, fit, scene description)."}],"review_version":1}