{"id":"e6a57572-c617-4b4b-bd7a-1b6118b89eb4","arxiv_id":"2608.05747","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"GST-Bench measures global spatial awareness from egocentric video and finds that current VLMs fall far below humans, with the gap traceable mainly to cross-frame integration rather than single-image perception.","lead":"GST-Bench is a new video question-answering benchmark that tests whether vision-language models can build a global top-down map of a scene from long egocentric camera footage. On its 12 spatial tasks, the best zero-shot model scores 42.68 while humans score 79.08, and targeted fine-tuning lifts an 8B model above all zero-shot proprietary systems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Top-Down Selection subtasks appear single-frame solvable, so the aggregate GST-Bench score mixes non-global matching tasks with true integration tasks; the headline gap should be recomputed on integration-requiring tasks only.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: Top-Down Selection and Trajectory Selection may be solvable from a single frame or local matching, undermining the claim that every GST-Bench task requires cross-frame global reasoning. I agree this is the most load-bearing threat to the central claim. The human baseline concern is real but secondary: even a generous error bar on 20 samples per task cannot close a 36-point gap, so the headline gap would likely persist even with better baselines. The construct-validity problem is more fundamental because it directly challenges whether the benchmark measures global spatial awareness at all. However, the benchmark does contain genuinely cross-frame tasks by construction (target-invisible object localization), and the fine-tuned model's large gain on those tasks suggests the phenomenon is real. Thus the appropriate verdict remains CONDITIONAL: the authors should either demonstrate that Top-Down Selection is not single-frame solvable, or explicitly scope the benchmark claims to the object-localization and self-localization competencies that enforce integration by construction. My read does not change the reader's conditional verdict.","tokens_in":13453,"tokens_out":6099,"duration_ms":59029,"concrete_test":"For each Top-Down Selection sample (easy, medium, hard), evaluate the same models twice: once with the full exploration video and once with a single randomly sampled frame (or a short 2-second clip) from that video. If single-frame accuracy matches full-video accuracy within noise for any difficulty level, that subtask is single-frame solvable and should be removed or reweighted. Then recompute the overall human-model gap and the proprietary-vs-open-source ordering on the remaining truly integration-requiring subtasks (Egocentric Direction, Egocentric Distance, Global Position, Top-Down Position/Orientation). The central claim survives only if the gap remains substantial and the cross-frame integration bottleneck persists after excluding single-frame-solvable tasks.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that GST-Bench measures global spatial awareness requires that every subtask demand cross-frame integration. The Introduction asserts this ('ensuring that every task requires integrating information across views'), but Section 3.3 enforces global reasoning by construction only for object-localization tasks, where the target is absent from the current view. Top-Down Selection (easy/medium/hard) instead supplies an exploration video plus four candidate whole-scene top-down maps. A model can identify the correct map by matching a single frame's visible room layout or object arrangement to one candidate; no long-horizon spatial memory is required when a single frame contains a distinctive layout. The paper's own Section 4.2 attributes open-source gains 'mainly' to Top-Down Selection (easy/medium) and calls these tasks 'coarse scene identification', effectively conceding they are not diagnostics of global spatial awareness. Since the headline Score is the arithmetic mean over all 12 subtasks, the reported gap (42.68 vs 79.08) blends single-frame-solvable matching tasks with genuinely integrative tasks. The conclusion that 'current VLMs cannot maintain globally consistent spatial representations from long egocentric video' is therefore not established by the aggregate benchmark alone. Separately, the Section 4.3 local-vs-global comparison is not fully controlled: Local-Video and Local-Image replace the current view with one that contains the target, simultaneously removing the integration requirement and changing the query image, so the claimed 'cross-frame integration bottleneck' conflates target visibility with integration difficulty.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GST-Bench introduces a synthetic video-based VQA benchmark for global spatial awareness, spanning 12 subtasks across self-localization, object localization, and scene structure understanding. The data are generated in OmniGibson/BEHAVIOR-1K with off-trajectory current views, target-invisible constraints for object-localization tasks, and top-down map variants, yielding 2,762 human-verified questions. The authors evaluate 22 VLMs and report that the strongest zero-shot model, Gemini-3-Pro, scores 42.68 versus a human score of 79.08. They also construct local variants that remove the cross-frame requirement and attribute the proprietary-model failure mainly to cross-frame integration, and they show that fine-tuning Qwen3-VL-8B on GST-Train improves its score from 25.89 to 53.52.","tokens_in":13640,"tokens_out":5745,"duration_ms":51643,"significance":"If the central claim holds, the benchmark fills a real gap: it targets cross-view spatial integration rather than single-image spatial relations, uses precise numerical metrics, and leverages simulator geometry for exact labels, avoiding the circularity risk of model-generated ground truth. The human verification stage, the broad model coverage, and the local/global diagnostic design are further strengths. The central result, however, depends on two load-bearing assumptions: that every subtask genuinely requires cross-frame integration, and that the human baseline is a reliable estimate. Both assumptions need additional evidence before the headline gap can be interpreted as measuring global spatial awareness rather than a mixture of task formats.","major_comments":[{"comment":"The claim in Section 1 that 'every task requires integrating information across views' is only enforced for object-localization tasks. Section 3.2 shows that Top-Down Selection (easy/medium/hard) takes an exploration video plus four candidate top-down images; a single distinctive frame can reveal the room layout or object arrangement needed to choose the correct map. Section 4.2 itself states that the open-source advantage is 'mainly driven by the Top-Down Selection (easy/medium) subtasks' and describes these as 'coarse scene identification,' which effectively concedes that these subtasks are not diagnostics of cross-frame integration. Because the overall Score in Eq. (5) is the arithmetic mean over all 12 subtasks, the headline gap (42.68 vs. 79.08) mixes single-frame-solvable matching tasks with genuinely integrative tasks. Please recompute the aggregate score and the human/model comparison on the set of tasks that strictly require cross-frame integration, and empirically test single-frame-solvability of Top-Down Selection and Trajectory Selection by feeding only one frame (or a short clip) to models.","section":"Section 3.2 / Eq. (5)"},{"comment":"The human baseline of 79.08 is reported as 'Human Level (20 samples/task)' with no protocol details, no participant information, no selection criteria, and no error bars. Per-task human scores such as EDv=75.00, EDist=41.50, and Ori=85.00 are proportions over only 20 samples, so the reported 36.4-point gap relative to Gemini-3-Pro may be statistically fragile; for example, on EDistv the human score (41.50) is essentially matched by Gemini-2.5-Pro (42.00). Please provide the full human evaluation protocol, sample sizes per task, and confidence intervals, and verify that the headline gap remains statistically significant after restricting the comparison to tasks that genuinely require cross-frame integration.","section":"Table 1"},{"comment":"The local-vs-global comparison is not fully controlled. In Local-Video and Local-Image, the current view is replaced by a view in which the target is visible; this changes both target visibility and the particular viewpoint/camera pose of the query. Improvements from Global to Local-Video can therefore reflect a more favorable query view rather than the mere removal of the cross-frame requirement. In addition, Table 2 covers only the two semantic tasks EDs and EDists, so the broader conclusion that proprietary models 'mainly fail at cross-frame spatial integration' is not directly tested for the visual modality, global position, orientation, or top-down tasks. Please hold the query viewpoint fixed while adding/removing the target through controlled rendering, and extend the local variants to additional subtasks to support the stated diagnosis.","section":"Section 4.3 / Table 2"}],"minor_comments":[{"comment":"Because GST-Train is generated with the same simulation pipeline and asset family as GST-Bench, the 27.63-point fine-tuning gain partly reflects distribution matching; the authors disclose that the scenes are disjoint, but they should explicitly state this limitation and report performance on held-out scenes from different simulators or with different rendering conditions.","section":"Section 4.4"},{"comment":"The threshold sets C_d, C_a, and C_p are fixed without a sensitivity analysis; since the overall Score is the mean over these thresholds, a short robustness check would strengthen the claim that the reported ranking is not an artifact of threshold choice.","section":"Equations (1)-(4)"},{"comment":"Human verification is described as checking answerability and target identifiability, but not as independently verifying the automatically generated geometric labels; the paper should state this explicitly, since label correctness is otherwise guaranteed only by simulator geometry.","section":"Section 3.3"},{"comment":"The random-guessing baselines for the distance, angle, and point metrics are not derived in the text; please describe how the per-subtask random baselines are computed, including the assumed answer distributions.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"I recommend major revision. The benchmark construction and the breadth of evaluation are valuable, and the reported failure mode is likely real, but the paper's headline claim currently rests on an aggregate score that includes clearly single-frame-solvable subtasks and on a human baseline with no statistical support. Both issues are fixable with additional analyses and reporting, so the manuscript is within scope for a major revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a solid benchmark paper with a real contribution, and the model-human gap is real, but the paper overclaims that every task requires cross-frame integration, and the human baseline is too thin to carry the headline number.\n\nWhat's new: GST-Bench combines off-trajectory query views, target-invisible object localization, and top-down scene alignment, with precise angle/distance/point metrics instead of coarse left/right judgments. The systematic evaluation of 22 VLMs, including the controlled local variants, is the most interesting part. The finding that proprietary models jump roughly 39 points when the target is visible while open-source models often do not is a clean, useful result.\n\nThe soft spots are real. Top-Down Selection (TDS) easy and medium tasks can largely be solved from a single frame by matching room layout or object arrangement to one of four candidate maps. The paper says 'every task requires integrating information across views,' but Section 3.3 explicitly enforces global reasoning by construction only for object-localization tasks. The authors' own Section 4.2 notes open-source gains are 'mainly driven by' TDS easy/medium and calls those 'coarse scene identification.' Since the overall score is a simple mean over 12 subtasks, the 42.68 vs 79.08 headline blends single-frame-solvable matching with genuinely integrative tasks. Recompute the gap on the integration-requiring tasks before drawing conclusions about long-horizon spatial memory.\n\nThe human baseline is also thin: 20 samples per task, no protocol or error bars. For a benchmark whose headline is a gap, that is a weakness, even if the direction of the gap is probably right. The local-vs-global comparison is not fully controlled: Local-Video replaces the current view with one containing the target, changing both target visibility and the query image, so the gain conflates removing the integration requirement with making the query easier. The fine-tuning experiment is disclosed as same-pipeline distribution matching, which is fair, but it limits the claim that GST-Train teaches general ability.\n\nWho is this for: the embodied AI and VLM evaluation community. It deserves a serious referee. The benchmark construction is careful, the metrics are precise, and the stress test on cross-frame integration is a fixable revision rather than a fatal flaw. I'd want the TDS tasks either reclassified or justified as integration-requiring, and a proper human baseline with more samples and variance before the headline gap is published as definitive.","headline":"A genuinely useful new benchmark for global spatial awareness in video, but the aggregate score overstates the integration claim because the scene-matching subtasks are partly single-frame solvable, and the human baseline is too thin to carry the headline gap.","tokens_in":14297,"tokens_out":1820,"would_cite":true,"duration_ms":16891,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GST-Bench shows current vision-language models cannot assemble a globally consistent spatial map from long egocentric video, with the strongest zero-shot model scoring 42.68 against a human baseline of 79.08.","keywords":["global spatial awareness","video understanding","vision-language models","embodied AI","spatial reasoning benchmark","top-down mapping","cross-frame integration","egocentric video"],"falsifier":"A concrete test would be to take GST-Bench's object-localization questions and run a model that receives only one randomly sampled frame from the exploration video plus the current view: if it still scores far above the random baseline on those questions, then the target-invisible constraint is not preventing single-frame solutions. Similarly, re-running Top-Down Selection with the exploration video replaced by a single frame would show whether that subtask measures cross-frame integration at all; if single-frame accuracy matches video accuracy, the subtask should be excluded from the global-reasoning claim.","tokens_in":13178,"feed_emoji":"🧭","tokens_out":4784,"duration_ms":37125,"temperature":0.7,"pith_summary":"This paper introduces GST-Bench, a video question-answering benchmark that tests whether vision-language models can build a global spatial understanding of a scene from a long egocentric exploration video. Across 22 models, the best zero-shot score is 42.68 versus 79.08 for human evaluators, a gap the authors trace to cross-frame integration rather than single-image perception. A controlled local variant shows that proprietary models improve sharply when the target object is visible in the current view, while open-source models improve little or even degrade. The paper also releases GST-Train, and fine-tuning a small open model on it raises its GST-Bench score from 25.89 to 53.52, above all zero-shot proprietary models. The finding matters because embodied agents need exactly this ability to navigate, remember, and retrieve objects after losing sight of them.","feed_headline":"VLMs fail to build global maps from video: 42.68 vs human 79.08","feed_subtitle":"New benchmark forces models to integrate egocentric video into global top-down maps; strongest model trails humans by 36 points.","key_machinery":"The benchmark itself is the machinery: synthetically generated egocentric exploration videos from simulated indoor scenes, with query viewpoints sampled off the video trajectory and, for object-localization tasks, a hard guarantee that the target object is invisible from the query view. Three abstraction levels of top-down images (photo-realistic bird's-eye view, occupancy-style map, and bare floor plan) let the benchmark test whether egocentric observations can be aligned to an explicit global representation. The controlled local variants (Global, Local-Video, and Local-Image) isolate whether a model's failure is perceptual or integrative by progressively removing the need for cross-frame reasoning.","core_discovery":"The central discovery is a measurable, systematic failure: current VLMs do not maintain a globally consistent spatial representation from long-horizon egocentric video. GST-Bench asks models to localize a novel viewpoint on a top-down map, infer where a target object is when it is absent from the current view, and match egocentric motion to a global trajectory. The strongest zero-shot model reaches 42.68, less than 55% of the human score of 79.08, and most open-source models sit near random guessing. Ablations that make the target visible in the query view lift proprietary models by up to 39 points, showing that their bottleneck is cross-frame integration, whereas open-source models stay weak even in the local setting. Targeted fine-tuning on GST-Train lifts Qwen3-VL-8B from 25.89 to 53.52, surpassing all zero-shot proprietary models.","pith_inferences":["A direct extension the paper does not pursue is whether the same gap appears in real-world egocentric video, since GST-Bench is entirely synthetic; a real-video probe would test whether simulation captures the bottleneck.","Top-Down Selection and Trajectory Selection subtasks may be solvable by matching a single frame's room layout to a candidate map, so the claim that every GST-Bench task strictly requires cross-frame integration likely overstates the benchmark's uniformity; re-analyzing per-task single-frame solvability would sharpen the claim.","If the integration bottleneck is architectural, explicit spatial memory or mapping modules may close the gap faster than simply scaling context length."],"forward_implications":["If the gap is real, current VLMs cannot reliably support embodied tasks that require remembering object locations after they leave view, such as household retrieval.","The disentanglement result implies that improving cross-frame spatial integration, not just single-image perception, is the priority for proprietary-scale models.","Open-source models need gains at both local perception and integration; local benchmarks alone will not predict global performance.","GST-Train demonstrates that targeted supervision can narrow the gap, so future training data design should include explicit global map supervision.","The low human score on metric distance suggests absolute distance estimation is intrinsically hard and should be scored with tolerance-aware metrics."],"supporting_citations":[{"why":"Supplies the simulated indoor scenes and object assets from which exploration videos and top-down images are generated.","marker":"[11]"},{"why":"Provides the Mean Relative Accuracy protocol for distance answers and a prior video spatial benchmark whose design GST-Bench extends.","marker":"[28]"},{"why":"Gemini-3-Pro is the strongest zero-shot model evaluated; its 42.68 score anchors the headline human-model gap.","marker":"[7]"},{"why":"Qwen3-VL is the open-source backbone family evaluated and the model fine-tuned on GST-Train.","marker":"[2]"},{"why":"InternVL3.5 is the strongest open-source family tested, used to show open-source models score near random.","marker":"[26]"},{"why":"RoboBrain2.5 is an embodied-understanding model that fails to beat its general-purpose backbone, supporting the claim that embodied tuning does not fix global reasoning.","marker":"[24]"},{"why":"Robix is another embodied-tuned model evaluated, showing the same inherited blind spot at larger scale.","marker":"[5]"}],"fun_headline_variants":["VLMs fall short on global spatial reasoning: 42.68 vs humans' 79.08","GST-Bench: VLMs can't map egocentric video to global views","Video VLMs struggle to build mental maps from long egocentric footage","Global spatial awareness gap: VLMs score 42.68, humans 79.08","New benchmark reveals VLMs miss global scene structure in video"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's central claim rests on the assumption that every GST-Bench question truly requires integrating information across video frames and cannot be answered from a single frame or by retrieving a matching frame; for Top-Down Selection and Trajectory Selection this is not fully enforced, and the human baseline is estimated from only 20 samples per task.","fun_headline_variants_meta":{"raw":{"variants":["VLMs fall short on global spatial reasoning: 42.68 vs humans' 79.08","GST-Bench: VLMs can't map egocentric video to global views","Video VLMs struggle to build mental maps from long egocentric footage","Global spatial awareness gap: VLMs score 42.68, humans 79.08","New benchmark reveals VLMs miss global scene structure in video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1444,"prompt_tokens":942,"completion_tokens":502,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":398}},"tokens_in":558,"tokens_out":502,"duration_ms":4656,"temperature":1.0,"reasoning_tokens":398,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T00:12:11.046497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test would be to take GST-Bench's object-localization questions and run a model that receives only one randomly sampled frame from the exploration video plus the current view: if it still scores far above the random baseline on those questions, then the target-invisible constraint is not preventing single-frame solutions. Similarly, re-running Top-Down Selection with the exploration video replaced by a single frame would show whether that subtask measures cross-frame integration at all; if single-frame accuracy matches video accuracy, the subtask should be excluded from the global-reasoning claim.","supporting_citations":[{"cited_title":"Gemini 3: Introducing the latest gemini ai model from google.https://blog.google/products/gemini/ gemini-3/, 2025","cited_arxiv_id":null,"evidence_quote":"Gemini-3-Pro is the strongest zero-shot model evaluated; its 42.68 score anchors the headline human-model gap."}],"review_version":1}