{"id":"e9d64547-8a0e-40a3-8255-ea73ace46b48","arxiv_id":"2507.10548","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"EmbRACE-3K provides a photorealistic, closed-loop embodied benchmark with stepwise reasoning annotations, and fine-tuning Qwen2.5-VL on it improves embodied task success in simulation.","lead":"EmbRACE-3K is a new dataset of over 3,000 language-guided embodied tasks in photorealistic Unreal Engine environments, with about 26,000 decision steps annotated with actions and reasoning traces. It is a candidate benchmark for testing whether large vision-language models can move from passive video understanding to interactive, first-person decision-making.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Stage-4 Gemini rationales are post-hoc: the dataset's grounding claim and the reasoning-supervision gains depend on unverified retrospective CoT, not online perception.","rationale":"The reader correctly identified Gemini-generated supervision as the weakest assumption. I partially agree rather than fully agree: the instructions in Stage 2 are manually verified, which mitigates the instruction-generation half of the concern. The sharper risk is the Stage-4 rationales, which are generated after the full trajectory is known, with no reported manual validation. If those rationales encode hindsight, the benchmark's central claim of temporally grounded 'thinking' supervision and the no-thinking ablation result are undermined. However, the paper's action-level training signal still comes from human demonstrations, so the SFT+RL gains may survive even if the rationales are imperfect. Because the concern is real but not yet demonstrated, and the paper currently provides no data, code, or splits to check it, the reader's CONDITIONAL verdict remains appropriate. My concrete audit would settle whether the rationales are actually grounded; if they are, the concern dissolves, and if they are not, the authors would need to either re-annotate or weaken the reasoning-supervision claim. The abstract's 'below 20%' phrasing also conflicts with the Basic-task rows in Table 2, but that is a presentation issue rather than the main load-bearing risk.","tokens_in":12691,"tokens_out":6701,"duration_ms":88890,"concrete_test":"Conduct a grounding audit on 100 sampled decision steps spanning all task types. For each step, show annotators only the instruction, the action history up to that step, and the egocentric frame at that step, masking all future frames and the task's final goal. Ask three independent annotators whether the published Gemini rationale for that step is inferable from this information alone, and require at least 70% agreement. If the pass rate falls below 70%, the rationales contain privileged trajectory-level information; the authors should re-annotate the affected tasks with concurrent think-aloud human rationales, retrain Qwen2.5-VL-sft-rl on the re-annotated subset, and compare held-out SR/GDE. If the retrained model shifts by more than 10% relative, the reported SFT+RL gains depend on ungrounded retrospective annotations.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is that EmbRACE-3K provides 'fine-grained, temporally grounded annotations that closely align perception with decision-making' (Abstract) and 'decision-level supervision tightly aligned with the perception-action cycle' (§4.2, Stage 4). But Stage 4 generates each stepwise rationale by giving Gemini the full egocentric video and the entire action trajectory after the demonstration is complete. This lets Gemini 'explain' an action using knowledge of the final goal, the global map, and future observations that were not available to the agent at the moment the action was chosen. The collected rationales may therefore be post-hoc rationalizations rather than genuine online reasoning. The fine-tuned model is trained to reproduce these traces, so the reported SFT+RL improvements (§5, Tables 2–3) and the no-thinking vs sft-only comparison (§6.3) could partly reflect imitation of fluent hindsight language rather than acquisition of embodied, online reasoning. The central claim that fine-tuning on EmbRACE-3K yields substantial embodied reasoning improvements is thus contingent on the rationales being inferable from the egocentric observations available at each step. The paper reports manual verification for instructions (Stage 2) but does not report any validation of the Stage-4 rationales, and the fact that Gemini is also an evaluated baseline makes this more than a stylistic concern: the dataset may be partially distilling Gemini's priors instead of environment-grounded decision processes.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EmbRACE-3K, a dataset of over 3,000 language-guided embodied tasks in photorealistic Unreal Engine environments, with egocentric observations, grounded actions, and step-wise natural language rationales. The authors benchmark GPT-4o, Gemini 2.5 Pro, and Qwen2.5-VL in zero-shot settings and fine-tune Qwen2.5-VL-7B with supervised learning followed by GRPO reinforcement learning. They report that the fine-tuned model outperforms the proprietary baselines and that step-wise reasoning annotations improve decision quality. The central claims are that current VLMs are weak at embodied interaction (with all zero-shot models supposedly below 20% success) and that training on EmbRACE-3K yields substantial embodied reasoning improvements.","tokens_in":13052,"tokens_out":4894,"duration_ms":56423,"significance":"If these claims are validated, EmbRACE-3K would be a valuable resource: it provides large-scale, photorealistic, closed-loop embodied tasks with human demonstrations and step-level annotation, filling a gap relative to passive video benchmarks and code-generation-only embodied datasets. The SFT+RL pipeline and the no-thinking versus sft-only comparison are informative for the community. However, the current manuscript contains an internal contradiction in the headline zero-shot claim, incomplete reporting of evaluation sizes and error bars, and a potentially circular annotation process: Gemini generates both the instructions and the stepwise rationales and is also evaluated as a baseline. The paper's central contribution depends on these rationales being genuinely grounded in online perception, but no validation is provided. These issues need to be resolved before the benchmark's validity and the reported improvements can be fully credited.","major_comments":[{"comment":"The claim that \"in zero-shot settings, all models achieve success rates below 20%\" is contradicted by Table 2: in-domain, GPT-4o reaches 53.6% SR on Basic and Gemini 2.5 Pro reaches 76.4% on Basic and 39.3% on Exploration. Even if the claim is intended to apply only to the three named challenge dimensions or to out-of-domain tasks, the current wording is false and must be qualified precisely, with the table rows cited.","section":"Abstract / §1 / Table 2"},{"comment":"Tables 2 and 3 report success rate, GDE, SSPL, Steps, and Timeout Rate without any indication of test-set size, number of evaluation episodes, or variance across runs. This makes it impossible to assess whether differences such as the sft-rl versus GPT-4o comparisons are significant. Additionally, §6.2 defines success as reaching \"within 300 meters of the target,\" which is implausibly large for an embodied indoor/outdoor benchmark and appears to be a unit error; the success thresholds and on-policy termination criteria must be stated explicitly and per task type.","section":"§6.2 / Tables 2-3"},{"comment":"The stepwise rationales are generated by Gemini after receiving the complete egocentric video and the entire action trajectory, so they may be post-hoc rationalizations rather than online, perception-conditioned reasoning. The paper reports manual verification for instructions (Stage 2) but not for Stage-4 rationales. Because Gemini is also an evaluated baseline, the dataset may partly distill Gemini's priors, and the claimed reasoning-annotation gains (no-thinking vs. sft-only) could reflect imitation of fluent hindsight language rather than acquisition of embodied online reasoning. Please provide evidence that each rationale is inferable from the observations available at the corresponding step, for example via human ratings or by withholding future frames during rationale generation.","section":"§4.2 Stage 4 / §6.3"},{"comment":"The GRPO training description is incomplete: the accuracy reward is not defined per task type, and the phrase \"trajectory-level reward shaping\" in §6.1 is never specified. The reader cannot tell how success is judged during RL, how the distance threshold is applied, how multi-stage subgoal completion is scored, or how interaction actions (open, pick, drop) are rewarded. Please specify the reward functions and hyperparameters so that the RL results are reproducible.","section":"§5.2 / §6.1"}],"minor_comments":[{"comment":"The dataset name is written \"EmRACE-3K\" in the abstract but \"EmbRACE-3K\" in the title and body; please unify.","section":"Abstract"},{"comment":"The units of GDE are stated as centimeters but the success threshold is \"300 meters\"; please check the unit consistency and define a per-task spatial threshold (and what constitutes a \"Finish action\").","section":"§6.2"},{"comment":"The task type distribution percentages appear to sum to more than 100% (50+12+13+8+5+12+17 = 117); please verify the pie chart labels or the legend.","section":"Figure 4"},{"comment":"The checkmark/cross notation in the table is not defined in the caption; please add a legend or spell out \"yes/no\".","section":"Table 1"},{"comment":"The GRPO objective is typeset with a broken line break after \"JGRP O(θ)\"; fix the equation formatting.","section":"§5.2"},{"comment":"The manuscript does not state whether the dataset, code, and evaluation scripts will be publicly released beyond a project page; please add an availability statement.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The main risk is that the benchmark's central contribution rests on the Stage-4 rationales being genuinely grounded, and the current paper provides no validation against the post-hoc-hindsight concern. The abstract's blanket \"below 20%\" claim is also inconsistent with the presented tables. If the authors can supply the missing validation and fix the reporting issues, the paper could become a useful contribution to embodied VLM evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: EmbRACE-3K is a credible dataset and the SFT+RL training recipe delivers real gains, but the paper overstates its difficulty and the stepwise rationales are less grounded than claimed. Worth a serious referee, not desk rejection.\n\nWhat is genuinely new is the combination: photorealistic indoor and outdoor UnrealZoo scenes, online closed-loop evaluation, human demonstrations, and stepwise visual grounding plus natural-language rationales. That combination is not in ALFRED, Octopus, or the other cited benchmarks. The training results are also plausible: Qwen2.5-VL-7B fine-tuned with SFT+RL beats GPT-4o and Gemini 2.5 Pro on several out-of-domain tasks, and the no-thinking vs. sft-only comparison does suggest the reasoning annotations contribute something. Those are real contributions.\n\nBut the soft spots are substantial. The abstract's \"all models achieve success rates below 20%\" is directly contradicted by Table 2, where GPT-4o hits 53.6% and Gemini 76.4% on in-domain Basic. Maybe they meant out-of-domain or only the hard categories, but as written it is simply false. There are no error bars or test-set sizes anywhere, and the \"300 meters\" success radius in §6.2 looks like a typo: GDE is reported in centimeters, so a 300-meter threshold would mean nearly any position counts as success in an indoor scene. That needs fixing before the numbers mean anything.\n\nThe deeper issue is the Stage 4 annotation pipeline. Gemini receives the complete egocentric video plus the entire action trajectory after the demonstration is over, and then explains each action. That is hindsight, not online reasoning: the model can use knowledge of the final goal and future frames that were unavailable at decision time. The stress-test note is right. Training on these rationales may partly distill Gemini's priors rather than teach embodied perception-action coupling. The paper validates the Stage 2 instructions manually but reports no validation for Stage 4 rationales, and the fact that Gemini is also an evaluated baseline makes the circularity more than stylistic.\n\nIs this fatal? Not quite. The dataset's behavioral trajectories and instructions are human-generated or human-verified, and the SFT+RL gains could survive even if the rationales are post-hoc. But the paper's central framing—\"decision-level supervision tightly aligned with the perception-action cycle\"—is not supported by the current annotation design. The authors should either validate that the rationales are inferable from the egocentric observations at each step, or reframe them as post-hoc explanations and temper the claims accordingly.\n\nWho is this for? Embodied AI researchers who want a photorealistic, closed-loop benchmark with dense annotations. If the dataset, code, and splits are released and the metrics fixed, it could be a useful resource. I would send it to peer review, with the clear expectation that the abstract be corrected, the evaluation reported with variance and proper thresholds, and the rationale-generation pipeline either redesigned or honestly characterized.","headline":"Useful photorealistic embodied VLM benchmark with a plausible SFT+RL recipe, but the zero-shot claim is contradicted by its own tables and the Gemini-generated rationales are post-hoc, not online reasoning.","tokens_in":13549,"tokens_out":2110,"would_cite":false,"duration_ms":27614,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"EmbRACE-3K, a 3,000-task photorealistic embodied benchmark, reports zero-shot VLM success below 20% and improves a 7B model beyond larger baselines after SFT and RL.","keywords":["embodied reasoning","vision-language models","benchmark dataset","spatial reasoning","reinforcement learning","chain-of-thought","navigation","multi-stage planning"],"falsifier":"Rebuild EmbRACE-3K's supervision with human-written instructions and rationales on the same captured trajectories, then rerun the zero-shot evaluation and the SFT+RL training; if the below-20% zero-shot numbers and the training gains largely disappear or change direction, the benchmark's difficulty and its utility are artifacts of Gemini's annotations rather than of embodied interaction itself.","tokens_in":12448,"feed_emoji":"🤖","tokens_out":7806,"duration_ms":83955,"temperature":0.7,"pith_summary":"This paper sets out to show that the gap between vision-language models' strong static-image performance and their weak embodied performance can be quantified and narrowed. It introduces EmbRACE-3K, a benchmark of more than 3,000 language-guided tasks and 26,000 decision steps in photorealistic indoor and outdoor environments, with each step annotated by an egocentric view, a discrete action, and a natural-language rationale. Zero-shot, current models such as GPT-4o, Gemini 2.5 Pro, and Qwen2.5-VL-7B all score below 20% success on the benchmark, exposing short-sighted exploration, spatial-semantic drift, and target forgetting. Fine-tuning Qwen2.5-VL-7B on the dataset with supervised learning and then reinforcement learning raises success rates in all categories and improves out-of-domain robustness, which the paper attributes to the step-wise reasoning annotations plus trajectory-level reward shaping. If the result holds, the dataset provides both a diagnostic test for embodied reasoning and a training resource for building embodied agents.","feed_headline":"3,000 embodied tasks drop zero-shot VLMs below 20% success","feed_subtitle":"Training a 7B vision-language model on the photo-realistic tasks more than doubles out-of-domain success rates.","key_machinery":"The central machinery is the per-step annotation unit: each decision in a trajectory pairs an egocentric RGB frame with the executed discrete action and a natural-language rationale expressing the agent's intent, so perception, reasoning, and action are aligned at the same time step. On top of this, training uses a two-stage recipe: supervised fine-tuning that teaches the model to emit <think> rationale and <action> tokens, then GRPO reinforcement learning with a rule-based reward that checks format and action accuracy and compares a group of candidate rollouts. The step-wise rationale is what lets the model learn perception-conditioned decision-making instead of end-to-end action mapping; the RL stage is what generalizes it to unseen scenes.","core_discovery":"The paper claims that current vision-language models, however strong on static image and video understanding, lack the embodied capabilities that online egocentric interaction demands, and that this gap can be measured and partly closed. EmbRACE-3K provides 3,000+ language-guided tasks with 26,000 decision steps in photorealistic indoor and outdoor environments, each step carrying an egocentric frame, a discrete action, and a natural-language rationale; on this benchmark, GPT-4o, Gemini 2.5 Pro, and Qwen2.5-VL-7B all score below 20% success zero-shot. Fine-tuning Qwen2.5-VL-7B with supervised learning on the rationales followed by GRPO reinforcement learning raises success rates across Exploration, Dynamic Spatial-Semantic Reasoning, and Multi-stage Goal Execution, in some out-of-domain categories surpassing the larger proprietary models. The authors take the SFT-only versus SFT+RL comparison as evidence that explicit step-wise reasoning supervision and trajectory-level reward signals both contribute, with RL supplying the out-of-domain robustness that pure imitation lacks.","pith_inferences":["If Gemini's stepwise rationales carry model priors rather than environmental grounding, the benchmark's difficulty and the SFT gains could both be inflated; a control version with human-authored rationales on identical trajectories would separate these.","The same annotation pipeline should transfer to continuous-control and real-robot settings; if it does, trajectory-level RL plus rationale supervision could become a general recipe rather than a simulator-specific one.","An independent test set with object layouts and instructions sampled from a different generator would clarify whether the out-of-domain robustness comes from learned exploration strategies or from similarity to training maps."],"forward_implications":["Current VLMs, fine-tuned only for passive visual tasks, fail at information-seeking behavior: they fixate on immediate cues, lose targets that leave view, and let spatial references drift as the camera moves.","Step-wise rationale supervision is doing real work: removing <think> annotations lowers success on spatial-semantic and multi-stage tasks compared with SFT that keeps them.","RL is the generalization lever: SFT-only models drop sharply out-of-domain, while SFT+RL retains more success and lower goal-distance error in new scenes.","The EmbRACE-3K-trained 7B model outperforms GPT-4o and Gemini 2.5 Pro on several out-of-domain task categories, so smaller open models can surpass larger closed models once trained on interaction-centric data."],"supporting_citations":[{"why":"Supplies the photorealistic simulation environments and the low-level first-person control API used for all data collection and evaluation.","marker":"[33]"},{"why":"Defines the Qwen2.5-VL architecture that serves as the base model for supervised fine-tuning and reinforcement learning.","marker":"[3]"},{"why":"The GPT-4o baseline against which EmbRACE-3K's zero-shot difficulty and the fine-tuned model's gains are compared.","marker":"[17]"},{"why":"The Gemini 2.5 model used both as a zero-shot comparison baseline and as the generator of task instructions and step-wise rationales.","marker":"[6]"},{"why":"The ALFRED benchmark that EmbRACE-3K positions itself against on step-wise visual grounding and online closed-loop interaction.","marker":"[23]"},{"why":"Provides the GRPO group-relative policy optimization algorithm adopted for the reinforcement-learning training stage.","marker":"[11]"},{"why":"The training framework on which the GRPO reinforcement-learning stage is implemented.","marker":"[4]"},{"why":"The training framework used to run the supervised fine-tuning stage.","marker":"[32]"},{"why":"Supplies the chain-of-thought reasoning formulation that EmbRACE-3K extends to step-level embodied annotations.","marker":"[21]"}],"fun_headline_variants":["Embodied AI fails zero-shot: under 20% on 3K tasks","New benchmark exposes VLM limits in embodied tasks","3K embodied tasks: VLMs score below 20% zero-shot","Fine-tuning on EmbRACE-3K boosts embodied reasoning","Embodied reasoning benchmark: VLMs struggle, RL helps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that Gemini-generated task instructions and step-by-step rationales, after human filtering, are genuinely grounded in the environment rather than reflecting the language model's own priors about what should be seen and done.","fun_headline_variants_meta":{"raw":{"variants":["Embodied AI fails zero-shot: under 20% on 3K tasks","New benchmark exposes VLM limits in embodied tasks","3K embodied tasks: VLMs score below 20% zero-shot","Fine-tuning on EmbRACE-3K boosts embodied reasoning","Embodied reasoning benchmark: VLMs struggle, RL helps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1251,"prompt_tokens":1091,"completion_tokens":160,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":707,"completion_tokens_details":{"reasoning_tokens":72}},"tokens_in":707,"tokens_out":160,"duration_ms":2711,"temperature":1.0,"reasoning_tokens":72,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:27:40.102140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild EmbRACE-3K's supervision with human-written instructions and rationales on the same captured trajectories, then rerun the zero-shot evaluation and the SFT+RL training; if the below-20% zero-shot numbers and the training gains largely disappear or change direction, the benchmark's difficulty and its utility are artifacts of Gemini's annotations rather than of embodied interaction itself.","supporting_citations":[{"cited_title":"Alfred: A benchmark for interpreting grounded instructions for everyday tasks","cited_arxiv_id":null,"evidence_quote":"The ALFRED benchmark that EmbRACE-3K positions itself against on step-wise visual grounding and online closed-loop interaction."},{"cited_title":"R1-v: Reinforcing super generalization ability in vision-language models with less than $3","cited_arxiv_id":null,"evidence_quote":"The training framework on which the GRPO reinforcement-learning stage is implemented."},{"cited_title":"Llamafactory: Unified efficient fine-tuning of 100+ language models","cited_arxiv_id":null,"evidence_quote":"The training framework used to run the supervised fine-tuning stage."},{"cited_title":"Visual cot: Advancing multi-modal language models with a comprehensive dataset and benchmark for chain-of-thought reasoning","cited_arxiv_id":null,"evidence_quote":"Supplies the chain-of-thought reasoning formulation that EmbRACE-3K extends to step-level embodied annotations."}],"review_version":1}