{"id":"4a84df3c-68d5-46f5-a444-69b6c6947c2c","arxiv_id":"2512.24125","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A 6K-question embodied-reasoning benchmark plus a flow-matching action tokenizer let one 3B vision-language model reason and manipulate better than continuous- or discrete-action VLA baselines.","lead":"This paper adds two pieces to robot foundation models: a benchmark (ERIQ) that measures a robot's embodied reasoning with multiple-choice questions, and a tokenizer (FACT) that turns continuous arm motions into discrete codes a language model can generate while preserving control precision. The authors claim reasoning ability and real-world task success are tightly linked, and their model beats both continuous- and discrete-action baselines.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ERIQ's contamination risk is the load-bearing weak point: no train/test split is shown between ERIQ and the AgiBot World-derived embodied VQA training set, and no correlation statistic is reported.","rationale":"The reader's weakest assumption—that ERIQ is a valid, uncontaminated measure—is exactly the most load-bearing vulnerability in the paper. If ERIQ overlaps with the training distribution, the headline ERIQ improvement and the correlation claim lose their evidential value. My independent reading of the full text confirms that no train/test separation is stated: ERIQ is described as derived from real-world robotic trials (Section III-A), while the embodied VQA training data is described as curated on AgiBot World (Section V-A) with content that mirrors ERIQ's reasoning dimensions. There is also no quantitative correlation analysis anywhere in Section V; the paper merely asserts the relationship. These are evaluation-validity gaps, not evidence of fabrication, and they can be addressed by releasing a clean split and reporting a correlation coefficient. Because my concern aligns with the reader's and does not change the appropriate final verdict, I recommend no adjustment: the paper should remain CONDITIONAL pending the authors' response.","tokens_in":20240,"tokens_out":5984,"duration_ms":60829,"concrete_test":"Release ERIQ and the training-data manifest with a documented split by task instance/scene (or at minimum by episode). Then recompute Ours-3B's ERIQ accuracy separately on (a) items whose underlying trials/episodes are present in the training distribution and (b) held-out trials/scenes not seen during training. If accuracy on held-out items is substantially below the reported 82.72 average—or if the gain over Qwen2.5-VL-3B collapses on held-out items—the benchmark is contaminated and the correlation claim must be re-examined. Additionally, compute a correlation statistic (e.g., Spearman's rho with confidence interval) between ERIQ scores and end-to-end success across at least 5–10 VLA models trained on the same embodiment; without this, 'strong positive correlation' remains unsupported by any reported quantity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that ERIQ 'reveals a strong positive correlation between embodied reasoning capability and end-to-end VLA generalization' (Section V-B)—requires ERIQ to be a clean, independent measure of embodied reasoning. Section III-A states that all 6,052 ERIQ QA pairs are 'derived entirely from authentic, real-world robotic trials'; Section V-A states that the embodied VQA training data is 'curated on AgiBot World' and includes 2D trajectory data, grounding annotations, sub-task planning, and scene understanding—precisely the content dimensions ERIQ is designed to test. The manuscript never specifies a task/scene/instance split between ERIQ and the training data. If the model's embodied-VQA pre-training includes episodes, scenes, or question templates that overlap with ERIQ, the reported 41% improvement (58.64 to 82.72) and the claimed correlation are confounded by distributional overlap rather than demonstrating reasoning capability. This is independent of FACT: even a perfect action tokenizer would not rescue the correlation claim if the measuring instrument is contaminated. The real-world superiority claim is also weakened by missing error bars and no stated adaptation of baselines to the AgiBot G1 embodiment, but the ERIQ validity issue is the more foundational because it underwrites the paper's proposed diagnostic contribution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two coupled contributions for generalist robot manipulation. First, it introduces ERIQ, a 6,052-question multiple-choice benchmark for embodied reasoning, covering Spatial Perception & Grounding, Planning & Monitoring, Error Detection & Recovery, and Human Intent Understanding. The authors use ERIQ to claim a strong positive correlation between embodied reasoning ability and end-to-end VLA generalization. Second, it proposes FACT, a flow-matching action tokenizer that maps continuous action trajectories to discrete sign-quantized tokens and reconstructs them with a rectified-flow ODE decoder. This tokenizer is embedded in GenieReasoner, a Qwen2.5-VL-based VLA that is jointly pre-trained on general VQA, embodied VQA, and tokenized action data, then post-trained with a mixed VQA+action objective. Experiments report large ERIQ gains over the base VLM (58.64 to 82.72), lower reconstruction MSE than FAST+ at matched code lengths, ablations in the GenieSim simulator, and real-world comparisons against GR00T, π0, π0.5, and π0-FAST.","tokens_in":20493,"tokens_out":3640,"duration_ms":38173,"significance":"If both central claims are valid, the paper would give the community a cheap, decoupled diagnostic for embodied reasoning and an action tokenizer that avoids the precision-reasoning trade-off in discrete VLAs. The paper has genuine strengths: the task-level 40:1 split for FACT evaluation is appropriate; the ablation in Table III cleanly separates the contributions of embodied VQA and action alignment; and the representation-level MSE comparison with FAST+ is informative. However, the headline correlation claim is currently not established: the ERIQ benchmark and the embodied VQA training set may share the same AgiBot World source with no reported train/test separation, and Table II mixes model sizes without reporting a correlation statistic. The real-world superiority claim likewise lacks essential experimental detail. These issues are fixable but are load-bearing for the paper's main contributions.","major_comments":[{"comment":"The central correlation claim depends on ERIQ being a clean, independent measure of embodied reasoning. ERIQ is 'derived entirely from authentic, real-world robotic trials' (Section III-A), while the embodied VQA training data is 'curated based on AgiBot World' and includes 2D trajectory data, grounding annotations, sub-task planning, and scene understanding (Section V-A) — precisely the content dimensions ERIQ tests. The manuscript never states a task/scene/instance split between ERIQ and the training data. If ERIQ items or their underlying episodes overlap with the embodied VQA pre-training data, the reported improvement from 58.64 to 82.72 and the claimed reasoning-to-generalization correlation are confounded by distributional leakage. Please specify the exact split, report whether any ERIQ items or scenes appear in training, and provide an analysis (e.g., per-dimension performance on","section":"Section III-A, V-A, V-B"},{"comment":"The assertion of a 'strong positive correlation between embodied reasoning capability and end-to-end VLA generalization' is not supported by any correlation coefficient, confidence interval, or significance test. Table II mixes models of different sizes (3B, 7B, 8B) and different training corpora, so the apparent ordering could reflect scale or data rather than embodied reasoning per se. Please report a quantitative correlation (e.g., Pearson/Spearman) between ERIQ-Avg and the downstream success/language-following metrics, ideally with a partial correlation controlling for parameter count or within a single model family, and state the number of models and p-value. The current text overstates what the data show.","section":"Section V-B, Table II"},{"comment":"The real-world superiority claim over π0.5, GR00T, π0, and π0-FAST is not fully assessable from the manuscript. Figures 8–10 present aggregate scores without error bars, trial counts, or standard deviations. It is also unclear whether the baselines were adapted to the AgiBot G1 embodiment (fine-tuning data, number of steps, action horizon, inference settings) or were used with their default configurations. The 'Total Score' is defined as a fixed weighted average, Score=(1.0×Success+0.5×Following)/1.5, with no sensitivity analysis. Please report per-trial results, the number of rollouts per setting, baseline adaptation details, and the raw success/following scores behind the aggregate.","section":"Section V-E, Figures 8-10"},{"comment":"The FACT tokenizer description is mostly clear, but the relation between the sign quantizer and the reported vocabulary size needs clarification. With c=sign(e), the effective codebook size is 2^D, where D is the latent dimension; the text later mentions a '212-entry vocabulary' (Section V-C), which appears to be a typo for 2^12 or another power of two. Please state D explicitly and correct the notation. This is not a fundamental flaw, but it is necessary for reproducing the tokenizer and interpreting Figure 7.","section":"Section IV-C, Eq. (3)-(8), Section V-C"}],"minor_comments":[{"comment":"'41% Accuracy Improvement' is ambiguous: the ERIQ score increases by 24.08 percentage points (58.64 to 82.72), which is a 41.1% relative improvement. Please label which convention is used.","section":"Figure 1"},{"comment":"The caption refers to 'ER-6K Benchmark'; this should likely be 'ERIQ Benchmark'.","section":"Table II caption"},{"comment":"The sentence 'Dualview Matching (an absolute increase of 31.01% over the base model and Relative Position Grounding (+24.9%)' is missing a closing parenthesis and a conjunction; please rephrase.","section":"Section V-B"},{"comment":"The task-level 40:1 split is good, but please also report the number of tasks/episodes in the split and whether the evaluation tasks are disjoint from those used for the VLM's embodied VQA training. This matters for the contamination concern raised above.","section":"Section V-C"},{"comment":"No data or code release is mentioned. For ERIQ to serve as a community benchmark and for FACT to be compared, the benchmark, tokenizer code, and training/inference recipes should be made available.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The contamination risk is substantial and, if real, undermines the paper's diagnostic contribution. However, the concern is addressable: the authors can provide a clean evaluation split, demonstrate non-overlap, and supply proper correlation statistics. If they cannot, the correlation claim should be removed or substantially weakened. The FACT representation-level results appear solid and could stand as a separate contribution, but the current manuscript's headline depends on the ERIQ claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things worth knowing before you read it. The action tokenizer, FACT, is a genuinely new combination — sign-quantized VQ codes paired with a rectified-flow decoder — and the reconstruction comparison against FAST+ is carried out on a task-level 40:1 split, which is more careful than much of this literature. That part of the paper is a real contribution. The other headline, that ERIQ 'reveals a strong positive correlation' between embodied reasoning and end-to-end VLA generalization, is not backed by any statistic. For that claim the paper offers one model's pre/post ERIQ score (58.64 to 82.72) and two training variants with identical ERIQ (80.39) but different success rates. That is evidence of something, but it is not a correlation.\n\nThe benchmark itself is also worth taking seriously. Six thousand MC QA pairs across fifteen sub-tasks, including error recovery and human intent — dimensions most embodied benchmarks skip. If released cleanly, it would be useful to the community. But there is a real contamination risk. The embodied VQA training data is curated on AgiBot World, and ERIQ is derived from the same platform. The paper never states a train/test split between the two. The 41% accuracy gain may be partly distributional overlap rather than reasoning ability. That needs to be pinned down before the ERIQ numbers can be interpreted.\n\nThe real-robot section is weaker. No error bars. Baselines like pi0.5 and GR00T are not adapted to the AgiBot G1 embodiment, so the 'outperforms' claims are uneven. The total score formula (1.0*success + 0.5*following)/1.5 is ad hoc. And no code, data, weights, or benchmark are released. For a paper whose main diagnostic contribution is a benchmark, that is a hard limitation for verification.\n\nNone of this suggests fabrication. The losses are standard, the FACT design is clean, and the reconstruction gap is plausible. But the central claim about ERIQ predicting generalization needs a proper correlation analysis on a held-out split, with the benchmark and training data clearly separated. This is a solid systems draft, not yet a finished scientific story. Send it to peer review, but expect the authors to release the benchmark and code and to fix the correlation claim before it can be trusted.","headline":"FACT is a real, checkable contribution; the ERIQ-to-generalization correlation is asserted, not shown, and the benchmark's train/test separation is the first thing to ask the authors for.","tokens_in":21128,"tokens_out":3779,"would_cite":false,"duration_ms":35993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two coupled claims carry this paper: embodied reasoning can be measured independently of execution, and a flow-matching action tokenizer lets one model reason and act precisely in a unified autoregressive space.","keywords":["embodied reasoning","vision-language-action model","action tokenization","flow matching","robotic manipulation benchmark","discrete action tokens","autoregressive policy","generalization"],"falsifier":"Build an ERIQ-style benchmark whose questions come from a completely different robot dataset or from held-out trials never seen in any training mixture, retrain the same model with the same recipe, and check two things: does the ERIQ score stay near 82.72 instead of dropping toward the base 58.64, and does the rank order of multiple VLMs on this new benchmark still match their rank order on real-robot success? A large drop or a broken ranking would refute the paper's claim that ERIQ measures a generalizable reasoning capability that drives generalization.","tokens_in":20012,"feed_emoji":"🤖","tokens_out":6698,"duration_ms":60385,"temperature":0.7,"pith_summary":"ERIQ, a new 6,052-question benchmark built from real robot trials, is offered as a way to measure a robot model's embodied reasoning — spatial perception, planning, error recovery, and human intent — without running any physical policy. The paper's central evidence is a strong positive correlation between scores on ERIQ and end-to-end task success, which it presents as proof that reasoning capability drives generalization in vision-language-action models. To close the gap from reasoning to action, it introduces FACT, a tokenizer that compresses continuous robot trajectories into short discrete codes using a sign-quantization encoder and a flow-matching decoder that reconstructs high-fidelity motion. The resulting model, GenieReasoner, jointly trains on general question-answering, embodied question-answering, and tokenized actions, and is claimed to beat both continuous-action and discrete-action baselines on real-world manipulation. If correct, this gives the field a cheap, decoupled diagnostic for reasoning and a way to remove the precision-versus-reasoning trade-off.","feed_headline":"One flow tokenizer solves the robot reasoning-precision tradeoff","feed_subtitle":"New benchmark splits thinking from moving, showing reasoning predicts real-world robot success.","key_machinery":"Two instruments carry the argument. ERIQ is the measuring stick: 6,052 multiple-choice questions across four reasoning dimensions, with deterministic scoring, meant to isolate cognition from control. FACT is the bridge: a vector-quantized encoder (a lookup-free sign quantizer that turns continuous action chunks into compact bit-like codes) paired with a flow-matching decoder that learns a velocity field and integrates an ordinary differential equation to reconstruct smooth trajectories from those codes. The mechanism that makes the system work is the alignment of the action space with the VLM's discrete token space: action prediction becomes next-token prediction, so cross-entropy reasoning","core_discovery":"The paper's central claim is double. First, ERIQ can quantify embodied reasoning separately from motor control: it is a 6,052-item multiple-choice question set spanning spatial perception and grounding, planning and monitoring, error detection and recovery, and human intent understanding, drawn entirely from real, first-person robotic trials. On this benchmark the base vision-language model scores 58.64%; after co-training on general and embodied question-answering data plus tokenized actions, the same backbone reaches 82.72%, a 41% relative improvement. Second, the paper claims this reasoning gain is not cosmetic: ERIQ scores correlate strongly with end-to-end manipulation success, and the","pith_inferences":["If ERIQ is a valid decoupled diagnostic, then the standard development loop could invert: iterate on the VLM's reasoning score first, and reserve real-robot rollouts for final verification—a workflow the paper describes only implicitly.","An independent replication could take a fixed policy-training pipeline and swap only the pretrained VLM backbone; if the real-world success ranking follows the ERIQ ranking, the correlation claim would be robust; the paper's own evidence comes from a single pipeline.","A natural stress test is to measure FACT's reconstruction fidelity on sub-millimeter precision tasks or high-frequency dynamic motions (fast reaching, cloth manipulation); the benchmark tasks here are mostly pick-and-place style, and the paper does not report how the tokenizer behaves at the edge of the action distribution.","Because the reasoning training data and the ERIQ benchmark share the same real-robot trials, an external, independently-sourced version of ERIQ would be needed to confirm that the measured reasoning gain reflects general embodied intelligence rather than memorized scene statistics."],"forward_implications":["ERIQ gives developers a way to measure embodied reasoning in minutes without expensive physical rollouts, enabling early screening of VLM backbones before committing to action training.","FACT's compact discrete codes let a single autoregressive model carry both reasoning and control, removing the need for a separate continuous action head and the gradient conflicts it introduces.","Because the flow-matching decoder preserves trajectory fidelity, discrete-action policies built this way should avoid the precision failures that plagued earlier tokenizers like uniform binning or variable-length encoding.","The claimed correlation implies that improving a model's ERIQ score—via co-training on embodied reasoning data—should directly improve end-to-end manipulation success, not just instruction following.","The joint post-training recipe (keeping embodied VQA in the final stage) is presented as necessary to preserve reasoning while aligning actions, so the paper's success hinges on the optimizer not un-learning reasoning when action tokens are added."],"fun_headline_variants":["Robot IQ test: reasoning scores predict manipulation success","Flow tokenizer synchronizes robot thinking and doing","ERIQ: new metric links robot reasoning to real-world skill","Benchmark proves robot reasoning boosts precision control"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that ERIQ truly measures embodied reasoning without contamination: the benchmark questions and the model's embodied question-answering training data come from the same real-world robotic trials, and the paper never states that ERIQ items were excluded from training, so if the model has memorized those scenes the reported 41% reasoning gain and its correlation with task success are partly artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Robot IQ test: reasoning scores predict manipulation success","Flow tokenizer synchronizes robot thinking and doing","ERIQ: new metric links robot reasoning to real-world skill","Benchmark proves robot reasoning boosts precision control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1543,"prompt_tokens":767,"completion_tokens":776,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":511,"completion_tokens_details":{"reasoning_tokens":715}},"tokens_in":511,"tokens_out":776,"duration_ms":7928,"temperature":1.0,"reasoning_tokens":715,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T13:25:31.809770+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build an ERIQ-style benchmark whose questions come from a completely different robot dataset or from held-out trials never seen in any training mixture, retrain the same model with the same recipe, and check two things: does the ERIQ score stay near 82.72 instead of dropping toward the base 58.64, and does the rank order of multiple VLMs on this new benchmark still match their rank order on real-robot success? A large drop or a broken ranking would refute the paper's claim that ERIQ measures a generalizable reasoning capability that drives generalization.","supporting_citations":[],"review_version":1}