{"id":"1c7a10fc-bb41-4301-a958-a40d307947e5","arxiv_id":"2607.15330","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Pre-training a VLA model on 100k hours of auto-labeled UMI trajectories, then post-training on robot data, yields SOTA simulated manipulation and data-efficient fine-tuning.","lead":"A large robotics team built a vision-language-action model trained on over 100k hours of handheld-gripper manipulation data, then aligned it to robot embodiments. The model reports state-of-the-art results on several manipulation benchmarks and shows scaling gains as data and model size grow.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Scaling claim is validated only on a 20k-hour subset; no evidence connects the 100k-hour headline to the reported SOTA or real-robot numbers.","rationale":"The reader's weakest assumption is caption quality. That is a legitimate concern (no validation of Qwen3.5-27B captions), but I judge the 100k/20k disconnect to be more load-bearing: it attacks the paper's headline empirical claim directly, and it is internally documented in §3.1–3.4. The scaling curves in Fig. 5 and the transfer curves in Fig. 8 are only defined on the 20k subset; the simulation SOTA tables do not identify the checkpoint. Thus the central statement 'consistently improves with increased data scales' is established only within a 20k-hour range, and the 100k-hour dataset is never shown to help. This does not make the in-range findings false; it makes the headline overreach. The fix is disclosure plus one comparative evaluation at 100k. Until then, CONDITIONAL (as the reader said) is the correct verdict; I would not move to REJECT because the 20k-scale evidence is internally consistent and no contradiction is demonstrated.","tokens_in":25391,"tokens_out":11073,"duration_ms":97900,"concrete_test":"Obtain (or retrain with identical recipe) two Xiaomi-Robotics-1-5B checkpoints: one pre-trained on the 20k-hour subset used in §3.1 and one on the full 100k-hour corpus. Post-train both identically and evaluate on the four OOTB tasks (§3.2) and on RoboCasa365 (§3.4, Table 3). If the 100k checkpoint does not improve over the 20k checkpoint on these evaluations, or if the reported SOTA numbers were actually produced by the 20k checkpoint, then the 'over 100K hours' scaling claim is unsupported. At minimum, the paper should state in the table captions which pre-training data scale and checkpoint produced each number.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1's data-scaling experiments pre-train on '12.5%, 25%, 50%, and 100% of about 20k hours of UMI data' — not the 100k hours in the title. Section 3.2 then post-trains checkpoints from those 20k-scale runs. Section 3.4 reports SOTA on RoboCasa, RoboCasa365, VLABench, and RoboDojo, but nowhere states which pre-training corpus or checkpoint produced these numbers. The abstract's claim that the model 'consistently improves with increased data scales' is therefore supported only up to 20k hours. If the benchmark/real-robot numbers come from a 100k-hours model, there is no apples-to-apples 20k-vs-100k comparison; if they come from the 20k model, the 'over 100K hours' contribution is not exercised. Either way, the central scaling-and-transfer claim is missing its headline data point. This is a missing-support gap, not an internal contradiction, but it is load-bearing: the paper's own stated contribution is scaling with over 100k hours.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Xiaomi-Robotics-1, a vision-language-action (VLA) foundation model trained in two stages: pre-training on 'over 100k hours' of real-world UMI gripper trajectories annotated by an automatic captioning pipeline that describes scene state transitions, then post-training on about 10k hours of cross-embodiment robot data with human-style imperative instructions. The authors report scaling experiments during pre-training (data and model size), transfer of these scaling gains to out-of-the-box post-training performance in unseen real environments, efficient fine-tuning on four novel tasks with under 10 hours of data per task, and state-of-the-art results on RoboCasa, RoboCasa365, VLABench, and RoboDojo. The central claimed contributions are the scaling law-like behavior of the pre-training recipe and the effectiveness of the two-stage alignment procedure.","tokens_in":25599,"tokens_out":4687,"duration_ms":42559,"significance":"If the results hold, this is one of the largest-scale demonstrations of VLA pre-training on real-world manipulation data, and the two-stage pre-training/post-training recipe with auto-labeled state-transition captions is a plausible path toward generalist robot policies. The paper provides a detailed and internally coherent architecture, explicit scaling experiments, and broad evaluation across four simulated benchmarks plus real-robot trials. The main strengths are the scale of the proposed data collection, the concrete description of the auto-labeling pipeline, and the direct evaluation of pre-training scale transfer to post-training and downstream fine-tuning. However, the headline 100k-hour claim is not exercised in the reported scaling experiments, the auto-labeling mechanism is unvalidated, and the real-robot evaluations use very small trial counts with no confidence intervals. These gaps currently prevent full confidence in the stated conclusions.","major_comments":[{"comment":"The scaling experiments are run on 'about 20k hours' of UMI data, not the 'over 100k hours' claimed in the abstract and §1. Specifically, §3.1 pre-trains on 12.5%, 25%, 50%, and 100% of about 20k hours, and §3.2 post-trains checkpoints from those runs. §3.4 (Tabs. 2–5) and §3.3 (Fig. 10) never state which pre-training corpus or checkpoint produced the reported numbers. If the benchmark/real-robot results come from a 100k-hours model, there is no apples-to-apples 20k-vs-100k comparison; if they come from a 20k model, the 'over 100K hours' contribution is not exercised. Please state the pre-training corpus and checkpoint for every downstream result, and add at least one data point at the full 100k-hour scale to support the paper's central scaling claim.","section":"§3.1, §3.2, §3.4"},{"comment":"The auto-labeling pipeline is load-bearing: the model learns to generate actions conditioned on language annotations of scene state transitions produced by Qwen3.5-27B. No quantitative evaluation of caption quality is reported — no accuracy against human labels, no human agreement, no sample size, and no ablation comparing auto-labels with human labels or with no language conditioning. If the captions are noisy or misaligned with the actual motion, the pre-training signal is corrupted and the stated mechanism does not explain the results. Please add a caption-quality evaluation and a small-scale human-label ablation.","section":"§2.2.1"},{"comment":"Real-robot evaluations use very small trial counts. §3.3 states 10 trials per task for downstream fine-tuning; Fig. 8 does not state the number of trials or environments for the out-of-the-box post-training evaluation. With 10 trials, a reported 75% success rate has an approximate standard error of 13.7 percentage points, making comparisons such as 75% vs. 40% far less conclusive than implied. Please report trial counts, confidence intervals, and the number of environments/object instances for all real-robot evaluations, and increase trial counts where feasible.","section":"§3.3, Fig. 8"},{"comment":"The 12.5% and 25% data-scaling runs were terminated early because of overfitting, yet the resulting checkpoints are used in §3.2 for the post-training transfer study. Early stopping is a reasonable practice, but the choice of stopping time affects the measured validation error and the subsequent transfer results. Please clarify how the early-stopped checkpoints were selected, whether the same criterion was applied across all runs, and whether the qualitative conclusions in §3.2 are robust to this selection.","section":"§3.1, Fig. 5"}],"minor_comments":[{"comment":"The RoboCasa365 state-of-the-art number is inconsistent: the abstract and Table 3 report 57.4%, while §1 reports 57.6%. Please correct.","section":"Abstract vs. §1"},{"comment":"The claim that performance 'shows no sign of saturation' is based on only four data points, two of which were early-stopped; the 50%→100% gain is 6 percentage points. This is a reasonable observation but should be phrased more cautiously.","section":"§3.2.1"},{"comment":"The decision to exclude action-related tokens from the DiT's attention is justified by a shortcut hypothesis, but no ablation is shown. Adding an ablation would strengthen the architectural contribution.","section":"§2.1"},{"comment":"The CoT labeling used for VLABench is adopted from ERVLA [61] with a 50% NTP loss probability, but no comparison to training without CoT is provided in this paper. A small ablation would clarify the contribution of this component.","section":"§3.4, VLABench"},{"comment":"In the provided version, the figures contain garbled character rendering (e.g., Fig. 8 axis labels). Please ensure all figures are legible and that numeric values in the figures match the text.","section":"Figures 5 and 8"}],"recommendation":"major_revision","confidential_remarks":"The central concern is the disconnect between the 'over 100K hours' headline and the actual scaling experiments, which top out at 20k hours. This is fixable in principle by clearly labeling which checkpoint produced each result and adding at least one full-scale data point. Also note that the paper relies heavily on self-cited, not-yet-published work (Xiaomi-Robotics-0 [8], ERVLA [61]) for core components of the recipe; given that code and checkpoints are only promised, the reproducibility of the benchmark claims cannot currently be assessed. If the authors can close the 100k-hour gap and add caption-quality validation, I would view the manuscript favorably."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read this paper with the stress-test note in hand, and the note lands. The central problem is real: every scaling experiment, both data and model, runs on about 20k hours of UMI data, not the 100k hours in the title. Section 3.4 reports SOTA on four benchmarks but never says which pre-training corpus or checkpoint produced those numbers. If they come from a 100k-hour model, there is no apples-to-apples comparison; if they come from the 20k model, the headline contribution is not exercised. This is a missing-support gap, not a contradiction, but it is load-bearing for the paper's stated contribution.\n\nWhat is genuinely new here is the scale itself: a 100k-hour UMI corpus, a producer-consumer auto-labeling pipeline for state-transition captions, and the scaling curves up to 20k hours. The data scaling results are plausible and internally consistent: validation action error decreases monotonically with data, and post-training success rates transfer the trend. The architecture is assembled from existing pieces (MoT, flow matching, Choice Policies), but the paper is honest about that; the contribution is empirical, not architectural. The benchmark results, if reproducible, are strong, particularly the RoboCasa365 composite-unseen jump from 7.9 to 32.1.\n\nThe soft spots are proportionate to how much they matter. First, as noted, the 100k/20k disconnect is the biggest issue. Second, the auto-labeling quality is never validated: no human agreement, no caption accuracy, no ablation replacing auto-labels with human labels. The mechanism explicitly depends on captions being accurate enough to supervise action learning, so this is not a minor omission. Third, the real-robot fine-tuning evaluation uses 10 trials per task with no confidence intervals; for a 75% vs 40% comparison that is acceptable, but not definitive. Fourth, code, checkpoints, and the dataset itself are promised but not yet released, which makes the SOTA claims impossible to verify.\n\nThat said, the paper does not oversell its internal logic. It terminates early runs when they overfit, it masks action dimensions appropriately, and it admits compute constraints. The writing is clear and the claims are stated precisely enough that the gaps are identifiable, which is a sign of careful thinking.\n\nWho is this for? Anyone working on VLA scaling, UMI data collection, or cross-embodiment pretraining. It deserves a serious referee, but the review should require the authors to connect the reported SOTA numbers to a specific pre-training scale, provide a caption-quality audit, and commit to artifact release before acceptance. I would not cite it yet, but I would keep an eye on the project page.","headline":"A serious scaling study for robot foundation models, but the 100k-hour headline claim is only validated on a 20k-hour subset and the caption-quality premise is unexamined.","tokens_in":26269,"tokens_out":1634,"would_cite":false,"duration_ms":16409,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pre-training a robot policy on over 100,000 hours of auto-captioned real-world trajectories yields steady gains from data and model scale, and the gains survive into unseen real-robot environments.","keywords":["vision-language-action model","robot foundation policy","scaling behavior","auto-labeling","UMI trajectories","flow matching","cross-embodiment transfer","manipulation"],"falsifier":"Re-annotate a random sample of the pre-training corpus by hand and compare: (a) measure human agreement with the auto-captions; (b) pre-train identical models on the human-labeled subset and the auto-labeled subset and compare held-out action error and post-training success. If the auto-labeled model does not at least match the human-labeled one, or if caption agreement is low, the stated caption-conditioning mechanism is not the source of the reported gains.","tokens_in":25219,"feed_emoji":"🤖","tokens_out":9015,"duration_ms":79474,"temperature":0.7,"pith_summary":"The paper's thesis is that robot manipulation can be scaled like a large language model if the data bottleneck is removed. It removes that bottleneck in two moves: collect trajectories with handheld UMI grippers instead of expensive teleoperated robots, and annotate them automatically with a vision-language model that writes captions describing how the scene's objects and gripper change within each clip. Trained on this corpus, the model learns to produce actions that move the scene from its current state to the language-described target state; a second, smaller stage of cross-embodiment data aligns it to real robots and to the imperative instructions humans actually use. The paper reports monotonic improvements with both data scale and model size, transfer of those gains to out-of-the-box unseen-environment success, state-of-the-art results on four simulation benchmarks, and 75% success on four new tasks fine-tuned with under 10 hours of data each. A sympathetic reader would care because this is concrete evidence that cheap, auto-labeled data, not just curated teleoperation, can drive a generalist robot policy.","feed_headline":"Auto-labeling turns 100K hours of robot data into a scaling policy","feed_subtitle":"Pre-trained on auto-captioned real-world trajectories, it tops four benchmarks and learns new tasks in under 10 hours.","key_machinery":"The load-bearing mechanism is the auto-labeling pipeline: a large pre-trained vision-language model (Qwen3.5-27B) captions fixed-length trajectory clips by describing how grippers and interacting objects change state, so each training example becomes 'generate actions that carry the scene from the observed state to the language-described target state.' That conversion makes 100,000 hours labelable in about two weeks and is the paper's answer to the robotics data bottleneck. The architecture is a Mixture-of-Transformers coupling a pre-trained VLM with a diffusion transformer (DiT) that emits action chunks via flow matching; an auxiliary Choice-Policy head on the VLM proposes action candidates","core_discovery":"The paper's central claim is that a vision-language-action model pre-trained on over 100,000 hours of UMI trajectories—annotated automatically with scene state-transition captions rather than by hand—scales cleanly in both data and model size, and that this scaling carries over to post-trained real-robot performance in unseen environments. Aligned to robot embodiments with about 10,000 hours of cross-embodiment data, the model reports new state-of-the-art results on four simulation benchmarks; fine-tuned on fewer than 10 hours per task, it reaches 75% average success across four new dexterous tasks against 40% for the leading open comparison policy. The authors take this as evidence that sca","pith_inferences":["[Editorial inference] If caption accuracy is the true driver, the natural next experiment is caption-quality control: measuring human agreement with the auto-captions or ablating against a human-labeled subset would tell whether the scaling curves come from the stated supervision mechanism.","[Editorial inference] The scaling curves are measured as action-prediction error on a 20k-hour subset at four data fractions; whether the trend extends to the full 100k+ hours and to success-rate metrics is open.","[Editorial inference] A testable consequence: if the recipe works as claimed, datasets gathered more cheaply (e.g., crowdsourced handheld-gripper recordings) should improve such policies roughly in proportion to hours added, making captioning cost the main scaling lever.","[Editorial inference] The attention-exclusion trick (keeping the DiT from copying VLM action tokens) is supported by a hypothesis—shortcut copying—not an ablation; an ablation at scale would clarify whether it is load-bearing."],"forward_implications":["Data volume, not model capacity, is the current bottleneck: at 2B–10B parameters the gains from scaling model size were smaller than from scaling data, so collecting more UMI hours should keep paying off.","Out-of-the-box behavior in unseen environments improves monotonically with both pre-training data and model size (26% success with no action pre-training, rising to 75–79% after full pre-training).","A pre-trained foundation policy can adapt to dexterous, long-horizon, and deformable-object tasks with under 10 hours of data per task, reaching 75% average success.","The composite-unseen split of RoboCasa365 (32.1% vs. 7.9% for the best prior method) suggests the model can recombine learned atomic skills for novel task compositions.","The two-stage recipe—state-transition pre-training on robot-free data, then imperative-instruction alignment on cross-embodiment data—transfers across mobile manipulators, dual-arm robots, and static arms."],"fun_headline_variants":["100K hours of robot data yield a scaling VLA model","Auto-captioned trajectories power Xiaomi's robot brain","Scaling VLA with 100K hours: SOTA on four benchmarks","Fewer than 10 hours of fine-tuning beats open baselines","From 100K hours to 75% success: data-efficient robot learning"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the automatically generated captions describing scene state transitions are accurate enough to supervise action learning; the paper reports no caption-accuracy numbers, no human-agreement check, and no ablation with human-labeled captions, so if those captions are noisy or misaligned with the recorded gripper motions, the scaling and transfer results are not explained by the stated mechanism.","fun_headline_variants_meta":{"raw":{"variants":["100K hours of robot data yield a scaling VLA model","Auto-captioned trajectories power Xiaomi's robot brain","Scaling VLA with 100K hours: SOTA on four benchmarks","Fewer than 10 hours of fine-tuning beats open baselines","From 100K hours to 75% success: data-efficient robot learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1324,"prompt_tokens":905,"completion_tokens":419,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":649,"completion_tokens_details":{"reasoning_tokens":327}},"tokens_in":649,"tokens_out":419,"duration_ms":4890,"temperature":1.0,"reasoning_tokens":327,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:57:28.916709+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random sample of the pre-training corpus by hand and compare: (a) measure human agreement with the auto-captions; (b) pre-train identical models on the human-labeled subset and the auto-labeled subset and compare held-out action error and post-training success. If the auto-labeled model does not at least match the human-labeled one, or if caption agreement is low, the stated caption-conditioning mechanism is not the source of the reported gains.","supporting_citations":[],"review_version":1}