{"id":"6f7e4953-8986-4f49-8d9e-47ae666143d1","arxiv_id":"2505.21906","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"ChatVLA-2 uses dynamic mixture-of-experts and a two-stage training recipe to let a vision-language-action model retain pretrained reasoning while following robot instructions.","lead":"The authors built a robot model that keeps its pretrained vision-language abilities alive while learning to control a robot, then tested it on two tabletop games: solving written math problems and placing toys by spatial instructions. The system beat several existing robot models on unseen equations and unseen objects.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim is internally contradicted: §3.3 trains on 600 math-matching and 300 spatial-placement trajectories, so OCR/math/spatial abilities are explicitly trained, not merely preserved from the VLM; the OOD split is unauditable.","rationale":"The reader's weakest_assumption is that the OOD test items may not be disjoint from training data. That is a necessary condition, but the deeper issue is that the described training data makes OCR/math/spatial reasoning explicitly trained inside the VLA, so even a clean disjoint test set would not establish the abstract's 'not explicitly trained' claim. The two concerns are related: without published registries and without a no-math-robot-data ablation, Tables 1-2 cannot falsify the memorization/training hypothesis. I retain the CONDITIONAL verdict because the architecture and two-stage recipe are plausible and a retraining audit could resolve the ambiguity; however, the paper must correct or heavily qualify the headline claim and release the relevant splits and/or the ablation.","tokens_in":15151,"tokens_out":6587,"duration_ms":71268,"concrete_test":"Retrain ChatVLA-2 with the same two-stage recipe and image-text data, but delete all 600 math-matching robot trajectories from Stage-1 robot data (replacing them with an equal number of non-math manipulation trajectories if total-step balance matters), then evaluate the open-world math/OCR split of Table 1. If OCR, math reasoning, and success stay near 3.58/4, 1.73/2, and 43/52, the pretrained-preservation explanation has support; if they drop toward the dense/no-Stage-1 baselines in Tables 3-4 (~0 OCR, ~0 math), the headline capability was trained in via the 600 robot demonstrations rather than preserved from the VLM.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline result is that ChatVLA-2 exhibits OCR and mathematical reasoning 'despite these abilities not being explicitly trained within the VLA.' That condition is contradicted by the training protocol in §3.3 and §B.2. The robot-data mixture includes 600 trajectories from the math-matching game and 300 from the toy-placement experiment; these trajectories contain whiteboard equations, handwritten digits, card values and positions, answer-selection actions, and spatial-relation instructions, all annotated with reasoning phrases and augmented by GPT-4o. The image-text mixture also includes TextVQA, GQA, RoboPoint, and 5k robotics VQA samples. This is supervised training for OCR, arithmetic, and spatial reasoning inside the VLA itself. The sentence 'We deliberately avoid selecting training data to bias the VLA toward specific skills such as OCR, mathematical reasoning, or spatial reasoning' (§3.3) is therefore not consistent with the described data. The load-bearing consequence is that Tables 1-2 cannot separate (a) retention of pretrained Qwen2-VL knowledge from (b) learning the task from 600 math and 300 placement demonstrations. Because the paper does not list the training/test equations, objects, card layouts, or direction-object pairs, the open-world claim cannot currently be audited.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents ChatVLA-2, a mixture-of-experts vision-language-action model built on Qwen2-VL and a ScaleDP action expert, with a two-stage training pipeline intended to preserve pretrained VLM knowledge while enabling robot control. The core claim is that the model exhibits open-world OCR, mathematical reasoning, and spatial reasoning in real-robot math-matching and toy-placement tasks, even though these abilities are said not to be explicitly trained. The authors report strong open-world results (e.g., 82.7% manipulation success on the math game, 81.4% on toy placement) and support the claims with ablations on the MoE design, training stages, expert count, and reasoning-injection layer.","tokens_in":15417,"tokens_out":3444,"duration_ms":34637,"significance":"If the central claim were established, the paper would make a valuable contribution: a practical recipe for preventing catastrophic forgetting of VLM capabilities during VLA fine-tuning, with real-robot evidence on two tasks. The dynamic-MoE architecture, two-stage co-training, and reasoning-following module are concrete, reproducible design choices, and the authors include multiple ablations (Tables 3-6) and comparisons against several strong baselines. However, the headline interpretation that the demonstrated reasoning abilities were 'not explicitly trained' is contradicted by the paper's own training data, and the open-world evaluation is not fully auditable. The engineering contribution is significant, but the scientific claim about retention of pretrained knowledge needs substantial reframing and additional evidence.","major_comments":[{"comment":"The abstract and Section 3.3 claim that mathematical reasoning and OCR capabilities are exhibited 'despite these abilities not being explicitly trained within the VLA.' This is contradicted by the training protocol in §3.3 and §B.2: the robot data consists of 600 math-matching trajectories and 300 toy-placement trajectories, which include whiteboard equations, handwritten digits, card values and positions, answer-selection actions, and spatial-relation instructions. These trajectories are used to train the VLA in Stage 1, so OCR, arithmetic, and spatial reasoning are explicitly trained. The sentence 'We deliberately avoid selecting training data to bias the VLA toward specific skills such as OCR, mathematical reasoning, or spatial reasoning' (§3.3) is therefore misleading. As a result, Tables 1 and 2 cannot separate the contribution of pretrained knowledge from direct supervised learning of the same task families.","section":"Abstract and §3.3"},{"comment":"The open-world evaluation is a held-out split of the same two task families used for training, but the paper does not list the training and test equation sets, object sets, card arrangements, or direction-object pairs. The statement that 'the mathematical equations tested are entirely absent from the training data' (§4.1) and the analogous claim for toy placement (§4.2) cannot be audited. Because the reasoning annotations are augmented with GPT-4o (§B.2), there is also a risk that test-like phrases or content are injected into training examples. Without a full disclosure of the train/test splits, the out-of-distribution premise underlying the open-world claim is unsupported.","section":"§4.1 and §4.2"},{"comment":"The comparisons with OpenVLA, DexVLA, π0, and other baselines do not state whether these baselines were fine-tuned on the same 600 math-matching and 300 toy-placement trajectories used for ChatVLA-2. If the baselines are evaluated with their publicly released checkpoints, the comparison conflates architecture differences with differences in training data exposure. A fair comparison would require training each baseline on the same robot data mixture; otherwise, the reported superiority in open-world settings may reflect the presence of task-specific training data rather than the proposed MoE and reasoning-following mechanisms.","section":"§4.1, Table 1 and §4.2, Table 2"},{"comment":"The ablation studies that select the number of experts and top-k (Table 5) and the reasoning-injection layer (Table 6) are evaluated on the same math-matching benchmark used for the main claims, with no separate validation set. This creates a risk that these hyperparameters are chosen to optimize the reported open-world scores, which would inflate the main results. The authors should either use a separate validation split or report the selection procedure explicitly.","section":"§4.3, Tables 3-6"}],"minor_comments":[{"comment":"The phrase 'despite these abilities not being explicitly trained within the VLA' should be revised or precisely qualified, since the robot training data in §3.3 directly supervises the same abilities.","section":"Abstract and Introduction"},{"comment":"The OCR scoring rubric assigns 1 point for hand-written numbers, 1 point for card values and positions, and 2 points for the sign, but the text does not explain how partial credit is handled when the model outputs a correct value but wrong position, or when only one of two digits is correct. A short clarification would improve reproducibility.","section":"§4.1, Evaluation metrics"},{"comment":"The 'open-world object recognition' metric refers to object recognition in the reasoning output, but the paper does not specify how the recognition score is computed from the output reasoning, e.g., whether it is based on the predicted bounding boxes or their labels. Please add a precise definition.","section":"§4.2, Evaluation metrics"},{"comment":"The paragraph 'Why static/shared experts are not used?' is more of a design discussion than an empirical result; consider moving it to the appendix or compressing it to keep the main text focused.","section":"§3.2"},{"comment":"Several references are cited as arXiv preprints without version numbers; while acceptable, the authors should update to peer-reviewed versions where available, particularly for OpenVLA and π0.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central contradiction between the 'not explicitly trained' claim and the training data in §3.3 is likely to be raised by any careful reader and needs to be addressed head-on. The paper's value may lie in the architecture and training recipe rather than in the overstated claim of zero-shot retention of pretrained knowledge; the authors should reframe the contribution and provide the missing data split details."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: ChatVLA-2 is a reasonable engineering contribution to VLA training, but the headline claim—that OCR and math emerge without explicit training—is contradicted by the paper's own training data. The method is worth a look; the claim as written is not.\n\nWhat's new: dynamic MoE inside a Qwen2-VL backbone, with 8 experts and top-2 routing, plus a reasoning-following module that injects reasoning tokens into the latter half of the action expert, and a two-stage recipe (co-train on image-text + robot data, then freeze VLM and train only the action expert). That combination is not in ChatVLA or DexVLA. The ablations support the design choices: dense models collapse, static/shared MoE variants degrade, and the two-stage pipeline matters. The real-robot experiments on math-matching and toy-placement are at least real.\n\nBut the central claim does not hold up. Section 3.3 says the model is trained on 600 math-matching and 300 toy-placement trajectories with reasoning annotations, and the image-text mix includes TextVQA, GQA, RoboPoint, and 5k robotics VQA samples. That is explicit supervised training for OCR, arithmetic, and spatial reasoning inside the VLA. The abstract's \"despite these abilities not being explicitly trained\" is internally contradicted. The sentence about deliberately avoiding skill-specific training data sits right next to the description of that data. So Tables 1 and 2 cannot separate retention of pretrained knowledge from learning the task from 600+300 demonstrations.\n\nThe OOD claim is also unauditable. The paper doesn't list training/test equations, objects, card layouts, or direction-object pairs. If the held-out split has leakage—e.g., through GPT-4o reasoning augmentation—the open-world numbers would not mean what the authors say. They need to release the data and the exact splits.\n\nBaseline fairness is another soft spot. The baselines (OpenVLA, DexVLA, pi-zero, etc.) are not fine-tuned on the math-matching or toy-placement data, as far as the paper says. Comparing a fine-tuned ChatVLA-2 against zero-shot or differently-trained baselines is not a like-for-like comparison, and it inflates the improvement.\n\nThe ablations on expert count, top-k, and injection layer are run on the same benchmark that the final numbers come from, so those hyperparameters are tuned on the evaluation. That's a circularity in the reported configuration, though not necessarily fatal.\n\nWho is this for? Researchers working on VLA training recipes and knowledge retention will find the architecture and two-stage idea useful. But they should not take the open-world generalization numbers at face value until the data and baselines are cleaned up.\n\nRecommendation: This deserves a serious referee, not a desk reject, because the problem is important and the method is plausible. But it needs major revision: reconcile the training data with the \"not explicitly trained\" claim, release data/splits, and rerun fair baseline comparisons. If those don't change the results, it could be a solid paper.","headline":"Plausible architecture and real-robot results, but the 'untrained generalization' claim is contradicted by the paper's own training data.","tokens_in":15957,"tokens_out":3250,"would_cite":false,"duration_ms":29733,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ChatVLA-2 shows a robot can solve math and spatial tasks it was never trained on by keeping its pretrained vision-language knowledge intact.","keywords":["vision-language-action model","open-world embodied reasoning","mixture of experts","mathematical reasoning","spatial reasoning","knowledge retention","robot manipulation","reasoning following"],"falsifier":"Audit the training and augmentation logs: list every equation, card layout, object name, and spatial phrase in the 600 math-matching and 300 placement trajectories, the image-text samples, and the model-augmented reasoning phrases, then check for exact or near-duplicate overlap with the open-world test items. If any test equation or toy-and-direction combination appears, the open-world success rates do not establish generalization. A cleaner falsifier is a pre-registered held-out split generated after training with guaranteed zero overlap, rerun under the same protocol.","tokens_in":14952,"feed_emoji":"🤖","tokens_out":8629,"duration_ms":77246,"temperature":0.7,"pith_summary":"This paper tries to establish that a vision-language-action (VLA) robot model can keep the broad knowledge of its pretrained vision-language backbone—reading text, doing arithmetic, judging spatial relations—even after fine-tuning on robot data, and can turn that knowledge into real pick-and-place actions without an external planner. The authors build ChatVLA-2 with a dynamic mixture-of-experts layer in the language backbone and a two-stage training scheme: first co-train on image-text and robot data, then freeze the backbone and train only the action expert to follow the model's own reasoning. They test it on a math-matching game, where the robot reads a handwritten equation from a whiteboard and picks the correct number cards, and a toy-placement task, where it follows novel directional instructions with unseen objects. Their reported open-world results—3.58 out of 4 on OCR, 1.73 out of 2 on math reasoning, 82.7% manipulation success on math matching, and 81.4% on toy placement—would show that VLM pretraining can survive robot fine-tuning and drive generalization to instructions never seen in training. If true, this matters because it suggests end-to-end VLAs can be generalizable foundation models rather than narrow imitation learners.","feed_headline":"Robot solves unseen math and spatial tasks via pretrained knowledge","feed_subtitle":"A dynamic mixture-of-experts and two-stage training keep VLM skills alive during robot fine-tuning.","key_machinery":"The load-bearing mechanism is a dynamic mixture-of-experts layer inside the pretrained vision-language backbone. Rather than a fixed assignment of experts, a router selects a small subset of experts for each input, so some experts specialize in multimodal understanding, some in robot control, and some stay shared; the paper argues this preserves the intact LLM structure and avoids the knowledge loss that static experts cause. The second mechanism is a reasoning-following enhancement module that replaces the observation embedding with projected reasoning tokens and uses them to generate scale-and-shift conditioning in the latter half of a pretrained diffusion action expert. Together with a two-stage training scheme—co-training on image-text and robot data, then freezing the VLM while training only the action expert—these components carry the claim that actions can be made to follow the model's internal reasoning.","core_discovery":"The central claim is that a VLA can preserve and actively use the pretrained knowledge of its vision-language backbone instead of eroding it during robotic fine-tuning. On the authors' own terms, ChatVLA-2 demonstrates this by separating multimodal understanding and robot control into different expert pathways within a dynamic mixture-of-experts layer, while keeping a set of shared experts that carry transferable features such as spatial reasoning. A reasoning-following module then feeds the backbone's reasoning tokens into the latter half of the action-expert layers, and a second training stage freezes the VLM so the action expert learns to act on out-of-distribution reasoning. The authors report that this recipe yields near-perfect in-domain manipulation and, unlike all compared baselines, nonzero open-world success on unseen equations and unseen spatial instructions, including math and OCR scores that were never explicitly trained in the VLA pipeline.","pith_inferences":["A natural next test the paper leaves implicit is a contamination audit: release the full training and test equation and object sets so independent readers can verify that the open-world numbers are not boosted by leakage through the language-model reasoning augmentation.","The dynamic-MoE principle could transfer to other VLA backbones and embodiments; if the retention result is robust, one would expect mobile or long-horizon versions to show the same out-of-distribution reasoning advantage, not just tabletop pick-and-place.","Because the method is framed as preserving pretrained knowledge, its ceiling is set by what the VLM already knows; extending the recipe to a stronger reasoning backbone or tool-augmented arithmetic would be a testable way to push beyond that ceiling."],"forward_implications":["If the central claim holds, robot fine-tuning no longer has to trade away a VLM's pretrained capabilities, so a single model can keep answering visual questions and doing math while also controlling a robot.","Open-world instructions could be executed end-to-end: a robot could read a novel equation or hear a novel spatial phrase and produce a correct action without an external planner or symbolic reasoner.","The two-stage recipe gives a concrete way to diagnose and repair reasoning-following failures: co-training restores the knowledge, and the frozen-backbone second stage binds that knowledge to action.","The same architecture could make smaller robot datasets sufficient for generalization, since the heavy lifting is done by preserved pretrained knowledge rather than by in-domain demonstrations."],"supporting_citations":[{"why":"Base open-source VLA architecture with unstructured reasoning output that ChatVLA-2 extends.","marker":"[2]"},{"why":"Prior result showing robot fine-tuning erodes VLM knowledge, motivating the dynamic mixture-of-experts design.","marker":"[7]"},{"why":"Pretrained vision-language backbone whose multimodal knowledge the two-stage training aims to preserve.","marker":"[62]"},{"why":"Pretrained diffusion action expert that is conditioned on reasoning tokens.","marker":"[63]"},{"why":"Routing mechanism adapted for dynamic expert selection.","marker":"[64]"},{"why":"Image-text data used in stage-one co-training to keep object recognition alive.","marker":"[67]"},{"why":"Image-text data used in stage-one co-training to keep text-reading ability alive.","marker":"[68]"},{"why":"Image-text data used in stage-one co-training to keep visual reasoning alive.","marker":"[69]"},{"why":"Open-source vision-language-action baseline compared in the open-world manipulation tasks.","marker":"[10]"},{"why":"Flow-based VLA baseline that sets the in-domain manipulation performance bar.","marker":"[1]"}],"fun_headline_variants":["VLA keeps VLM smarts, nails unseen math and spatial tasks","Pretrained knowledge survives robot tuning, unlocks math and OCR","Mixture-of-experts VLA does math and spatial reasoning it never learned","VLM reasoning preserved in robot: solves unseen math and spatial tasks","Robot inherits VLM's open-world reasoning for unseen equations and cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the open-world test items are genuinely out of distribution: the handwritten equations, cards, toys, and directional phrases used at test time must not appear in the 600 math-matching trajectories, the 300 placement trajectories, the co-trained image-text data, or the language-model-augmented reasoning annotations.","fun_headline_variants_meta":{"raw":{"variants":["VLA keeps VLM smarts, nails unseen math and spatial tasks","Pretrained knowledge survives robot tuning, unlocks math and OCR","Mixture-of-experts VLA does math and spatial reasoning it never learned","VLM reasoning preserved in robot: solves unseen math and spatial tasks","Robot inherits VLM's open-world reasoning for unseen equations and cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001051,"raw_usage":{"total_tokens":4452,"prompt_tokens":1022,"completion_tokens":3430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":638,"completion_tokens_details":{"reasoning_tokens":3337}},"tokens_in":638,"tokens_out":3430,"duration_ms":21046,"temperature":1.0,"reasoning_tokens":3337,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:19:51.020684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Audit the training and augmentation logs: list every equation, card layout, object name, and spatial phrase in the 600 math-matching and 300 placement trajectories, the image-text samples, and the model-augmented reasoning phrases, then check for exact or near-duplicate overlap with the open-world test items. If any test equation or toy-and-direction combination appears, the open-world success rates do not establish generalization. A cleaner falsifier is a pre-registered held-out split generated after training with guaranteed zero overlap, rerun under the same protocol.","supporting_citations":[{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"Image-text data used in stage-one co-training to keep object recognition alive."},{"cited_title":"Towards vqa models that can read","cited_arxiv_id":null,"evidence_quote":"Image-text data used in stage-one co-training to keep text-reading ability alive."},{"cited_title":"Openvla: An open-source vision-language-action model","cited_arxiv_id":null,"evidence_quote":"Open-source vision-language-action baseline compared in the open-world manipulation tasks."}],"review_version":1}