{"id":"0ea33059-574f-4bae-911f-9027f3411ebc","arxiv_id":"2508.06571","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"IRL-VLA fine-tunes a vision-language-action driving policy with PPO against a learned reward world model trained on NAVSIM's EPDMS metrics, reaching 74.9 EPDMS on navhard-real.","lead":"This paper trains a self-driving model that improves itself with a lightweight learned reward network instead of a heavy simulator, reporting top results on the NAVSIM v2 driving benchmark. The generalist takeaway: closed-loop driving skills may be trainable without expensive sensor simulation, using a learned stand-in for the simulator's scoring.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim rests on the Reward World Model remaining accurate on PPO-visited trajectories; without any RWM validation or distribution-shift check, the 74.9 EPDMS gain may be reward-model exploitation rather than a true closed-loop improvement.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the Reward World Model's validity under the RL policy's distribution shift. This is the most critical point because the paper's central quantitative claim is a 0.5-point EPDMS improvement over its own imitation baseline, and the subscore trends are consistent with the policy exploiting RWM prediction error. The paper provides no RWM validation accuracy, no calibration curve, and no analysis of how RWM error changes as PPO moves the policy away from the pretrained distribution. The additional underspecification of Algorithm 1 (transition model and critic) and the contradiction between 'without relying on simulator' and using simulator feedback to train the RWM reinforce the need for caution but are secondary to the reward-model validity question. The appropriate verdict remains CONDITIONAL: the framework is coherent and externally benchmarked, but acceptance should be conditioned on a direct validation of the RWM on RL-policy trajectories and a true-simulator evaluation of the final RL checkpoint.","tokens_in":11752,"tokens_out":4507,"duration_ms":53666,"concrete_test":"Evaluate the released IRL-VLA-RL and IRL-VLA-PT checkpoints on NAVSIM navhard-real using the true EPDMS simulator (not the RWM), over at least 3 seeds. Also collect trajectories from the RL policy, compute RWM-predicted vs simulator EPDMS per scene, and report calibration error, bias, and correlation. If the true simulator EPDMS of the RL policy is not at least as high as the PT policy, or if RWM prediction error is substantially larger on RL-policy trajectories than on the training distribution, the 74.9 result is attributable to reward-model exploitation rather than improved driving.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the RWM trained offline on trajectories from the pretrained policy (Sec. 3.3) remains an unbiased estimator of true NAVSIM EPDMS on trajectories visited by the policy after many PPO updates (Sec. 3.4). The paper reports no RWM validation accuracy and no distribution-shift check. Because PPO maximizes RWM-predicted reward, any systematic overestimate on out-of-distribution trajectories will be exploited. Table 1 shows exactly the signature of such exploitation: after RL, Ego Progress rises from 83.9 to 96.2, while NC falls from 98.3 to 96.9, DAC from 92.4 to 91.3, TTC from 97.1 to 96.2, and EC from 76.0 to 72.4. The net EPDMS gain is only 0.5 (74.4 to 74.9), so the result is fragile if the RWM is miscalibrated on the RL policy distribution. Additionally, Sec. 4 states the RWM is trained 'leveraging expert demonstrations and simulator feedbacks,' which undercuts the abstract's claim of training 'without relying on simulator.' Algorithm 1 also invokes an unspecified transition model \\(\\hat{T}\\) and critic, leaving the RL loop underspecified. The combination of marginal improvement, unvalidated learned reward, and reward-hacking-consistent subscore declines makes the central empirical claim insecure without additional evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IRL-VLA, a three-stage autonomous-driving framework: (1) imitation-learning pretraining of a VLA policy that combines semantic reasoning, 3D/BEV reasoning, and a diffusion-based planner; (2) training a lightweight Reward World Model (RWM) by inverse reinforcement learning to predict EPDMS sub-scores from sensor features and planned trajectories; (3) fine-tuning the diffusion VLA policy with PPO using RWM rewards, with an added imitation term and a KL regularizer. The authors report 45.0 EPDMS as first runner-up in the CVPR 2025 NAVSIM v2 challenge and 74.9 EPDMS on the navhard-real benchmark, claiming the first closed-loop VLA trained via end-to-end RL without a simulator.","tokens_in":12051,"tokens_out":6104,"duration_ms":73585,"significance":"If the learned RWM is an accurate and distribution-robust proxy for the EPDMS simulator, replacing sensor-rendering simulation with a lightweight reward model is a meaningful step toward scalable closed-loop training for large VLA policies. The evaluation is anchored to the external NAVSIM benchmark, so the final numbers are not produced by the authors' own machinery; this keeps the circularity risk low. The idea of applying PPO to the diffusion denoising chain with a learned reward model is also timely. However, the central empirical claim rests on a 0.5 EPDMS gain over the authors' own imitation baseline (74.4 to 74.9), and the paper does not validate the RWM on the policy's actual rollout distribution. Several load-bearing implementation details are also underspecified.","major_comments":[{"comment":"The RWM is the sole reward source for PPO (Eq. 5), yet no validation accuracy, calibration curve, or distribution-shift check is reported. If the RWM over-scores trajectories visited after PPO updates, the policy can exploit regression error rather than improve true EPDMS. Table 1 shows a pattern consistent with such exploitation: EP rises 83.9→96.2 while NC (98.3→96.9), DAC (92.4→91.3), TTC (97.1→96.2), and EC (76.0→72.4) all decline, with total EPDMS improving by only 0.5. Please report per-metric RWM accuracy on held-out navtrain data and on rollouts before/after RL, and show how RWM-predicted reward tracks true EPDMS during PPO training. Also justify excluding EC from RWM while the final EPDMS includes it.","section":"Sec. 3.3, 3.4; Table 1"},{"comment":"Algorithm 1 line 6 invokes an unexplained transition model \\hat{T}(s_t,a_t); no definition, training procedure, or ablation for \\hat{T} appears anywhere in the paper. Section 3.3 only defines the reward predictor, not a dynamics model. The critic/value function V_phi is also unspecified, and Sec. 3.4 refers to a 'critical network' rather than a critic. Without these components, the RL loop cannot be reproduced, and the claim of 'end-to-end reinforcement learning including sensor input' is not verifiable.","section":"Sec. 3.4, Algorithm 1"},{"comment":"The abstract and Contribution 1 say the VLA is trained 'without relying on simulator during training,' but Sec. 4 states that the RWM is trained 'leveraging expert demonstrations and simulator feedbacks.' The actual contribution is avoiding sensor rendering / high-fidelity simulation at RL time, not avoiding simulator feedback altogether. Please restate the claim precisely and list exactly which simulator computations are needed: RWM label generation, the two-simulation EC metric, and final evaluation.","section":"Abstract, Sec. 1, Sec. 4"},{"comment":"The headline '45.0 EDPMS as 1st runner up' is not reconciled with Table 1, which reports 74.9 EPDMS on navhard-real. If 45.0 is from a different challenge split/protocol, that protocol must be described; as written, the primary advertised result cannot be checked against the paper's own table. The authors should clearly state the relationship between the two numbers.","section":"Abstract, Sec. 1, Table 1"},{"comment":"All EPDMS scores are reported as single runs with no standard deviations or confidence intervals. The central RL improvement is 74.4→74.9 (0.5 points) and the best-vs-second w_IL gap in Table 3 is 1.0 point; these differences are within typical run-to-run variation for such benchmarks. Please provide multiple seeds / error bars, at least for IRL-VLA-PT, IRL-VLA-RL, and the w_IL ablation, so the reader can judge whether the claimed improvement is statistically meaningful.","section":"Table 1, Table 3"}],"minor_comments":[{"comment":"Typo: 'guidence' should be 'guidance.'","section":"Abstract"},{"comment":"The RWM loss is written as a weighted L1 error over all metrics, but Sec. 4 says BCE is used for binary metrics, MSE for [0,1] metrics, and cross-entropy for {0,0.5,1} metrics. Reconcile the two descriptions.","section":"Sec. 3.3, Sec. 4"},{"comment":"The KL term DKL(pi_theta || pi_ref) uses pi_ref, which is not defined before its first use. Also, the discount gamma over denoising steps 'mitigating instability in early denoising steps' is unclear because the summation index t=1..tau is not tied to the denoising direction.","section":"Sec. 3.4, Eq. (5)"},{"comment":"Variable T is overloaded: for t = 1 to T for trajectory collection, while l is already defined as trajectory length. Rename to avoid confusion.","section":"Sec. 3.4, Algorithm 1"},{"comment":"The caption references subfigures (e), (f), and (g), but the text and figure do not consistently describe what (g) contains; check numbering and ensure all panels are referenced.","section":"Fig. 2"},{"comment":"Reference [27] is missing venue/arXiv number; several other references are duplicated (e.g., DiffusionDrive appears as [20] and [21]).","section":"References"}],"recommendation":"major_revision","confidential_remarks":"This appears to be a competition report with strong 'first' and 'state-of-the-art' claims. The most important missing evidence is RWM validation under distribution shift; without it, the 0.5-point EPDMS gain is not convincing. I recommend major revision rather than rejection because the missing evidence is obtainable and the overall direction is not inherently flawed. If the authors cannot provide RWM accuracy or multi-seed results, I would consider the central claim unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is worth reading for the combination: imitation pretraining, a learned EPDMS reward model, and PPO fine-tuning of a full VLA policy on NAVSIM v2. That specific recipe is not in prior work — RecogDrive uses the simulator directly, GTRS learns scorers but not for RL rewards, DiffVLA is pure IL. The authors also build a genuinely self-contained VLA architecture with semantic and 3D reasoning plus a diffusion planner, and the ablation in Table 2 shows each piece contributes. The central quantitative claim is anchored to an external benchmark, the NAVSIM EPDMS scorer, not their own machinery, so circularity is low.\n\nBut the load-bearing assumption is the RWM's accuracy on PPO-visited trajectories, and the paper never reports RWM validation accuracy or any distribution-shift check. The subscore pattern — EP jumps from 83.9 to 96.2 while NC, DAC, TTC, and EC all drop — is consistent with PPO exploiting regression error rather than improving true driving. The net EPDMS gain is 0.5, so the result is fragile. Algorithm 1 also references a transition model T_hat that is never defined, and the critic network is only mentioned. The claim of training \"without relying on simulator\" is contradicted by Sec. 3.3 and the implementation details, where the RWM is trained using simulator feedback. That offline use of simulator labels is fine, but the abstract overstates it.\n\nThe ablations are thin: three values of w_IL chosen on the same navhard-real set, no variance, and missing baselines like DiffVLA and RecogDrive make it hard to see where the gain actually comes from. The stress-test concern lands. That said, the method is clearly written and reproducible in principle if the missing pieces are supplied.\n\nThis paper is for researchers working on closed-loop driving RL and VLA planners. It deserves a serious referee — the combination is novel and the problem (learned reward models to replace simulators) is real. But it needs another round: validate the RWM, specify or remove T_hat and the critic, correct the simulator-free framing, add DiffVLA and RecogDrive, report variance, and ideally show the RL gain on a held-out split. I'd send it to review, not desk reject, but I'd expect major revision.","headline":"A plausible three-stage recipe for closed-loop VLA training via a learned reward model; the central result is externally benchmarked but the reward model is unvalidated, so the 74.9 EPDMS gain could partly be reward hacking.","tokens_in":12662,"tokens_out":1954,"would_cite":false,"duration_ms":21004,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A simulator-free reward world model lets reinforcement learning fine-tune a VLA driving policy, and the result beats imitation-only training on NAVSIM v2.","keywords":["vision-language-action","closed-loop reinforcement learning","reward world model","inverse reinforcement learning","diffusion policy","autonomous driving","NAVSIM","PPO"],"falsifier":"Compare RWM-predicted EPDMS with true simulator EPDMS on trajectories sampled from the final PPO policy; if predictions are systematically optimistic, or if per-metric prediction error grows across PPO iterations, the RL gain is an artifact of the learned reward. A sharper version: retrain the RWM with EC included and check whether the 74.9 navhard-real EPDMS gain survives.","tokens_in":11537,"feed_emoji":"🚗","tokens_out":10310,"duration_ms":107307,"temperature":0.7,"pith_summary":"The paper tries to show that a vision-language-action (VLA) driving model can be improved by closed-loop reinforcement learning without a simulator, by replacing simulator-based reward computation with a lightweight learned Reward World Model (RWM). The pipeline first imitates human driving with a diffusion-based VLA planner, then trains the RWM on offline trajectories and their EPDMS sub-scores, and finally fine-tunes the policy with PPO using RWM rewards plus a behavior-cloning regularizer. On the NAVSIM navhard-real split, the RL-tuned model reaches 74.9 EPDMS versus 74.4 for the same model after imitation pretraining, and the approach reports first runner-up placement in the challenge benchmark. If correct, this matters because it offers a scalable path to closed-loop VLA training that avoids high-fidelity sensor simulation and the sim-to-real gap.","feed_headline":"Reward world model lifts VLA driving score to 74.9","feed_subtitle":"A vision-language driving policy improves via PPO on a learned world-model reward—no simulator needed for closed-loop training.","key_machinery":"The load-bearing object is the Reward World Model (RWM): an inverse-RL trained network that maps each trajectory's BEV-space features to per-metric predictions, $$\\hat{r}_m = \\mathrm{MLP}_m(f_{\\mathrm{traj}}),\\quad \\hat{r}_{\\mathrm{epdms}}=\\sum_m w_m \\hat{r}_m,$$ with $m \\in \\{\\text{NC, DAC, DDC, TLC, EP, TTC, LK, HC}\\}$ and weights from EPDMS. It replaces simulator-based scoring during PPO. The second mechanism is the diffusion-policy-as-MDP view: each denoising step is a transition modeled by a Gaussian policy $\\pi_\\theta(x_{t-1}|x_t,c)$, whose joint log-likelihood forms the PPO objective with KL-to-reference and an imitation term.","core_discovery":"IRL-VLA is a three-stage method for training a VLA driving policy in closed loop without a simulator. Stage one is imitation pretraining of a VLA whose planner is a conditional diffusion model over anchored trajectories, conditioned on semantic (VLM) and 3D (BEV, detection, map) reasoning. Stage two trains a lightweight Reward World Model: for each of eight EPDMS sub-metrics it predicts the sub-score from BEV trajectory features, and the sub-scores are weighted into a single reward. The training data deliberately includes intermediate diffusion steps, K-means-sampled trajectory patterns with K from 32 to 8192, and multiple ego poses to cover diverse behavior. Stage three fine-tunes the diffu","pith_inferences":["A testable consequence of excluding EC from the RWM: the RL stage is blind to Extended Comfort, and the reported table shows EC dropping from 76.0 to 72.4 after RL; including EC in the reward would probably shrink the net EPDMS gain.","The paper does not report any distribution-shift validation of the RWM; measuring RWM prediction error on post-PPO trajectories would decide whether the improvement is a real policy gain or exploitation of reward-model regression error.","The RWM could also be used at inference time as a differentiable guide for diffusion denoising (reward-guided trajectory scoring), giving a cheaper way to exploit the same learned reward without further policy updates.","If the RWM stays unbiased for other sensor modalities, the same BEV-trajectory-feature interface should let IRL-VLA extend to LiDAR- or radar-only inputs; that is a direct experiment suggested by the architecture."],"forward_implications":["Closed-loop RL for VLA policies is feasible without high-fidelity sensor simulation: the RWM supplies real-time per-trajectory rewards, so PPO no longer needs rendering or physics simulation.","RL fine-tuning changes the trade-off surface: Ego Progress rises from 83.9 to 96.2 while most safety and comfort sub-scores fall slightly, yet net EPDMS improves from 74.4 to 74.9 on navhard-real.","The balance between RL and imitation matters: imitation weight 0.5 gives 74.9, while 1.0 and 0.1 give 73.9 and 73.4, so the diffusion policy needs the behavior-cloning term for stable optimization.","Because the RWM is trained offline on diverse samples (intermediate diffusion steps, K-means trajectory modes, multiple ego poses), the same reward infrastructure can be reused across many scenes without rerunning a simulator per policy update."],"supporting_citations":[{"why":"Defines the EPDMS score and its per-metric weights that the Reward World Model is trained to reproduce.","marker":"[3]"},{"why":"Supplies the NAVSIM dataset, the non-reactive simulation, and the benchmark scoring used for reward data collection and final evaluation.","marker":"[6]"},{"why":"Provides the visual-language-model command-guidance module adopted by the semantic-reasoning branch of the VLA architecture.","marker":"[14]"},{"why":"Contributes the group-standardized advantage and joint log-likelihood recipe used in the PPO update of the diffusion policy.","marker":"[17]"},{"why":"Establishes the diffusion-policy-as-MDP formulation that lets PPO optimize the denoising process.","marker":"[22]"},{"why":"Formalizes the conditional Gaussian policy likelihood of each denoising step, the quantity the RL objective maximizes.","marker":"[2]"},{"why":"Generalized advantage estimation supplies the advantage values used in the PPO objective.","marker":"[24]"},{"why":"DiffusionDrive is a strong IL baseline in Table 1 (63.2 EPDMS) that the pretrained model (74.4) outperforms, motivating the RL extension.","marker":"[20]"},{"why":"Generalized trajectory scoring supplies the trajectory-scoring idea and Table 1 baselines (73.1 and 74.3 EPDMS) that the pretrained model beats.","marker":"[19]"}],"fun_headline_variants":["Closed-loop VLA training via learned reward, no simulator needed","IRL-VLA beats sim-free closed-loop driving with PPO on world-model reward","World-model reward enables simulator-free fine-tuning of VLA driving","Reward world model lets VLA learn closed-loop without sensor sim","VLA hits 74.9 NAVSIM via IRL reward model and PPO"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The learned Reward World Model, trained once on offline trajectories and scores, stays accurate on the different trajectories the policy visits during PPO; if it over-scores those, PPO will maximize the model's mistakes rather than driving quality.","fun_headline_variants_meta":{"raw":{"variants":["Closed-loop VLA training via learned reward, no simulator needed","IRL-VLA beats sim-free closed-loop driving with PPO on world-model reward","World-model reward enables simulator-free fine-tuning of VLA driving","Reward world model lets VLA learn closed-loop without sensor sim","VLA hits 74.9 NAVSIM via IRL reward model and PPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000478,"raw_usage":{"total_tokens":2232,"prompt_tokens":796,"completion_tokens":1436,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":1349}},"tokens_in":540,"tokens_out":1436,"duration_ms":11491,"temperature":1.0,"reasoning_tokens":1349,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:35:36.345565+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare RWM-predicted EPDMS with true simulator EPDMS on trajectories sampled from the final PPO policy; if predictions are systematically optimistic, or if per-metric prediction error grows across PPO iterations, the RL gain is an artifact of the learned reward. A sharper version: retrain the RWM with EC included and check whether the 74.9 navhard-real EPDMS gain survives.","supporting_citations":[{"cited_title":"Navsim: Data-driven non-reactive autonomous vehicle simulation and benchmarking","cited_arxiv_id":null,"evidence_quote":"Supplies the NAVSIM dataset, the non-reactive simulation, and the benchmark scoring used for reward data collection and final evaluation."},{"cited_title":"Training diffusion models with reinforcement learning, 2024","cited_arxiv_id":null,"evidence_quote":"Formalizes the conditional Gaussian policy likelihood of each denoising step, the quantity the RL objective maximizes."},{"cited_title":"High-dimensional continuous control using generalized advantage estima- tion, 2018","cited_arxiv_id":null,"evidence_quote":"Generalized advantage estimation supplies the advantage values used in the PPO objective."}],"review_version":1}