{"id":"61ed25ea-22cd-49b4-830b-051bf6a91801","arxiv_id":"2504.14709","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"On a new Waymo-based causal benchmark, an imitation-plus-reinforcement planner (MTR-SAC) reaches goals more often than imitation alone but crashes and goes offroad more, contradicting the claimed safety improvement.","lead":"This paper introduces a Waymo-based closed-loop simulator and a causal benchmark designed to expose the copycat problem in imitation-learning planners, then proposes an IL-RL baseline (MTR-SAC) that improves goal completion but increases collisions and offroad events. A generalist reader might use the benchmark as a testbed for goal-conditioned planning, but the safety claim in the abstract is not supported by the paper's own tables.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"MTR-SAC's completion gain comes with substantially higher collision and offroad rates in Tables III–IV, so the abstract's 'safety' claim is contradicted by the paper's own measurements.","rationale":"The reader's verdict is REJECT, and the basis in the strongest claim is correct. The single most load-bearing issue is not the DFS goal-generation procedure, but the mismatch between the abstract's safety claim and the paper's own outcome tables. Tables III and IV define mutually exclusive outcome categories that sum to 1, so the comparison is internally consistent. MTR-SAC improves completion and reduces stuck, but it increases collision and offroad in both Test4k and Causality9k. Since the paper's reward design (Section III-C) explicitly treats collision and offroad as safety penalties, these metrics are the paper's own definition of safety. Under that definition MTR-SAC is less safe, not safer. The safety claim therefore fails on the reported evidence regardless of whether the causality benchmark is valid. The DFS concern raised by the reader is worth checking for the benchmark's causal interpretation, but it is not needed to reject the current paper: the central contribution's advertised safety improvement is contradicted by the authors' numbers. A concrete check is to recompute the no-safety-violation rate (1 - collision - offroad), which drops from 0.755 for IL-only to 0.509 for MTR-SAC on Causality9k; if this reproduces with hard termination and standard errors, the abstract should be revised and the paper's central claim withdrawn. Thus I recommend no change to the reader's REJECT verdict.","tokens_in":11230,"tokens_out":6976,"duration_ms":59205,"concrete_test":"Recompute the no-safety-violation rate as 1 - collision - offroad for each row of Tables III and IV. On Causality9k this is 0.755 for MTR-Close* IL-only (1 - 0.156 - 0.089) and 0.509 for MTR-SAC (1 - 0.223 - 0.268); on Test4k it is 0.837 for IL-only and 0.585 for MTR-SAC. If these drops reproduce when the simulator enforces hard termination at the first collision or offroad event and when per-seed variability is reported, the abstract's safety-improvement claim is refuted. This check isolates the safety claim from the separate question of whether the DFS-generated goals are realistic.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the joint IL-RL method MTR-SAC 'significantly improves the safety and adaptability' over imitation-only planning. The evidence offered is Tables III and IV. On Causality9k (Table IV), MTR-Close* IL-only gives completion 0.308, collision 0.156, offroad 0.089, stuck 0.447; MTR-SAC gives completion 0.496, collision 0.223, offroad 0.268, stuck 0.013. The four outcome categories are mutually exclusive and sum to 1.0 in every row, so these are not artifacts of overlapping labels. On the paper's own safety metrics, MTR-SAC increases collision by 43% and offroad by about 3x on Causality9k, and it increases collision from 0.122 to 0.273 and offroad from 0.041 to 0.142 on Test4k (Table III). The method therefore reaches the goal more often and gets stuck less often, but it does so while dramatically increasing the two events that the paper itself treats as safety violations. Unless 'safety' is redefined as 'not stuck'—which is inconsistent with the reward design in Section III-C, where collision and offroad are penalized as safety terms—the abstract's safety claim fails on the authors' own numbers. The benchmark may still be useful, but the headline contribution as stated is not supported.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses the \"copycat problem\" in imitation-learning-based planners by contributing three artifacts: (1) a lightweight closed-loop simulator for the Waymo Open Dataset that supports both IL and RL training; (2) a \"causality benchmark\" (Causality9k) in which alternative goal endpoints are generated for the same historical context via a DFS-based procedure; and (3) a joint IL-RL planner, MTR-SAC, that concatenates MTR's transformer features with the simulator state and trains a SAC policy. The authors claim in the abstract and introduction that MTR-SAC \"significantly improves the safety and adaptability\" over imitation-only baselines. The experiments compare MTR-Close* (IL-only), StateSAC (RL-only), and MTR-SAC on Test4k and Causality9k, and also ablate reward weights. The paper reports large gains in completion rate and large reductions in \"stuck\" rate for MTR-SAC, but also substantially higher collision and offroad rates.","tokens_in":11515,"tokens_out":4292,"duration_ms":39111,"significance":"If the causality benchmark were properly validated and the joint IL-RL method truly improved safety, this would be a worthwhile contribution: the community lacks lightweight PyTorch-based closed-loop simulators for Waymo data, and a causal evaluation protocol for the copycat problem is a good idea. The paper is also transparent about some limitations and points to a reproducible release. However, the central empirical claim as stated is not supported by the paper's own tables (collision and offroad rates worsen under MTR-SAC), and the benchmark's validity is not established because the goal-generation parameters are ad hoc and there is no reachability/difficulty analysis. The absence of error bars, test-set reward-weight tuning, and lack of comparison to existing IL-RL hybrids further weaken the evidence. The benchmark and simulator may be useful seeds for future work, but the paper in its current form does not substantiate its headline claims.","major_comments":[{"comment":"The abstract and Section I claim that MTR-SAC \"significantly improves the safety and adaptability\" of learned policies compared to imitation-only methods. On the paper's own safety metrics, the opposite holds. On Causality9k (Table IV), MTR-Close* (IL-only) has collision 0.156 and offroad 0.089, while MTR-SAC has collision 0.223 and offroad 0.268; on Test4k (Table III), collision rises from 0.122 to 0.273 and offroad from 0.041 to 0.142. These are exactly the events that the reward function in Section III-C penalizes as safety terms. The improvement is confined to completion rate and the \"stuck\" category, so the safety claim is contradicted by the paper's own measurements. This is a load-bearing problem for the central contribution.","section":"IV-D, Tables III and IV"},{"comment":"The reward weights are tuned on Causality9k, the same benchmark used for the final comparison, and the configuration selected for Table IV (1,1,10) is one of the rows of the tuning table. This is test-set tuning, and no held-out validation is reported. As a result, the completion advantage of MTR-SAC over IL-only in Table IV could reflect selection on the test set rather than a generalizable property of the method. The paper should report results on a separate validation split and give multiple-seed statistics with error bars for all metrics.","section":"IV-D-4, Table V, with IV-C"},{"comment":"The validity of the causality benchmark rests on the assumption that the DFS-generated goals are realistic, reachable, and comparable in difficulty to the original goals. The parameters in Algorithm 1 (CostTable {LC:5, TL:1, TR:1, Go:1}, MaxCostThres=10, NmsThres=2.5 m) are given without justification, and the paper provides no analysis of the distribution of generated goal distances, no reachability check under the bicycle model, and no measure of task difficulty across goals. Without such validation, completion or collision differences on Causality9k do not cleanly isolate the planner's response to endpoints; they could be artifacts of goal generation. This is crucial because the benchmark is the basis for the paper's causal claims.","section":"III-B, Algorithm 1"},{"comment":"The paper credits prior IL-RL hybrid methods such as Lu et al. [22], Huang et al. [23], and Booher et al. [24], but it does not compare MTR-SAC against any of them. The only RL baseline, StateSAC, has very poor safety metrics (collision 0.328-0.339 and offroad 0.560-0.563 on both benchmarks), which makes MTR-SAC look favorable by contrast. To support the statement that the joint IL-RL framework is a strong baseline, the paper needs at least one comparison to an existing hybrid method under the same simulator and evaluation protocol.","section":"IV-D-3 and II-A"}],"minor_comments":[{"comment":"The smoothness reward term is written as \"-0.5 * logit or(∆accel > 1.5, ∆turning > 0.1)\"; it is unclear whether \"logit\" is a typo for \"logical\" (i.e., a logical OR returning a binary flag) or a logistic function. Please clarify the notation and the intended functional form.","section":"III-C-2"},{"comment":"The caption contains a typo: \"COMPARSION\" should be \"COMPARISON\". Similarly, the tables use inconsistent column header spacing, e.g., \"Compl.\" vs \"UnCompl.\"; standardizing would improve readability.","section":"IV-D, Table II caption"},{"comment":"The limitations section states that certain aspects are overlooked \"such as the physical constraints of vehicles,\" but it does not explicitly acknowledge that the reported MTR-SAC results show large increases in collision and offroad rates relative to IL-only. This is directly relevant to the safety claim and should be stated plainly.","section":"VI, Limitations"},{"comment":"The abstract promises \"The code for this work will be released soon,\" but no repository link or artifact description is provided. Given that the simulator is a major contribution, the paper should either include a link or state the intended release platform.","section":"Abstract and I"}],"recommendation":"reject","confidential_remarks":"The reader's report and my own reading agree: the abstract's safety claim is directly contradicted by Tables III and IV, the reward weights are tuned on the test benchmark, and the causal benchmark lacks validation of the generated goals. These are not merely presentation issues. I would not rule out future value in the simulator and benchmark idea, but in its current form the manuscript's central claims are not supported, and the required fixes—retraining with safety-oriented weights, proper validation splits, baseline comparisons, and benchmark validation—are beyond a light revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part is the benchmark: generating alternative goals via DFS on Waymo and scoring planners closed-loop is a concrete new artifact, and a lightweight PyTorch simulator lowers the barrier for the community. The MTR-SAC baseline is conceptually simple, but it is a reasonable first step, and the limitations section is candid about the fusion strategy being open.\n\nThe problem is the headline. The abstract says the method significantly improves safety and adaptability over imitation-only planning. Adaptability holds: completion rises from 0.308 to 0.496 on Causality9k and stuck drops from 0.447 to 0.013. But on the paper's own safety metrics, MTR-SAC is worse: collision goes from 0.156 to 0.223 on Causality9k and from 0.122 to 0.273 on Test4k; offroad goes from 0.089 to 0.268 and from 0.041 to 0.142. The categories are mutually exclusive, so this is a real trade-off, and the reward in Section III-C explicitly penalizes collision and offroad as safety terms. You cannot rescue the claim by redefining safety as 'not stuck.' The abstract overclaims on the authors' own numbers.\n\nSecondary concerns are real but less severe. Reward weights were tuned on the same Causality9k set used for the final comparison, there are no error bars, no code is released despite the promise, and no prior IL-RL baseline is compared. The DFS cost table and thresholds are hand-set, and there is no validation that generated goals are always reachable or comparable in difficulty; that is the weakest link in the benchmark's causal interpretation. None of this kills the benchmark idea, but it means the evaluation needs substantial cleaning before the numbers can be trusted.\n\nThe citation pattern is fine; using MTR is natural even with the author overlap. Overall, this is a paper with a useful artifact and a central claim that fails on its own evidence. The safety overclaim is fixable with a revised abstract and a more careful evaluation, so I would send it to reviewers rather than desk reject it, but I would not cite it in its current form.","headline":"Useful causal benchmark undermined by an abstract safety claim the paper's own tables contradict.","tokens_in":12046,"tokens_out":2520,"would_cite":false,"duration_ms":22372,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"IL-RL doubles goal completion on new causal driving benchmark","keywords":["copycat problem","imitation learning","reinforcement learning","closed-loop simulator","causal benchmark","autonomous driving","Waymo Open Dataset","MTR-SAC"],"falsifier":"Run a reward-weight Pareto sweep for MTR-SAC on Causality9k and check whether any setting reaches both a completion rate at or above 49.6% and collision and off-road rates at or below the imitation baseline's 15.6% and 8.9%. If no point on that frontier dominates the imitation-only policy, the paper's claim that joint IL-RL improves safety alongside adaptability is not supported.","tokens_in":11024,"feed_emoji":"🚗","tokens_out":9613,"duration_ms":78013,"temperature":0.7,"pith_summary":"This paper argues that imitation-learned driving planners often cheat by extrapolating from the ego vehicle's own recent state rather than understanding the scene, a failure it calls the copycat problem. To expose and measure that failure, the authors build a lightweight closed-loop simulator and a causal benchmark, Causality9k, on top of the Waymo Open Dataset: each scenario keeps the same historical context but assigns new destinations generated by a depth-first search over lane-level actions (lane change, turn left, turn right, go straight). They then propose MTR-SAC, a simple joint imitation-reinforcement learning baseline that feeds MTR's transformer features together with the agent's position, velocity, and goal into an offline Soft Actor-Critic policy. In closed-loop simulation with MPC control and IDM-driven traffic, the imitation-only baseline completes 30.8% of the causal goals while MTR-SAC completes 49.6%, and the 'stuck' rate drops from 44.7% to 1.3%. The paper's central thesis is that endpoint-varying evaluation reveals whether a planner responds to goals at all, and that a joint IL-RL objective makes the learned policy far more adaptable than pure imitation.","feed_headline":"IL-RL doubles goal completion on new causal driving benchmark","feed_subtitle":"With goals changed, imitation-only planners finish 30.8% of scenarios; adding RL reaches 49.6%.","key_machinery":"The load-bearing object is Causality9k, a causal benchmark created by re-goaling Waymo scenarios. Algorithm 1 enumerates alternative paths with a depth-first search over lane-change, left-turn, right-turn, and straight actions, using a cost table (LC:5, TL:1, TR:1, Go:1) with a maximum total cost of 10, discards paths longer than the original distance to the goal, samples an endpoint at that distance on each surviving path, and applies non-maximum suppression at 2.5 meters so the goals are distinct. The second carrier is MTR-SAC, which fuses the imitation model's learned transformer features with a low-dimensional state (location, velocity, goal) as the observation for an offline Soft Actor-Critic policy trained from a replay buffer of simulated rollouts. Together they turn the 'planner fails in new situations' observation into a measurable gap: the difference in completion between a policy that responds to the goal and one that copies its history.","core_discovery":"The paper's central claim is that imitation-only planners such as MTR exhibit the copycat problem: they output trajectories by extrapolating from the ego vehicle's historical state instead of learning the mapping from the full scene and a specified destination. To demonstrate this, the authors construct Causality9k, which re-goals Waymo scenarios at the same travel distance using a DFS over lane-level actions with a cost table (lane change costs 5, turns and straight cost 1, total budget 10) and non-maximum suppression at 2.5 meters. In the resulting closed-loop evaluations, the MTR baseline that selects the predicted trajectory closest to the goal completes only 30.8% of the scenarios, while the proposed MTR-SAC, which concatenates MTR's pre-head features with the simulator state and trains an offline SAC policy with a five-term reward, completes 49.6% and reduces the fraction of vehicles that become stuck from 44.7% to 1.3%. The authors conclude that combining imitation and reinforcement learning in a single model overcomes the copycat limitation, and they present the simulator and causal benchmark as tools for assessing whether ML planners genuinely understand driving goals.","pith_inferences":["A natural next experiment is a full Pareto sweep over the five reward terms; the paper only varies three weights, and mapping the frontier would show whether the completion gain can be obtained without moving collision and off-road rates above the imitation baseline.","The DFS goal generator is specific to Waymo's lane graph; porting the same re-goaling idea to OpenDRIVE-based datasets such as nuPlan would test whether the copycat effect is a dataset artifact or a general IL failure.","An input-masking diagnostic could isolate the mechanism: on Causality9k, drop the ego vehicle's historical trajectory from the encoder input and measure the completion drop; a large drop indicates the planner was relying on ego extrapolation.","MTR-SAC's feature concatenation is the simplest point in the IL-RL fusion design space; comparing it with residual-action or KL-regularized combinations on the same benchmark would attribute the gain to the RL objective versus the feature representation."],"forward_implications":["Planners evaluated only on their original datasets overstate their generalization, because in those datasets goals and histories are correlated; Causality9k removes that confound.","The completion gap between MTR-Close* (30.8%) and MTR-SAC (49.6%) indicates that pure IL policies stall when the endpoint is novel, and an RL objective that rewards progress directly counters that.","The drop in the stuck rate from 44.7% to 1.3% suggests that the RL component learns to keep moving toward a goal rather than reproducing a memorized trajectory.","A closed-loop simulator with an MPC controller is necessary for fair evaluation: MTR's open-loop performance (47.4% completion) does not predict its closed-loop behavior in the paper's comparison."],"supporting_citations":[{"why":"Supplies the imitation learning baseline (MTR) whose features and predictions anchor both the pure-IL comparison and the MTR-SAC fusion.","marker":"[11]"},{"why":"Supplies the Soft Actor-Critic algorithm used for the RL component of MTR-SAC and for the StateSAC baseline.","marker":"[20]"},{"why":"Provides the Waymo Open Dataset scenarios that the simulator, training data, and Causality9k benchmark are built from.","marker":"[10]"},{"why":"Documents the ego-status shortcut that motivates the copycat framing the benchmark is designed to expose.","marker":"[7]"},{"why":"Provides the copycat-problem formulation in imitation learning that this paper's benchmark and IL-RL baseline target.","marker":"[9]"},{"why":"Waymax is the simulator whose design the new closed-loop simulator adopts features from.","marker":"[28]"},{"why":"Causality benchmark from Waymo that the paper compares against as a prior effort focused on agent removal rather than varied SDC goals.","marker":"[36]"},{"why":"Intelligent Driver Model controls NPC vehicles in the closed-loop evaluations, making the traffic reactive.","marker":"[21]"},{"why":"PDM is cited for the typical nuPlan evaluation scale (~1,118 scenarios) that the paper contrasts with Waymo's larger set.","marker":"[2]"}],"fun_headline_variants":["Causal benchmark exposes copycat planners; IL-RL lifts goal success to 49.6%","Imitation planners copy past states; IL-RL improves goal completion","New closed-loop simulator and causal benchmark reveal IL-RL outperforms imitation","IL-RL baseline overcomes copycat problem: 30.8% to 49.6% goal completion","Joint IL-RL planner fixes copycat issue on new causal driving benchmark"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The DFS-generated alternative goals are realistic, reachable, and similar in difficulty to the original goal, so that performance differences on Causality9k measure the planner's response to different endpoints rather than artifacts of impossible or trivially easy destinations.","fun_headline_variants_meta":{"raw":{"variants":["Causal benchmark exposes copycat planners; IL-RL lifts goal success to 49.6%","Imitation planners copy past states; IL-RL improves goal completion","New closed-loop simulator and causal benchmark reveal IL-RL outperforms imitation","IL-RL baseline overcomes copycat problem: 30.8% to 49.6% goal completion","Joint IL-RL planner fixes copycat issue on new causal driving benchmark"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000803,"raw_usage":{"total_tokens":3564,"prompt_tokens":1017,"completion_tokens":2547,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":2437}},"tokens_in":633,"tokens_out":2547,"duration_ms":17558,"temperature":1.0,"reasoning_tokens":2437,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:42:39.129109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a reward-weight Pareto sweep for MTR-SAC on Causality9k and check whether any setting reaches both a completion rate at or above 49.6% and collision and off-road rates at or below the imitation baseline's 15.6% and 8.9%. If no point on that frontier dominates the imitation-only policy, the paper's claim that joint IL-RL improves safety alongside adaptability is not supported.","supporting_citations":[{"cited_title":"Waymo open dataset,","cited_arxiv_id":null,"evidence_quote":"Provides the Waymo Open Dataset scenarios that the simulator, training data, and Causality9k benchmark are built from."},{"cited_title":"Resolving copycat problems in visual imitation learning via residual action prediction,","cited_arxiv_id":null,"evidence_quote":"Provides the copycat-problem formulation in imitation learning that this paper's benchmark and IL-RL baseline target."},{"cited_title":"Waymax: An accelerated, data-driven simulator for large-scale autonomous driving research,","cited_arxiv_id":null,"evidence_quote":"Waymax is the simulator whose design the new closed-loop simulator adopts features from."},{"cited_title":"Causalagents: A robustness benchmark for motion forecasting,","cited_arxiv_id":null,"evidence_quote":"Causality benchmark from Waymo that the paper compares against as a prior effort focused on agent removal rather than varied SDC goals."},{"cited_title":"Parting with misconceptions about learning-based vehicle motion planning,","cited_arxiv_id":null,"evidence_quote":"PDM is cited for the typical nuPlan evaluation scale (~1,118 scenarios) that the paper contrasts with Waymo's larger set."}],"review_version":1}