{"id":"2b8fae9b-947e-42cd-8ae6-92c31c1858db","arxiv_id":"2607.07844","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"A dual-track zero-shot benchmark (DeepPlan semantic shift + actuation-noise drift) reveals imitation planners collapse under novel urban density and correlated noise while an RL planner remains more robust.","lead":"Shift & Drift is a dual-track benchmark that zero-shot tests autonomous-driving planners on novel European city layouts with dense pedestrians and on noisy vehicle dynamics. It shows imitation learners that look strong on standard tests fail hard under both shifts, while a reinforcement-learning planner degrades more gracefully.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"Threshold re-calibration for DeepPlan may systematically alter relative planner rankings without a sensitivity check.","rationale":"The Reader correctly isolates the hand-adjusted thresholds as the weakest assumption supporting the strongest empirical claim. The paper’s own sanity check (log score 90\to65) shows that defaults are too strict for the new domain, but that does not prove the chosen relaxations preserve comparative ordering. Because CLS is a hard-gated composite, even modest changes in the safety gates can re-rank methods that differ mainly in how close they come to the original bounds. No other methodological choice (single RL agent, log-playback filtering) is as directly load-bearing for the quantitative “up to 76 % degradation” and “≤8 % decay” numbers that drive the trade-off narrative. The concrete re-scoring test is cheap, fully determined by already-released code and data, and decisive: either the gaps survive under defaults (strengthening the claim) or they shrink (forcing a more cautious interpretation). Hence the verdict remains CONDITIONAL, exactly as the Reader concluded; no stronger rejection is warranted because the rest of the dual-track design is sound and the concern is addressable.","tokens_in":14471,"tokens_out":646,"duration_ms":8898,"concrete_test":"Re-score all five planners on the full 1 182 DeepPlan scenarios (NR and R) using the original nuPlan default thresholds listed in Table II; if any IL–CaRL CLS gap shrinks by more than ~15 points or the ordering of PlanTF/PLUTO/Diffusion vs. CaRL reverses, the headline trade-off claim is threshold-dependent and must be qualified.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that IL planners suffer large zero-shot CLS drops on DeepPlan (e.g., Diffusion Planner 89.87\to21.50 in Munich) while CaRL retains ~70–73 rests on the modified nuPlan thresholds in Table II (drivable-area violation 0.3\to3.0 m, min TTC 1.0\to0.5 s, relaxed jerk/accel bounds). The paper justifies the change solely by noting that log-playback score falls from ~90 to ~65 under defaults, arguing that defaults would “systematically penalize reasonable behavior.” It never reports the five planners’ CLS (or NCR/PER/TTC components) under the original thresholds, nor any ablation that varies the thresholds continuously. Because the composite CLS zeros a scenario on hard safety violations, a 10\times relaxation of the drivable-area bound and a 50 % relaxation of TTC can convert many near-miss or curb-clip events into non-zero scores. If IL methods produce more such borderline events than CaRL or PDM-Closed, the reported ranking gap is partly an artifact of the re-calibration rather than pure semantic-shift fragility. The same thresholds also apply to the combined DeepPlan+noise results in Table IV, so any inflation propagates into the robustness claim.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces Shift & Drift, a dual-track closed-loop benchmark for object-level autonomous-driving motion planners. The Semantic Shift Track converts the aerial DSC3D dataset into a nuPlan-compatible suite (DeepPlan: 1,182 scenarios across four German cities and San Francisco) to enable zero-shot evaluation of planners trained only on nuPlan’s North-American/Singaporean data, stressing novel topologies and dense pedestrian–cyclist interactions. The State-Distribution Drift Track injects AWGN and Ornstein–Uhlenbeck actuation noise into the ego command vector to quantify recovery from compounding execution error. Five SOTA planners (PDM-Closed, PlanTF, PLUTO, Diffusion Planner, CaRL) are evaluated under both tracks. The central empirical claim is that high-ID imitation-learning methods suffer large CLS drops under semantic shift (e.g., Diffusion Planner Val14 89.87 → Munich 21.50) and further collapse under temporally correlated OU noise, whereas the evaluated RL planner CaRL degrades more gracefully (DeepPlan CLS ~70–73; ≤8 % decay under high noise), revealing a trade-off between imitation fidelity and closed-loop resilience.","tokens_in":14774,"tokens_out":1368,"duration_ms":27979,"significance":"If the reported rankings hold under scrutiny of the scoring protocol, the work supplies a timely, large-scale, publicly released stress test that goes beyond i.i.d. nuPlan splits and handcrafted long-tail suites such as interPlan. The conversion pipeline that turns occlusion-free aerial trajectories into standardized nuPlan logs and maps, the dual-axis design (semantic + state-distribution), and the multi-seed noise results are concrete community assets. Explicit release of code and data further strengthens the contribution. The empirical contrast between IL fragility and RL resilience is actionable for paradigm selection and hybrid design, even if limited to the five evaluated agents.","major_comments":[{"comment":"§III-C and Table II: The DeepPlan CLS numbers that underwrite the central zero-shot claim (Table III) and the combined-stress claim (Table IV) rest on substantially relaxed hard-constraint thresholds (drivable-area violation 0.3→3.0 m, min TTC 1.0→0.5 s, relaxed jerk/acceleration bounds). The sole justification is that log-playback CLS falls from ~90 to ~65 under defaults. Because CLS zeros an entire scenario on any hard safety violation, a 10× relaxation of the curb bound and a 50 % relaxation of TTC can convert many near-miss or curb-clip events into non-zero scores. No planner-level CLS (or NCR/PER/TTC) under the original nuPlan thresholds, nor any continuous sensitivity sweep, is reported. Without that ablation it is impossible to separate genuine semantic-shift fragility from an artifact of re-calibration that may differentially favor or penalize the five methods.","section":"§III-C, Table II"},{"comment":"§III-A.4 (Post-Processing): Scenarios are discarded if the log-playback planner itself produces at-fault collisions, drivable-area violations, or insufficient progress, followed by manual route correction. The paper does not report the fraction discarded, the geographic or interaction-density distribution of the discarded set, or any comparison of planner rankings on the unfiltered versus filtered suite. Because the filter is defined by the same safety/progress criteria that later appear in the score, it risks systematically removing precisely the long-tail topologies that the Semantic Shift Track claims to stress, thereby inflating absolute CLS values and potentially compressing relative gaps.","section":"§III-A.4"},{"comment":"§IV-B / Table III and abstract: The paradigm-level language (“IL methods \\ldots exhibit significant failures”, “the evaluated reinforcement-learning-based planner demonstrates more graceful degradation”) is supported by only a single RL agent (CaRL). While the manuscript is careful in places to say “the evaluated” RL planner, the abstract and conclusions still frame an empirical trade-off between imitation and reinforcement learning as paradigms. With a single RL data point the claim remains suggestive rather than comparative; either additional closed-loop RL baselines or a clearer restriction of language to the specific agents is required for the load-bearing generalization.","section":"§IV-B, Abstract, §V"}],"minor_comments":[{"comment":"Fig. 1 caption and panel label contain the typo “State-Dristribution Drift Track” (missing ‘t’).","section":"Fig. 1"},{"comment":"§III-B.2: The choice of OU mean-reversion rate θ = 2.0 s⁻¹ (correlation time 0.5 s) is stated without reference to measured actuator or vehicle dynamics; a short justification or sensitivity note would help readers assess realism.","section":"§III-B.2"},{"comment":"Table I lists “Scenario Count 1118” for nuPlan while the text elsewhere uses Val14; a consistent reference (or explicit note that 1118 is the full Val14 split) would avoid confusion.","section":"Table I"},{"comment":"Fig. 3 qualitative panels are informative but the bottom-row captions are dense; labeling each sub-panel (a–e) and referring to them in the text would improve readability.","section":"Fig. 3"},{"comment":"Eqs. (1)–(4) use both continuous-time SDE and discrete Euler–Maruyama forms; explicitly stating the simulation Δt used for integration would make the noise process fully reproducible from the text alone.","section":"§III-B"}],"recommendation":"major_revision","confidential_remarks":"The conversion pipeline and dual-track design are genuine contributions and the empirical patterns look real; the main risk is that the DeepPlan ranking (and therefore the headline IL-vs-RL story) is partly an artifact of the un-ablated threshold change. If the authors supply the missing default-threshold and filter-statistics tables, the paper becomes a strong accept for a robotics/ML systems venue. Scope is appropriate for a journal that values benchmarks and reproducible evaluation protocols."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful thing here is the evaluation suite, not a new planner. They convert aerial DSC3D into 1,182 nuPlan-compatible scenarios (DeepPlan) across four German cities plus SF, then add a clean AWGN/OU actuation-noise track. That combination is new relative to nuPlan, interPlan, and ScenarioNet, and the occlusion-free multi-agent context is a genuine improvement for closed-loop testing.\n\nWhat they do well: the pipeline is specified (map conversion, ego selection, post-filtering), code/data are linked, and Tables III–IV give multi-city CLS plus multi-seed noise runs. The pattern is clear and reproducible: IL methods that sit at 84–93 CLS on Val14 drop hard on DeepPlan (overall ~34–37 NR; Diffusion Planner to 21.5 in Munich) and keep sliding under correlated OU noise, while CaRL stays near 70–73 with ≤8 % decay. Qualitative frames match the numbers. That is enough to change how people claim “generalization” on object-level planners.\n\nSoft spots, in proportion. The Table II threshold relaxations (drivable-area 0.3→3 m, TTC 1.0→0.5 s, looser jerk/accel) are justified only by log-playback falling from ~90 to ~65 under defaults. They never show the five planners under the original thresholds or any sensitivity sweep. Because CLS zeros on hard violations, that re-calibration can inflate absolute scores and possibly the ranking gap if IL produces more borderline events. Scenario filtering on log-playback success can also bias difficulty. And the RL claim rests on one agent (CaRL) whose reward is already aligned with nuPlan metrics. These are real but addressable; they do not invent the IL collapse or the conversion work.\n\nThis is for people who build or evaluate closed-loop AD planners and want a harder zero-shot + robustness protocol. It advances evaluation practice more than planning theory. I would send it to peer review; the data and protocol are solid enough to deserve referee time, with the threshold ablation as the main revision request. Worth engaging if you care about how we measure planner reliability.","headline":"Useful dual-track closed-loop stress suite (DeepPlan conversion + AWGN/OU noise) with clear IL fragility numbers; the threshold re-calibration is a real soft spot but does not erase the contribution.","tokens_in":15442,"tokens_out":544,"would_cite":true,"duration_ms":6438,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Imitation planners that ace standard driving tests collapse under new cities and actuation noise; the tested RL planner does not.","keywords":["autonomous driving","motion planning","zero-shot generalization","distribution shift","imitation learning","reinforcement learning","closed-loop simulation","actuation noise"],"falsifier":"Re-score all five planners on the full DeepPlan set with the original unadjusted nuPlan thresholds and check whether the large gap between the RL planner and the imitation planners shrinks, vanishes, or reverses.","tokens_in":15303,"feed_emoji":"🚗","tokens_out":722,"duration_ms":12203,"temperature":0.7,"pith_summary":"Closed-loop motion planners for autonomous driving look strong when tested on the same geographic and traffic distributions they were trained on, but that leaves two practical questions unanswered: do they still work on novel city layouts with dense pedestrians and cyclists, and can they recover when small execution errors accumulate? This paper introduces Shift & Drift, a dual-track benchmark that answers both. One track converts high-precision aerial recordings from four German cities and San Francisco into the standard nuPlan simulator so models trained only on North American and Singaporean data can be evaluated zero-shot on 1,182 new scenarios. The second track injects white Gaussian jitter and temporally correlated Ornstein–Uhlenbeck noise into the vehicle’s acceleration and steering commands. Across rule-based, imitation-learning, and reinforcement-learning planners, the paper shows that high imitation scores on the familiar benchmark hide large failures under semantic shift and under persistent drift, while the evaluated RL planner degrades far more gracefully. The result is a concrete empirical trade-off between how faithfully a planner copies expert driving and how resilient it remains once the world or the actuators leave the training distribution.","feed_headline":"Imitation planners collapse under new cities and noise","feed_subtitle":"A dual-track benchmark shows RL degrades far more gracefully than high-scoring IL models","key_machinery":"Shift & Drift dual-track benchmark: (1) DeepPlan, a conversion of the aerial DeepScenario Open 3D dataset into 1,182 nuPlan-compatible scenarios for zero-shot semantic-shift evaluation, and (2) controlled injection of AWGN and Ornstein–Uhlenbeck noise into ego acceleration and steering-rate commands to measure recovery from state-distribution drift.","core_discovery":"Imitation-learning planners that score highly on the standard in-distribution nuPlan validation set (closed-loop scores roughly 84–93) suffer large zero-shot drops on the new DeepPlan scenarios (overall scores roughly 34–37 in non-reactive mode, with one model falling 76 percent in Munich) and degrade further under temporally correlated actuation noise, whereas the evaluated reinforcement-learning planner retains substantially higher safety and progress (DeepPlan scores roughly 70–73, at most an 8 percent decay under high noise). The paper therefore claims an empirical trade-off between imitation fidelity and closed-loop resilience under semantic and state-distribution shift.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["IL planners collapse on new cities and noise; RL degrades gracefully","Zero-shot DeepPlan drops crush high-ID IL scores; RL holds steady","Semantic shift and drift expose IL fragility versus RL resilience","Imitation fidelity trades off against closed-loop resilience under shifts","RL retains safety on novel topologies where IL fails zero-shot"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The hand-relaxed safety and comfort thresholds used to score the new European pedestrian-heavy scenarios correctly measure planning quality rather than quietly masking failures or rearranging the ranking among the tested planners.","fun_headline_variants_meta":{"raw":{"variants":["IL planners collapse on new cities and noise; RL degrades gracefully","Zero-shot DeepPlan drops crush high-ID IL scores; RL holds steady","Semantic shift and drift expose IL fragility versus RL resilience","Imitation fidelity trades off against closed-loop resilience under shifts","RL retains safety on novel topologies where IL fails zero-shot"]},"model":"grok-4.5","effort":"low","cost_usd":0.004386,"raw_usage":{"total_tokens":1351,"prompt_tokens":882,"num_sources_used":0,"completion_tokens":72,"cost_in_usd_ticks":43860000,"prompt_tokens_details":{"text_tokens":882,"audio_tokens":0,"image_tokens":0,"cached_tokens":128},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":397,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":882,"tokens_out":72,"duration_ms":4719,"temperature":1.0,"reasoning_tokens":397,"cache_read_input_tokens":128,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T16:51:36.095045+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-score all five planners on the full DeepPlan set with the original unadjusted nuPlan thresholds and check whether the large gap between the RL planner and the imitation planners shrinks, vanishes, or reverses.","supporting_citations":[],"review_version":1}