{"id":"cbe8e5b8-d1b6-4776-8a3a-bced7283c232","arxiv_id":"2506.09800","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":8,"one_line_summary":"R2SE refines pretrained end-to-end driving policies on hard cases via residual LoRA reinforcement learning and switches between specialist and generalist policies using GPD-based uncertainty.","lead":"This paper presents R2SE, a pipeline that retrains an end-to-end driving model on hard cases with reinforcement learning while using small adapters and an uncertainty gate to avoid forgetting. The authors report top benchmark numbers, but key formal statements about the method contradict each other.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The self-aware expansion gate is inverted between Eq. 15/Algorithm 2 and the prose/ablation, so the paper does not specify which policy actually runs at test time; this is load-bearing for the central claim.","rationale":"The reader's REJECT is well-founded. I focus on one load-bearing inconsistency because it alone would undo the central claim. Eq. 15 and Algorithm 2 specify specialist use for P_GPD > sigma, but the prose and Fig. 2 specify the reverse, and Table VIII's ER-versus-sigma trend is only consistent with the reverse. If the implemented system follows Eq. 15, expansion is used exactly on high-uncertainty/OOD inputs, the opposite of the intended safeguard; if it follows the prose, the paper's formal definition is wrong. Either way, the reader cannot audit the test-time policy from the manuscript. Additional sign errors (KL objective, entropy difficulty, Eq. 5's inequality) are in the same direction and compound this. The absence of released code, data, and error bars means there is no external artifact to disambiguate. I do not claim fraud; the inconsistency may be a correctable typo, but it is load-bearing because the central contribution is the expansion mechanism. Thus the reader's REJECT remains appropriate. Agreement is partial because the reader also flagged this cluster, although their weakest_assumption emphasized PDMS circularity; I judge the expansion-gate inversion as the more decisive internal problem.","tokens_in":29140,"tokens_out":5635,"duration_ms":71187,"concrete_test":"Ask the authors to release the gating code, or independently re-implement both variants on the Hydra-MDP base and run navtest at sigma=0.05, 0.25, 0.50, 0.75, 0.95. Reproduce the Expand Rates in Table VIII: if only the gate 'use specialist when P_GPD(U) <= sigma' matches the reported monotonic increase (2.09 to 82.8), then Eq. 15 and Algorithm 2 are inverted relative to the executed system. Then compare PDMS of the two gates on navtest; if the implemented gate obtains 91.6 and the Eq. 15 gate does not, the paper's formal description is wrong at exactly the point that matters, and the central numerical claim cannot be attributed to the described algorithm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-C defines the expansion policy in Eq. 15 as using the specialist when P_GPD(U_test) > sigma and the generalist when P_GPD(U_test) <= sigma, and Algorithm 2 repeats this. The prose immediately above Eq. 15 states the opposite: OOD/high-uncertainty cases should revert to the generalist, and only certain cases should use the specialist. The ablation in Table VIII reports Expand Rate increasing with sigma (2.09 at sigma=0.05 to 82.8 at sigma=0.95). Under Eq. 15, raising sigma makes the specialist condition P_GPD > sigma harder to satisfy, so ER would fall; the observed rise is only consistent with the opposite gate (specialist when P_GPD <= sigma). Since the self-aware adapter expansion is the component credited with the +1.9 PDMS gain in Table III (ID-4 vs ID-0), an inverted gate would mean the system deployed at test time is not the one described, and the claimed anti-forgetting behavior is not established. The same section contains related sign inconsistencies (max vs. min KL in Eq. 1; F_Ent sign in Eq. 3; and the subset inequality in Eq. 5), reinforcing that the written specification cannot be used to determine the executed policy. This is not a mild typo: the central mechanism's behavior changes qualitatively under the two readings.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes R2SE, a model-agnostic pipeline for refining end-to-end autonomous driving policies. A generalist IL policy is first pretrained; hard cases are identified from closed-loop log simulation using PDMS and an entropy-based uncertainty term; LoRA adapter ensembles are fine-tuned on these cases with GRPO under cost constraints; and at test time a GPD-based uncertainty estimate is used to decide whether to \"expand\" to the specialist adapters or fall back to the generalist. Experiments are reported on the NAVSIM nuPlan leaderboard and on CARLA Bench2Drive, with ablations over refinement components, adapter ranks, hard-case thresholds, expansion distributions, and expansion confidence. The main reported results are a PDMS gain from 87.0 (Hydra-MDP baseline) to 91.6 for R2SE-MDP, and improved driving score / success rate on Bench2Drive relative to several end-to-end baselines.","tokens_in":29380,"tokens_out":7209,"duration_ms":79254,"significance":"If the mechanism were as described in the prose, R2SE would be a useful contribution: the pipeline is model-agnostic, combines IL pretraining with cost-constrained RL fine-tuning, and explicitly targets anti-forgetting via low-rank adapters and uncertainty-gated expansion. The empirical breadth is a strength: two closed-loop benchmarks, multiple base architectures (categorical and diffusion), test-time adaptation variants, and ablations covering the main design choices. However, the formal specification of the core mechanism is internally inconsistent. The expansion gate in Eq. (15) and Algorithm 2 is inverted relative to the prose and to the ablation trend in Table VIII, and Eqs. (1), (3), and (5) contain sign/inequality errors that affect the stated pretraining objective, the difficulty score, and the hard-case selection rule. Because the expansion step is credited with a +1.9 PDMS gain in Table III and with the anti-forgetting behavior, the manuscript does not currently pin down which policy is actually executed at test time. These issues are fixable, but they are load-bearing for the central claim.","major_comments":[{"comment":"The adapter expansion gate is specified with opposite semantics in different places. Eq. (15) uses the specialist when P_GPD(U_test) > σ and the generalist when P_GPD(U_test) ≤ σ, and Algorithm 2 lines 9–12 repeat this. The prose immediately above Eq. (15) states that OOD/high-uncertainty cases should revert to the generalist, with the specialist used only for confident cases. Table VIII reports Expand Rate increasing from 2.09 at σ=0.05 to 82.8 at σ=0.95. Under Eq. (15), increasing σ makes the specialist condition harder to satisfy, so Expand Rate should fall; the observed rise is consistent only with the opposite gate (specialist when P_GPD ≤ σ). Since the expansion component is credited with a +1.9 PDMS gain in Table III (ID-4 vs. ID-0) and with the paper's anti-forgetting claim, the manuscript does not currently specify which policy actually runs at test time. The authors must correct Eq. (15)/Algorithm 2 to match the implemented policy and confirm that the ablation trend in Table VIII is interpreted consistently (a rising Expand Rate cannot simultaneously mean more frequent fallback to the generalist).","section":"§III-C, Eq. (15), Algorithm 2; §IV-C5, Table VIII"},{"comment":"The behavior-cloning term is written as max KL(π_exp∥πθ) = max Σ_M π_exp(X) log(π_exp(X)/πθ(X)). This maximizes the divergence from the expert distribution rather than matching it; the standard BC objective is to minimize KL(π_exp∥πθ), which is equivalent to maximizing E_{π_exp}[log πθ(X)] when the entropy of π_exp is constant. As written, Eq. (1) also has the wrong sign for the stated pretraining objective, since subtracting a positive KL term would reward divergence from the expert. This needs correction because the generalist policy is the foundation on which the specialist fine-tuning and expansion are built.","section":"§III-A1, Eq. (1)"},{"comment":"The difficulty score has sign problems that contradict the stated monotonic relationship with case difficulty. Eq. (3) defines F_Ent(Ŷ) as Σ_m πθ(ŷ_m|X) log πθ(ŷ_m|X), which is the negative of Shannon entropy; with this sign, high planning uncertainty decreases F_X in Eq. (4) and can even make it negative, exactly opposite to the intended \"difficulty increases with uncertainty\" behavior. In addition, Eq. (5) selects D_hard = {d | F_X ≤ Q_ϵ(F)}, which is the set of lowest-difficulty cases, while the text and implementation (Sec. IV-A3, \"99th percentile (ϵ=1)\") describe selecting the hardest cases. The authors should either define F as a loss-like quantity consistently or change the inequality in Eq. (5) so that hard-case allocation matches the reported experiments.","section":"§III-A2, Eqs. (3)–(5)"}],"minor_comments":[{"comment":"The sentence \"Following GRPO, we emit the clip term for simplicity, and KL(·||·) term as it already presents in L_pretrain\" is unclear: the final objective in Eq. (10) does not show the clip, the KL, or the relative coefficient, so the exact GRPO surrogate used in the experiments is not fully specified.","section":"§III-B2, Eq. (10)"},{"comment":"Theorem 1 states a PAC-Bayes bound for the adapter ensemble, but the proof is not actually supplied; the sentence \"This follows directly from the PAC-Bayes bound\" is not a derivation. If this theorem is meant to support a formal guarantee, the proof should be included.","section":"Appendix C, Theorem 1"},{"comment":"Algorithm 1 line 22 writes the combined loss as L_RL + α_Pretrain L_Pretrain, but α_Pretrain is not defined in the main text or in Table IX. Please define this coefficient and state its value for each experimental setup.","section":"§IV-A3, Algorithm 1"},{"comment":"The text states that EWC has overhead \"exceeding ×4.1 MFLOPs\" while Fig. 5b reports FLOPs values in units of M; please clarify whether the values are mega-FLOPs or relative multipliers, and make the text consistent with the figure.","section":"Fig. 5b and §IV-B4"},{"comment":"The column ER is introduced as \"Expand Rate\" but the surrounding text describes it as a measure of fallback to the generalist. Please define ER explicitly and align the terminology with the gate in Eq. (15).","section":"§IV-C5, Table VIII"}],"recommendation":"major_revision","confidential_remarks":"The reader's report raised circularity because PDMS is used for hard-case selection, as RL reward, and as the main evaluation metric. I agree this is a concern, though the independent CARLA Bench2Drive metric partially mitigates it. The more immediate blocker is the internal inconsistency of the expansion gate and the sign errors in Eqs. (1), (3), and (5); the authors should be asked to state explicitly which policy was executed in each experiment and to confirm that the ablation tables correspond to that policy. Given the breadth of the empirical study, the paper is worth a major revision rather than rejection, provided these formal issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The R2SE paper is worth a look, but read the equations carefully before believing the pipeline. The core idea — use a closed-loop score to find hard cases, fine-tune LoRA adapters with constrained RL, then gate the specialist by uncertainty — is sensible, and the experiments are broad. The authors show consistent PDMS gains on nuPlan and DS/SR gains on CARLA across two base planners, and the ablation roadmap (Table III) plus forgetting comparison (Fig. 5) are genuinely informative. If the method holds up, it is a useful recipe.\n\nThe problem is that the written method does not pin down the executed policy. The stress-test concern is correct: Eq. 15 and Algorithm 2 expand to the specialist when P_GPD(U_test) > σ, but the prose says revert to the generalist on OOD/high uncertainty, and the ablation (Table VIII) shows Expand Rate increasing with σ, which only matches the prose gate. That inversion is not cosmetic — expansion is credited with +1.9 PDMS in Table III. On top of that, Eq. 1 writes BC as maximizing KL, Eq. 3 drops the minus sign in entropy, and Eq. 5 selects hard cases with FX ≤ Q_ϵ, which is backwards. Together these errors mean a reader cannot reconstruct the system.\n\nThere is also a real circularity issue: PDMS is used to select hard cases, as the RL reward, and as the evaluation metric on nuPlan. The CARLA results partially answer that criticism because they use different metrics, but the nuPlan gains should be treated as benchmark-optimization evidence, not independent validation. No code, data, or error bars are provided, which makes the numerical claims hard to verify.\n\nMy read: the intended method is recoverable from the prose, and the empirical core is plausible. But the current text does not specify the test-time policy, so acceptance would be premature. It deserves a serious referee — the experiments are substantial and the approach is relevant — but the referee should demand corrected equations, a consistent expansion rule, and ideally code or per-seed variance.\n\nRecommendation: send to peer review, expect major revision.","headline":"Plausible refinement pipeline with strong experiments, but the paper as written inverts its own expansion gate and contains other sign errors, so the executed test-time policy is underspecified.","tokens_in":29963,"tokens_out":4930,"would_cite":false,"duration_ms":50845,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"R2SE: a self-aware refinement loop that turns imitation-learned end-to-end driving into a hard-case specialist without forgetting the generalist.","keywords":["end-to-end autonomous driving","reinforcement learning","imitation learning","catastrophic forgetting","low-rank adapters","uncertainty estimation","closed-loop simulation","hard-case refinement"],"falsifier":"Run the same R2SE-refined policy in a reactive closed-loop simulator (or on a real vehicle) on the navtest scenarios and compare the score gap against the non-reactive PDMS gap: if the refined policy no longer outperforms the baseline, or if the improvement reverses, then the claim that non-reactive PDMS is a faithful training and evaluation signal for hard-case refinement is falsified.","tokens_in":28883,"feed_emoji":"🚗","tokens_out":4252,"duration_ms":48830,"temperature":0.7,"pith_summary":"The paper proposes a learning pipeline, R2SE, that refines an already-trained end-to-end driving policy on hard, failure-prone cases while preserving the general driving skill learned by imitation. The motivation is that pure imitation learning generalizes poorly to corner cases, while pure reinforcement learning overfits and catastrophically forgets general knowledge. R2SE's central claim is that a residual correction trained by cost-constrained reinforcement on a small set of hard clips, combined with a test-time uncertainty gate, can improve hard-case performance and keep the generalist intact. The authors report closed-loop gains on the NAVSIM navtest leaderboard (PDMS from 87.0 for the Hydra-MDP baseline to 91.6) and on CARLA Bench2Drive, with lower forgetting rates than standard fine-tuning, EWC, or dynamic networks.","feed_headline":"Hard-case refinement lifts driving score from 87.0 to 91.6","feed_subtitle":"Reinforced LoRA specialists plus an uncertainty gate improve hard scenarios while preserving general driving skill.","key_machinery":"The load-bearing machinery is a three-part closed loop: (1) a difficulty score $F_X$ that fuses the closed-loop PDM Score, normalized perception loss, and SOTIF Shannon entropy to select the hard-case subset $\\mathcal{D}_{\\text{hard}}$; (2) a residual reinforced specialist composed of $K$ low-rank adapters (LoRA) attached to detached generalist weights, trained by GRPO with a constrained-MDP penalty where rewards and costs derive from non-reactive log simulation; and (3) a self-aware expansion policy that models the specialist's ensemble uncertainty with a Generalized Pareto Distribution (GPD) and activates the specialist only when the GPD cumulative probability exceeds a threshold $\\sigma$. The GPD tail model is the piece that lets the system decide when a specialist is confident enough to overrule the generalist, and the adapter ensemble provides the uncertainty estimate for free.","core_discovery":"The key discovery is that hard-case refinement can be decoupled from general-knowledge retention by freezing the pretrained policy and attaching a small ensemble of low-rank adapters that learn only residual corrections. These adapters are trained with a GRPO-based reinforcement learning objective that includes cost constraints, using non-reactive log simulation to provide dense per-step process rewards and costs. At test time, the system fits a Generalized Pareto Distribution to the ensemble uncertainty observed on the hard-case training set and expands the specialist adapter only when the test input falls within that tail; otherwise it reverts to the generalist. This mechanism yields a PDMS of 91.6 on the NAVSIM navtest split (up from 87.0 for the Hydra-MDP baseline), the highest reported driving score and success rate on CARLA Bench2Drive among compared end-to-end systems, and a lower forgetting rate than full fine-tuning, EWC, and dynamic networks.","pith_inferences":["Since non-reactive PDMS is both the training reward and the evaluation metric, the reported gains could partly reflect optimization toward that specific closed-loop proxy; a reactive simulator or real-vehicle test would be needed to confirm that the refinement generalizes to true closed-loop safety.","The GPD gate depends on a hard-case uncertainty buffer collected offline; under distribution shift beyond that buffer, the gate may silently revert to the generalist, so continual expansion of the hard-case buffer would be needed for long-term deployment.","The data-efficiency numbers suggest that logged-data-only RL refinement of large end-to-end models is feasible in practice; extending R2SE with generative scene or behavior augmentation, as the paper lists as future work, could remove the residual dependence on having hard clips already present in the log.","Because the method is architecture-agnostic, it could plausibly be applied to vision-language-model planners or world-model planners that currently address data rarity but not the hard-case underfitting that behavioral cloning leaves unsolved."],"forward_implications":["Any pretrained end-to-end planner, whether categorical, Gaussian-mixture, or diffusion-based, can be refined on hard cases by attaching LoRA ensembles and running the same GRPO-plus-cost loop, since the RL objective reduces to a negative log-likelihood for categorical/GMM policies and to conditional-guidance optimization for diffusion policies.","The hard-case allocation step means only a small percentage of training data (top 1% by difficulty) drives refinement; the paper shows that roughly 500 hard clips (0.45% of the training set) already produce most of the PDMS gain, so the pipeline is substantially more data-efficient than full RL fine-tuning.","Forgetting is structurally bounded: generalist weights are detached and adapters are low-rank, so refinement cannot overwrite the pretrained policy, and the GPD gate reverts to the generalist on out-of-tail inputs.","The same adapter ensemble yields uncertainty estimates that plug into online adaptation schemes, as demonstrated by the paper's R2SE-TTT and R2SE-RS variants, which further improve PDMS beyond the base refinement.","If the claims hold, the standard practice of fine-tuning an end-to-end driving policy on all available data can be replaced by targeted, logged-only refinement on the few hard cases, which would make continual improvement of deployed systems more practical."],"supporting_citations":[{"why":"Defines the NAVSIM non-reactive log-simulation benchmark and the PDM Score metric that R2SE uses for difficulty scoring, rewards, costs, and final evaluation.","marker":"[2]"},{"why":"Supplies the low-rank adapter (LoRA) formulation that R2SE uses to build the residual specialist ensembles on top of frozen generalist weights.","marker":"[31]"},{"why":"Provides the GRPO reinforcement learning objective that R2SE adapts to optimize residual corrections with process rewards and cost penalties.","marker":"[74]"},{"why":"Supplies the Generalized Pareto Distribution as the tail model for specialist uncertainty in the self-aware expansion policy.","marker":"[78]"},{"why":"Provides the Pickands-Balkema-de Haan theorem that justifies using GPD for threshold-excess tail modeling in the expansion mechanism.","marker":"[79]"},{"why":"Is the Hydra-MDP baseline that R2SE starts from and compares against on the NAVSIM leaderboard, yielding the 87.0 to 91.6 PDMS improvement.","marker":"[75]"},{"why":"Represents the existing RL-fine-tuning approach that R2SE contrasts with for hard-case refinement and forgetting behavior.","marker":"[22]"},{"why":"Provides the privilege-based PDM-Closed planner whose closed-loop score serves as the reference metric for PDMS in NAVSIM evaluation.","marker":"[70]"}],"fun_headline_variants":["Self-aware adapter expansion lifts driving score to 91.6","From 87.0 to 91.6: self-aware expansion for driving","Frozen generalist, RL adapters: driving score hits 91.6 on NAVSIM","Uncertainty-gated adaptive policy beats full fine-tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole refinement loop—selecting hard cases, computing rewards and costs, and evaluating success—uses non-reactive log simulation with the PDM Score as a proxy for real closed-loop driving quality, so if that proxy diverges from true on-road safety, the reported gains may not transfer to real vehicles.","fun_headline_variants_meta":{"raw":{"variants":["Self-aware adapter expansion lifts driving score to 91.6","From 87.0 to 91.6: self-aware expansion for driving","Frozen generalist, RL adapters: driving score hits 91.6 on NAVSIM","Uncertainty-gated adaptive policy beats full fine-tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000736,"raw_usage":{"total_tokens":3319,"prompt_tokens":1001,"completion_tokens":2318,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":617,"completion_tokens_details":{"reasoning_tokens":2237}},"tokens_in":617,"tokens_out":2318,"duration_ms":17272,"temperature":1.0,"reasoning_tokens":2237,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:40:41.629108+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same R2SE-refined policy in a reactive closed-loop simulator (or on a real vehicle) on the navtest scenarios and compare the score gap against the non-reactive PDMS gap: if the refined policy no longer outperforms the baseline, or if the improvement reverses, then the claim that non-reactive PDMS is a faithful training and evaluation signal for hard-case refinement is falsified.","supporting_citations":[{"cited_title":"Statistical inference using extreme order statistics,","cited_arxiv_id":null,"evidence_quote":"Provides the Pickands-Balkema-de Haan theorem that justifies using GPD for threshold-excess tail modeling in the expansion mechanism."},{"cited_title":"Parting with misconceptions about learning-based vehicle motion planning,","cited_arxiv_id":null,"evidence_quote":"Provides the privilege-based PDM-Closed planner whose closed-loop score serves as the reference metric for PDMS in NAVSIM evaluation."}],"review_version":1}