{"id":"de66fc30-ed31-4bf2-a875-f346399ed0fb","arxiv_id":"2505.05223","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A single vision-based reinforcement learning policy can steer an autonomous vehicle in simulation with different driving styles by feeding it a user preference vector, without retraining.","lead":"This paper trains a single self-driving agent in the CARLA simulator whose driving style, such as speed or comfort, can be adjusted at runtime by feeding it a preference vector. It is a step toward letting passengers personalize autonomous vehicles without retraining the system.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claim of dynamic preference adaptation is untested: every experiment fixes the preference vector for the whole episode, so intra-episode preference changes are never evaluated.","rationale":"The reader's CONDITIONAL verdict is appropriate, and our concern corresponds to the reader's first stated gap, but not to their chosen weakest_assumption about reward operationalization. The fixed-preference evidence is solid: Figure 4 shows statistically significant, monotonic shifts in velocity, acceleration, and jerk with preference weights, and the code is public. The reward-design worry is real for any human-meaning claim, but because the paper operationally defines a preference as a reward weight, the behavioral changes do demonstrate the intended mechanism under that definition. The missing intra-episode preference switch is more load-bearing: the introduction and abstract explicitly promise dynamic, changing preferences, and the training procedure never provides within-episode preference changes, so the dynamic component is an extrapolation. A targeted switch experiment would settle the matter. Absence of baselines weakens the novelty comparison but does not invalidate the capability claim. The recommended verdict therefore remains CONDITIONAL, and this stress-test does not move it.","tokens_in":12642,"tokens_out":6804,"duration_ms":75340,"concrete_test":"Use the released agent on the seven fixed evaluation scenarios and run episodes with a preference switch at the midpoint (e.g., comfort-dominant to aggressiveness-dominant and the reverse). Compare post-switch velocity, acceleration, jerk, route completion, and collision rate against two fixed-lambda baselines: the new lambda from the episode start and the old lambda throughout. If post-switch behavior tracks the fixed-new-lambda baseline within the reported variance, the dynamic-adaptation claim is supported; if it tracks the old lambda or shows a transition artifact, the claim must be restricted to inter-episode adaptation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires a single policy to adapt at runtime to changing preferences. In the evaluation, each episode samples one preference vector at the start and holds it fixed (Section III-E/F; Figure 4 states the x-axis is the 'fixed sampled weight'), and no experiment changes lambda mid-episode. Training likewise samples lambda once per episode; the HER mechanism relabels experiences with alternative constant vectors rather than exposing the policy to observed within-episode preference switches. Since lambda is part of the state, the policy could in principle react to a change, but there is no evidence that it does: the Q-values and angle loss are optimized for constant-lambda returns, and a mid-episode switch changes the future reward stream in a way never seen in training. The abstract and conclusion repeatedly promise 'changing preferences' and 'dynamically adapts,' so this is not a side remark. If the intended meaning is only adaptation between episodes, the wording overstates; if intra-episode adaptation is intended, the evidence is missing. The fixed-preference behavioral evidence in Figure 4 is real, but it does not establish the dynamic component that distinguishes the paper's 'runtime adaptation' claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a preference-driven multi-objective reinforcement learning (PD-MORL) agent for end-to-end, vision-based autonomous driving in CARLA. Preferences over aggressiveness, comfort, speed, and efficiency are encoded as a four-dimensional weight vector that conditions a single TD3-based policy, trained with an angle loss and hindsight experience replay. The evaluation has two parts: a 'preference reflection' analysis showing that varying a preference weight shifts mean velocity, acceleration, and jerk in the expected directions, and an ablation over visual encoders and input modalities reporting driving and preference metrics. The paper claims runtime adaptation to changing preferences without retraining.","tokens_in":1181,"tokens_out":1532,"duration_ms":97976,"significance":"The contribution is potentially useful: continuous preference conditioning in a high-dimensional, vision-based driving task with code release is a timely extension of PD-MORL, and the use of statistical tests for behavioral shifts is a positive feature. However, the evidence as presented supports preference-conditioned behavior under fixed per-episode preferences rather than true runtime adaptation during an episode, and the preference-related metrics are constructed from the same reward and Q-values that training optimizes. The significance of the personalization claim therefore depends on additional evidence.","major_comments":[{"comment":"All experiments fix the preference vector for the entire episode: training samples lambda once at episode start, and Figure 4's x-axis is the fixed sampled weight. No experiment changes lambda mid-episode, and the HER mechanism relabels experiences with alternative constant vectors rather than exposing the policy to within-episode switches. Since lambda is part of the state, the policy could in principle react to a change, but the abstract and conclusion promise 'dynamically adapts according to changing preferences,' and this central claim is not demonstrated. Please add intra-episode switching experiments (e.g., changing lambda at a known waypoint and comparing behavior before and after the switch, including safety and route-completion metrics) or revise the claims to inter-episode adaptation.","section":"III-E, IV-B, Fig. 4"},{"comment":"Preference Score is defined as the preference-weighted return of the hand-designed reward vector, and Preference Alignment is the cosine angle between the preference vector and critic Q-values, which the angle loss explicitly aligns during training. Both metrics therefore measure consistency with the reward design, not whether the behavior matches an independent notion of the corresponding driving style. The behavioral shifts in Figure 4 largely confirm that the reward shaping influences control, which is a weaker statement than personalized driving. Please either provide an external validation (user ratings, independent style metrics, or human-driver comparisons) or clearly restrict the claims to reward-defined style objectives.","section":"III-F, III-G"},{"comment":"The evaluation contains no comparison against a non-preference-conditioned baseline, a fixed-preference policy trained for a single weight, or an existing MORL/end-to-end driving method. The ablations show that encoder and modality choices matter, but they do not establish that the proposed single-policy approach is preferable to simpler alternatives or that it is competitive with prior work. At minimum, a fixed-preference TD3 and a single-objective baseline should be included with the same scenarios and metrics.","section":"IV-B, Table I"},{"comment":"The state and critic description is internally inconsistent: the reward vector is five-dimensional and the critic outputs Q in R^5, while the preference vector is in R^4 and the paper refers to element-wise multiplication lambda \\odot Q. It is not stated how the core reward is included in the scalarization or how a four-dimensional lambda multiplies a five-dimensional Q. Please specify the exact preference-conditioning mechanism, including the treatment of the core objective, or the implementation cannot be assessed.","section":"III-B, III-D"}],"minor_comments":[{"comment":"Figure 2 caption states a 'six-dimensional reward vector,' whereas Section III-F defines a five-dimensional vector; please reconcile.","section":"Fig. 2"},{"comment":"The column header 'Ours Feature Extractor Ablation Vision Modality Ablation' combines three panels into one line; the table should separate the two ablations with clear panel headers.","section":"Table I"},{"comment":"The efficiency reward formula uses vmax and amax without defining their values or units; the source of these constants should be given, for example in Table II.","section":"III-F"},{"comment":"Define the exact quantities plotted on the y-axes (mean per-episode velocity and acceleration magnitudes, and jerk magnitude), including the unit of jerk, and state whether the 480 episodes were collected in one scenario or across all seven evaluation scenarios.","section":"Fig. 4"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the gap between the abstract's 'changing preferences' and the fixed-preference evaluation; if the authors cannot produce intra-episode switching results, the manuscript should be revised to an inter-episode adaptation claim. The paper is otherwise a reasonable engineering contribution, but the absence of baselines and external validation makes acceptance at the current level premature. I would also encourage the editor to verify that the code release reproduces the reported numbers."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read this if you care about personalization in end-to-end driving. It is the first application of single-policy preference-driven MORL (PD-MORL) to vision-based urban driving in CARLA. The core behavioral claim is supported: changing preference weights moves measured velocity, acceleration, and jerk in the expected directions, with p < 0.001 on most objective-metric pairs. The code is public, the reward decomposition (core task vs. four style objectives) is clean, and the ablations over encoders and input modalities are systematic.\n\nThe paper also does several things right. The single-policy design conditioned on a continuous preference vector is a meaningful step beyond per-policy training or discrete style switching. The authors are honest about scope—no traffic lights, simulation only, modest Driving Score around 42%. The citation pattern is fine, and building on Basaklar et al.'s PD-MORL is appropriate.\n\nThe soft spots are real but addressable. The biggest one is that the central 'dynamic' adaptation claim is not tested. Every experiment fixes λ for the entire episode; no mid-episode preference switch is evaluated, and training also samples λ once per episode. Because λ is part of the state, the policy might react to a change, but there is no evidence. The abstract and conclusion say 'changing preferences' and 'dynamically adapts'—that overstates what was measured. Second, there are no baseline comparisons, so the advantage over retraining a scalarized policy is asserted, not demonstrated. Third, Preference Score and Preference Alignment are defined from the reward vector and critic Q-values, both of which training optimizes with an explicit angle loss. They partly measure the training objective, not user-perceived style. The behavioral metrics (velocity, acceleration, jerk) are independent of that circularity, which is why the central finding holds.\n\nThe stress-test note is correct on all three counts. This is a solid paper that needs revision, not a flawed one. A genuine intra-episode preference-switch experiment, one or two baselines, and a qualified claim ('between-episode adaptation') would resolve most concerns. I would bring it to a reading group and would cite it if I worked on MORL for driving. It deserves serious peer review, with major revision expected; the public code and the statistically solid behavioral results make it worth referee time.","headline":"Solid first application of preference-driven MORL to vision-based end-to-end driving, but the 'dynamic adaptation' claim is not actually tested—every episode fixes the preference vector.","tokens_in":13458,"tokens_out":3187,"would_cite":true,"duration_ms":30578,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single trained policy can switch a vision-based driving agent's style at runtime, no retraining required.","keywords":["multi-objective reinforcement learning","preference-driven adaptation","end-to-end autonomous driving","driving style personalization","runtime preference adaptation","vision-based driving","PD-MORL"],"falsifier":"A concrete check would be to run the trained agent on an unseen map of the same simulator with identical traffic seeds under comfort weight 0 versus 1; if mean jerk no longer drops significantly at the high comfort setting, while route completion remains unchanged, the preference adaptation fails to generalize beyond the training towns. A sharper version is a forced-choice video study: if human raters cannot distinguish high-aggressiveness from high-comfort runs at better than chance, the behavior changes are artifacts of the reward shaper rather than perceived driving style.","tokens_in":12324,"feed_emoji":"🚗","tokens_out":7714,"duration_ms":78951,"temperature":0.7,"pith_summary":"This paper argues that one reinforcement-learned driving policy can serve many users: a continuous preference vector—weights for comfort, speed, aggressiveness, and efficiency—is fed to the same trained network, and the car changes how it drives immediately. That matters because current end-to-end driving agents typically freeze one style at training time, require repeated human feedback, or must be re-optimized for each new preference. The authors train a vision-based agent in dense mixed-traffic urban simulation and show that raising the comfort weight lowers jerk and acceleration, raising the speed weight raises velocities, and raising the aggressiveness weight increases lateral acceleration and yaw rate, while collisions and route failures stay low. If the result is right, personalization becomes a runtime input rather than a retraining problem, and one model could cover a whole spectrum of driving styles.","feed_headline":"A single driving policy changes style on demand without retraining","feed_subtitle":"Preference weights for comfort, speed, aggressiveness, and efficiency steer a vision-based agent through urban traffic.","key_machinery":"The load-bearing mechanism is the preference-conditioned critic of the PD-MORL algorithm, with two ingredients that make one policy span all preferences. First, a preference interpolator $I(\\lambda) = \\lambda^p$ projects the user vector into a normalized latent space, and an angle loss $L_{\\text{angle}}(\\lambda_p, Q)$ penalizes directional mismatch between the projected preference and the critic's Q-vector, so the critic learns which behaviors the preference points to. Second, preference-aware hindsight experience replay resamples stored transitions with alternative preference vectors, so the policy sees every preference region during training. Preferences enter the update through element-wise multiplication $\\lambda \\odot Q_j$. Around this, the paper wraps a five-dimensional reward vector: the first dimension is a static core reward for collision avoidance, lane keeping, and route progress, and the other four are the style rewards, which lets preference modulation happen on top of a fixed safety and navigation base.","core_discovery":"The central discovery is that a preference-conditioned multi-objective reinforcement learning agent, trained once on a five-dimensional reward vector, learns a single policy that covers the entire continuous preference space rather than one operating point. At evaluation, the user's four preference weights are part of the state, the actor-critic acts accordingly, and the resulting behavior shifts in the intended directions: the reported differences are statistically significant, with speed weight raising mean velocity, comfort weight lowering mean jerk, and aggressiveness weight raising acceleration and yaw rate, while core driving metrics remain intact. The best configuration, which uses semantic-segmentation input and a truncated ResNet-18 encoder, attains a Driving Score of 42±17, route completion of 57±16 percent, and Preference Alignment of 11.3 degrees, indicating that the critic's Q-vector points close to the user's preference direction. In the paper's own terms, this is the first demonstration of preference-driven MORL applied to vision-based end-to-end autonomous driving.","pith_inferences":["The current evidence for personalization is measured with the same reward terms the training optimizes; a forced-choice human study comparing recorded runs would test whether passengers actually perceive the intended style differences.","Because the policy responds across a continuous weight interval, one could define a context scheduler that maps road type or traffic density to preference weights and switches automatically segment by segment; the paper does not evaluate that use.","A natural extension is to learn the preference vector itself from sparse user feedback, since the policy already parameterizes the full preference manifold and only the mapping from user to weights is missing.","The Preference Alignment angle, which measures how well the critic's Q-values point along the user preference, could serve as a general diagnostic in other multi-objective control tasks beyond driving."],"forward_implications":["A passenger or route planner can change the preference weights mid-journey and the same policy will change its driving style, with no retraining and no swap to a different model.","A single model trained once on the preference space can serve many users with differing tastes, avoiding the need to store or train one policy per style.","Because each style objective is tied to concrete measurable behavior, an operator can audit the personalization: higher comfort should visibly lower jerk, higher speed should raise velocity, and higher aggressiveness should raise yaw rate.","The reward decomposition into a static core plus dynamic preferences means the safety-critical parts of driving are not re-negotiated when the style changes, which is what the low collision and lane-invasion rates support."],"supporting_citations":[{"why":"This work supplies the preference-driven MORL algorithm, including the preference interpolator, angle loss, and hindsight experience replay that the method builds on.","marker":"[27]"},{"why":"It provides the urban driving simulator used for all training and evaluation scenarios.","marker":"[29]"},{"why":"It supplies the pretrained ResNet-18 weights used for the shared visual feature extractor.","marker":"[35]"},{"why":"It contributes components of the core reward, including collision, boundary, and navigation penalties.","marker":"[36]"},{"why":"It supplies end-to-end urban driving reward design that the core reward formulation follows.","marker":"[37]"},{"why":"It defines the Driving Score metric used to summarize route completion with infraction penalties.","marker":"[39]"},{"why":"It provides the truncated six-residual-block ResNet variant that yields the best feature extractor in the ablation.","marker":"[40]"}],"fun_headline_variants":["Single vision policy adapts driving style via preference weights","No retraining: preference weights steer autonomous driving style","One MORL policy covers all driving style preferences","Preference weights dial comfort, speed, and aggression in one policy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that the hand-written reward terms for aggressiveness, comfort, speed, and efficiency faithfully capture what humans mean by those driving styles; if users' comfort or sportiness is not reflected in weighted jerk, throttle variability, and yaw rate, the demonstrated behavioral changes do not amount to personalization.","fun_headline_variants_meta":{"raw":{"variants":["Single vision policy adapts driving style via preference weights","No retraining: preference weights steer autonomous driving style","One MORL policy covers all driving style preferences","Preference weights dial comfort, speed, and aggression in one policy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000735,"raw_usage":{"total_tokens":3267,"prompt_tokens":905,"completion_tokens":2362,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":2297}},"tokens_in":521,"tokens_out":2362,"duration_ms":19030,"temperature":1.0,"reasoning_tokens":2297,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:09:35.726026+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to run the trained agent on an unseen map of the same simulator with identical traffic seeds under comfort weight 0 versus 1; if mean jerk no longer drops significantly at the high comfort setting, while route completion remains unchanged, the preference adaptation fails to generalize beyond the training towns. A sharper version is a forced-choice video study: if human raters cannot distinguish high-aggressiveness from high-comfort runs at better than chance, the behavior changes are artifacts of the reward shaper rather than perceived driving style.","supporting_citations":[{"cited_title":"PD-MORL: Preference-driven multi-objective reinforcement learning algorithm,","cited_arxiv_id":null,"evidence_quote":"This work supplies the preference-driven MORL algorithm, including the preference interpolator, angle loss, and hindsight experience replay that the method builds on."},{"cited_title":"CARLA: An open urban driving simulator,","cited_arxiv_id":null,"evidence_quote":"It provides the urban driving simulator used for all training and evaluation scenarios."},{"cited_title":"Model-free deep reinforcement learning for urban autonomous driving,","cited_arxiv_id":null,"evidence_quote":"It contributes components of the core reward, including collision, boundary, and navigation penalties."},{"cited_title":"End-to-end model-free reinforcement learning for urban driving using implicit affordances,","cited_arxiv_id":null,"evidence_quote":"It supplies end-to-end urban driving reward design that the core reward formulation follows."},{"cited_title":"Carla leaderboard,","cited_arxiv_id":null,"evidence_quote":"It defines the Driving Score metric used to summarize route completion with infraction penalties."},{"cited_title":"Privileged sensing scaffolds reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"It provides the truncated six-residual-block ResNet variant that yields the best feature extractor in the ablation."}],"review_version":1}