{"id":"9146a57c-059c-48e1-a092-15020432709f","arxiv_id":"2412.02951","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A reinforcement learning framework that uses formal rulebook-based safety scores as rewards can fine-tune a perception model to reduce simulated driving rule violations.","lead":"This paper fine-tunes an object detection model using reinforcement learning, with rewards derived from formal safety rules, and reports lower safety-rule violation scores in CARLA simulation than a perception-reward-only baseline. The approach is plausible, but the evaluation lacks error bars, a comparison to the original pre-trained model, and a clear description of the optimization objective.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Remark 1 is the load-bearing assumption: if the controller with ground truth does not achieve zero violations, the rulebook reward is confounded and the central safety claim is not established.","rationale":"I agree with the reader that Remark 1 is the weakest assumption. The central claim is empirical, but the method's validity as described depends on this attribution: without it, the rulebook reward does not cleanly measure perception-induced safety failures. I also considered the apparent mismatch between Algorithm 1 and the reward-replacement description in Section IV-E; that is a serious reproducibility issue, but it is an implementation/presentation ambiguity that could be clarified with code or a corrected pseudocode. The Remark 1 issue is more fundamental because it concerns whether the reward measures what the paper claims to optimize. The proposed oracle-rollout test directly settles it: run the controller with ground-truth perception on the evaluation scenarios and check for violations. If the test fails, the central claim is not established and the verdict should move toward rejection unless the method or evaluation is revised. If it passes, the main attribution concern is resolved and the conditional acceptance stands. Therefore the reader's CONDITIONAL verdict remains appropriate.","tokens_in":11481,"tokens_out":10253,"duration_ms":110110,"concrete_test":"Run the evaluation scenarios with an oracle perception module: feed ground-truth 3D bounding boxes and speeds directly into the controller in the same CARLA maps, weathers, and random object spawns used in Section V-C, over the same 100-step episodes. Compute the total rule violation scores r1–r4. If any episode has nonzero violation, Remark 1 is violated on the evaluation distribution and the reward is confounded; the paper would need to restrict evaluation to scenarios where the oracle controller is violation-free or subtract the oracle baseline. If all episodes have zero violations, the attribution concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section III-A4's Remark 1 asserts that when the controller is given ground-truth environment e, it yields zero violations across all rules. This is what lets the authors attribute every nonzero rulebook violation to perception error and shape rrb accordingly. The support for Remark 1 is Lemmas 1–3 and Corollary 1 in Section IV-D, but these are stated without proof and depend on strong conditions: the set O of relevant objects must remain constant over a 0.1 s step, maximum response time is taken to be 0, ve ≤ vlim, and τ ≥ Δt. In the CARLA evaluation, vehicles and pedestrians can enter the ego lane within a step, and response time is not zero, so the lemmas do not establish Remark 1 on the evaluation distribution. If Remark 1 fails, rrb includes violations caused by the controller or environment rather than by perception, so the policy gradient rewards or penalizes the perception model for events outside its control. The reported lower violation scores for rb and mix over pc could then reflect a noisy, confounded reward rather than genuine system-level safety improvement. No empirical verification of Remark 1 is provided in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a reinforcement learning fine-tuning framework for perception models in autonomous driving. The perception model (PIX2SEQ) is treated as a stochastic policy, and its training reward combines a detection-quality reward with a rulebook-based safety score computed from CARLA simulation trajectories. The authors report experiments comparing perception-only (pc), rulebook-only (rb), and combined (mix) reward training, and claim that rb and mix improve system-level safety, especially under severe weather. The main claim is that perception can be directly optimized for downstream safety outcomes even when the controller is non-differentiable.","tokens_in":11685,"tokens_out":7613,"duration_ms":79642,"significance":"If the claim were fully established, the contribution would be valuable: it provides a practical way to inject formal system-level safety specifications into perception training without requiring a differentiable controller, and it uses the rulebook formalism to weight perceptual errors by their downstream safety impact. The evaluation on a different CARLA map and under two weather conditions is a genuine strength, as is the use of a realistic, non-differentiable simulator in the training loop. However, the current evidence does not fully support the central claim: the safety metric used for evaluation is the same rulebook score used as the training reward, the load-bearing assumption that a ground-truth controller yields zero violations is not verified, and the empirical results have no statistical grounding.","major_comments":[{"comment":"Remark 1 is load-bearing and is not established for the evaluation distribution. The lemmas and corollary that would justify it are stated without proofs, and their hypotheses include that the set O of relevant objects remains constant over a 0.1 s time step, that the maximum response time is zero, that v_e <= v_lim, and that tau >= Delta t. In the CARLA evaluation, vehicles and pedestrians can enter the ego lane within a step and the controller is not response-time-free, so the lemmas do not apply. Without an empirical check that the exact controller with ground-truth perception achieves zero rule violations on the evaluation scenarios, the rulebook reward rrb confounds perception errors with controller and environment limitations. Please run that ground-truth controller experiment and report per-rule violation scores, or revise the reward design to account for nonzero baseline violations.","section":"Section III-A4 and Section IV-D"},{"comment":"The evaluation metric is essentially the same rulebook violation score that is used as the training reward. Tables I and II report r1-r4 totals, which are exactly the quantities optimized by the rb and mix training objectives. The reported improvement is therefore partly a direct consequence of reward optimization rather than an independent measure of system-level safety. To support the abstract's claim, the paper should include independent safety metrics (for example, collision and near-miss counts, time-to-collision, or a held-out rulebook not used during training), and should also report the original pre-trained model at epoch 0 as an additional baseline.","section":"Section V-C and Section IV-E"},{"comment":"The gradient update as written is not the REINFORCE gradient for the policy that generated the trajectory. Algorithm 1 labels the summed log-probability term 'Loss' but performs theta = theta + alpha * grad(Loss), which is gradient ascent; if 'Loss' is meant to be a loss, the sign is inconsistent with gradient descent. More importantly, Section IV-E replaces the sampled token with the ground-truth token whenever the detection is wrong or the rulebook reward is nonzero, so the log-probability term is evaluated at actions that were not actually taken. This is not a standard policy-gradient estimator. Please state the exact objective being optimized and derive the update rule, or justify why the replacement rule is a valid estimator of that objective.","section":"Algorithm 1 and Section IV-E"},{"comment":"No number of evaluation episodes, random seeds, or error bars is reported for Tables I-III. Given that CARLA scenarios are randomly spawned and the violation scores vary widely across conditions (for example, r1 for pc increases from 42.786 under normal weather to 303.59 under severe weather), a single run cannot support the claimed improvement. Please report means and variances over multiple seeds and scenario counts, and ideally use paired same-scenario comparisons across the pc, rb, and mix models.","section":"Section V"}],"minor_comments":[{"comment":"The symbols N, T, and n are not defined in the pseudocode; please define them and clarify whether the displayed quantity is a loss or a return.","section":"Algorithm 1"},{"comment":"The column layout of Table III is ambiguous: it is not immediately clear which accuracy values correspond to rb, pc, and mix under each weather condition. Use explicit subheaders or separate columns.","section":"Table III"},{"comment":"The numerical values of the rulebook parameters (abrake, tau, epsilon, vlim, amax, amin, r, and the IoU threshold) are not reported, so the violation scores cannot be reproduced from the text alone.","section":"Section IV-C and Section IV-D"}],"recommendation":"major_revision","confidential_remarks":"I am recommending major revision rather than rejection because the central idea is defensible and the main weaknesses are addressable with additional experiments and clarifications. The most important check is the ground-truth controller experiment: if it shows nonzero rule violations in the evaluation scenarios, the current formulation of rrb cannot support the safety claim and the paper would need substantial reworking. The absence of code, checkpoints, and seed details also makes the empirical claims hard to verify; the authors should be asked to provide them if the journal allows."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the core idea—use rulebook safety scores as an RL reward to fine-tune a perception model with a black-box controller in the loop—is genuinely new relative to what it cites, and the CARLA experiments are a reasonable first demonstration. But the central safety claim depends on Remark 1, which is not verified, and the empirical section is thinner than it looks. The paper deserves a referee, but it needs major revision before the headline claim is credible.\n\nWhat's new: the combination of formal rulebook specifications (Censi et al.) with the task-reward RL fine-tuning setup of Pinto et al. Applying policy gradients to PIX2SEQ-style detection with non-differentiable simulator/controller is a natural but nontrivial extension. The rulebook reward design, with four concrete rules for longitudinal control, is well specified. The evaluation on a different CARLA map and weather conditions is a good instinct, and the prioritized-object accuracy results are suggestive.\n\nThe soft spots are real. Remark 1 is load-bearing: the claim that with ground-truth environment the controller achieves zero violations across all rules is what allows every nonzero violation to be blamed on perception. The supporting Lemmas 1–3 are stated without proof, and their hypotheses (fixed set of relevant objects over a 0.1s step, zero response time, τ ≥ Δt, ve ≤ vlim) are not checked on the evaluation distribution. If the assumption fails, the rulebook reward penalizes the perception model for controller or environment behavior, and the reported improvement could be driven by a confounded signal. The paper needs either a proof that the controller satisfies the rules under realistic conditions, or an empirical check (e.g., run the controller with ground-truth detection in the same scenarios).\n\nTwo smaller issues. The empirical comparison lacks error bars or run counts; with five trajectories per epoch and a single seed, the differences in Tables I–II could be noise. The baseline is a perception-reward fine-tuned model, not the original pretrained model, so part of the safety gain could just be fine-tuning on the target domain. And Algorithm 1 is inconsistent with the text: it performs gradient ascent on a quantity named 'Loss', while the surrounding text talks about loss minimization. That needs to be cleaned up, though it is probably a notation issue.\n\nWho this is for: people working on safe autonomy, formal specifications for ML, and task-reward fine-tuning. It is a credible idea paper, not a complete empirical study. If I were editing, I would send it to review—there is enough novelty and a real evaluation target—but I would expect the reviewers to push hard on Remark 1 and the statistics.","headline":"Genuinely new rulebook-reward RL fine-tuning for perception, but the safety claim rests on an unverified controller assumption and thin empirical evidence.","tokens_in":12240,"tokens_out":2260,"would_cite":false,"duration_ms":22227,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training perception models with a safety-score reward improves system-level safety over detection-only training.","keywords":["reinforcement learning","perception model fine-tuning","system-level safety","rulebook formalism","autonomous driving","object detection","policy gradient","CARLA simulator"],"falsifier":"Run the same controller with ground-truth perception substituted for the detector's outputs across the evaluation scenarios and record the rulebook scores; any nonzero violation under perfect perception would show the safety reward is at least partly controller-induced, invalidating the clean attribution of violations to perception.","tokens_in":11259,"feed_emoji":"🚗","tokens_out":7232,"duration_ms":71681,"temperature":0.7,"pith_summary":"Perception models for autonomous driving are usually trained to maximize detection accuracy, with no feedback about whether a detection error causes a crash or a rule violation. This paper proposes closing that gap by translating formal safety requirements, written as rulebook rules, into scalar violation scores and folding those scores into the reward used to fine-tune an object detector with reinforcement learning. The authors claim that in CARLA simulations, models fine-tuned with rulebook rewards, either alone or mixed with a detection reward, produce fewer and less severe rule violations than models trained on detection reward alone, with the largest gains in fog where baseline detectors degrade sharply. If this holds, perception can be made safety-aware even when the downstream controller and simulator are non-differentiable black boxes.","feed_headline":"Rulebook rewards for perception cut driving violations by 73% in fog","feed_subtitle":"Reinforcement learning fine-tunes object detectors on formal safety scores, beating detection-only training in CARLA.","key_machinery":"The mechanism is the rulebook-to-reward channel. A rulebook rule is a function from a realization, a sequence of world states plus the ground-truth environment, to a non-negative violation severity, and the sum of these severities over rules and time steps becomes the rulebook reward. In the reinforcement-learning formulation, the perception model is the stochastic policy, its detection output is the action, and the simulator plus controller act as a black-box environment that returns this safety-derived reward. Because policy gradients only require $\\nabla_\\theta \\log \\pi_\\theta(a|s)$ times the return, the whole loop works without differentiating through the controller or simulator.","core_discovery":"On the paper's own terms, the central claim is that a perception model trained with a reward combining detection quality with rulebook-derived safety scores becomes a better component of the overall driving system than one trained on perception quality alone. In the reported experiments, rulebook-only fine-tuning lowers the total violation score from 105.61 to 76.46 under normal weather and from 479.31 to 129.37 under heavy fog, while also raising detection accuracy on safety-prioritized objects. The authors interpret this as evidence that the perception model can be aligned with system-level safety objectives directly, without requiring the controller to be differentiable, because policy-gradient training only needs the model's action probabilities and the scalar reward returned by the simulator and rulebook.","pith_inferences":["Varying the mixing weight $\\beta$ between perception and rulebook rewards would map the trade-off between detection quality and safety; the reported experiments leave that curve unexplored.","Because the rules only score objects in the same lane and direction, the trained detector may become biased toward those objects; cut-in and cross-traffic scenarios would test whether the safety benefit generalizes.","Running the same controller on ground-truth detections would quantify how many violations are caused by the controller itself, separating perception-induced from controller-induced safety failures."],"forward_implications":["Perception can be fine-tuned for system-level safety even when the planner and simulator are non-differentiable, because policy-gradient training only needs action probabilities and observed rewards.","Rulebook rewards shift the detector's attention onto safety-critical objects; in the reported results, accuracy on prioritized objects rises from 0.54 to 0.84 under normal weather with rulebook-only training.","Rulebook-only training can beat perception-only training on total violation scores, showing that raw detection accuracy is not a reliable proxy for downstream safety.","Models trained only in clear weather keep low violation scores when evaluated under fog density 40, suggesting that the safety objective transfers across conditions."],"supporting_citations":[{"why":"Supplies the rulebook formalism used to specify system-level safety rules and to compute violation scores from realizations.","marker":"[25]"},{"why":"Provides the task-reward tuning method whose policy-gradient view and IoU-based reward design the paper adapts to inject rulebook rewards.","marker":"[29]"},{"why":"PIX2SEQ is the probabilistic object detection model that is treated as a stochastic policy and fine-tuned in the experiments.","marker":"[26]"},{"why":"CARLA is the high-fidelity simulator used to generate trajectory rollouts and evaluate the fine-tuned perception models.","marker":"[31]"},{"why":"The RSS condition is used to define the obstacle-clearance and unnecessary-brake rules.","marker":"[32]"},{"why":"COCO supplies the pre-training data for the detection model before reinforcement-learning fine-tuning.","marker":"[33]"}],"fun_headline_variants":["RL fine-tuning with rulebook rewards slashes fog-time driving errors","Safety-aware perception: rulebook rewards cut violations in fog","Formal safety rules guide RL to better perception for driving","Perception learns safety: rulebook rewards beat detection loss","Rulebook rewards reduce driving violations under fog"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a controller with perfect perception would never violate any safety rule, so every violation seen during training can be blamed on perception; in the paper this premise is assumed outright and supported only by lemmas with idealized conditions such as an unchanging obstacle set and zero response time.","fun_headline_variants_meta":{"raw":{"variants":["RL fine-tuning with rulebook rewards slashes fog-time driving errors","Safety-aware perception: rulebook rewards cut violations in fog","Formal safety rules guide RL to better perception for driving","Perception learns safety: rulebook rewards beat detection loss","Rulebook rewards reduce driving violations under fog"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000642,"raw_usage":{"total_tokens":2902,"prompt_tokens":841,"completion_tokens":2061,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":457,"completion_tokens_details":{"reasoning_tokens":1981}},"tokens_in":457,"tokens_out":2061,"duration_ms":14931,"temperature":1.0,"reasoning_tokens":1981,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:55:49.180871+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same controller with ground-truth perception substituted for the detector's outputs across the evaluation scenarios and record the rulebook scores; any nonzero violation under perfect perception would show the safety reward is at least partly controller-induced, invalidating the clean attribution of violations to perception.","supporting_citations":[{"cited_title":"Liability, ethics, and culture-aware behavior specification using rulebooks,","cited_arxiv_id":null,"evidence_quote":"Supplies the rulebook formalism used to specify system-level safety rules and to compute violation scores from realizations."},{"cited_title":"Tuning computer vision models with task rewards,","cited_arxiv_id":null,"evidence_quote":"Provides the task-reward tuning method whose policy-gradient view and IoU-based reward design the paper adapts to inject rulebook rewards."},{"cited_title":"Responsibility-Sensitive Safety: an Introduction with an Eye to Logical Foundations and Formalization","cited_arxiv_id":"2206.03418","evidence_quote":"The RSS condition is used to define the obstacle-clearance and unnecessary-brake rules."},{"cited_title":"Microsoft coco: Common objects in context,","cited_arxiv_id":null,"evidence_quote":"COCO supplies the pre-training data for the detection model before reinforcement-learning fine-tuning."}],"review_version":1}