REVIEW 4 major objections 3 minor 33 references
Incorporating System-level Safety Requirements in Perception Models via Reinforcement Learning
T0 review · 4 major / 3 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Training perception models with a safety-score reward improves system-level safety over detection-only training.
desk verdict Genuinely new rulebook-reward RL fine-tuning for perception, but the safety claim rests on an unverified controller assumption and thin empirical evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The mechanism is the rulebook-to-reward channel. A rulebook rule is a function from a realization, a sequence of world states plus the ground-truth environment, to a non-negative violation severity, and the sum of these severities over rules and time steps becomes the rulebook reward. In the reinforcement-learning formulation, the perception model is the stochastic policy, its detection output is the action, and the simulator plus controller act as a black-box environment that returns this safety-derived reward. Because policy gradients only require $\nabla_\theta \log \pi_\theta(a|s)$ times the return, the whole loop works without differentiating through the controller or simulator.
What would settle it
Run the same controller with ground-truth perception substituted for the detector's outputs across the evaluation scenarios and record the rulebook scores; any nonzero violation under perfect perception would show the safety reward is at least partly controller-induced, invalidating the clean attribution of violations to perception.
Extended reading notes
Core claim
On the paper's own terms, the central claim is that a perception model trained with a reward combining detection quality with rulebook-derived safety scores becomes a better component of the overall driving system than one trained on perception quality alone. In the reported experiments, rulebook-only fine-tuning lowers the total violation score from 105.61 to 76.46 under normal weather and from 479.31 to 129.37 under heavy fog, while also raising detection accuracy on safety-prioritized objects. The authors interpret this as evidence that the perception model can be aligned with system-level safety objectives directly, without requiring the controller to be differentiable, because policy-gradient training only needs the model's action probabilities and the scalar reward returned by the simulator and rulebook.
Load-bearing premise
The load-bearing premise is that a controller with perfect perception would never violate any safety rule, so every violation seen during training can be blamed on perception; in the paper this premise is assumed outright and supported only by lemmas with idealized conditions such as an unchanging obstacle set and zero response time.
Editorial extensions
If this is right
- Perception can be fine-tuned for system-level safety even when the planner and simulator are non-differentiable, because policy-gradient training only needs action probabilities and observed rewards.
- Rulebook rewards shift the detector's attention onto safety-critical objects; in the reported results, accuracy on prioritized objects rises from 0.54 to 0.84 under normal weather with rulebook-only training.
- Rulebook-only training can beat perception-only training on total violation scores, showing that raw detection accuracy is not a reliable proxy for downstream safety.
- Models trained only in clear weather keep low violation scores when evaluated under fog density 40, suggesting that the safety objective transfers across conditions.
Reading between the lines
- Varying the mixing weight $\beta$ between perception and rulebook rewards would map the trade-off between detection quality and safety; the reported experiments leave that curve unexplored.
- Because the rules only score objects in the same lane and direction, the trained detector may become biased toward those objects; cut-in and cross-traffic scenarios would test whether the safety benefit generalizes.
- Running the same controller on ground-truth detections would quantify how many violations are caused by the controller itself, separating perception-induced from controller-induced safety failures.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a reinforcement learning fine-tuning framework for perception models in autonomous driving. The perception model (PIX2SEQ) is treated as a stochastic policy, and its training reward combines a detection-quality reward with a rulebook-based safety score computed from CARLA simulation trajectories. The authors report experiments comparing perception-only (pc), rulebook-only (rb), and combined (mix) reward training, and claim that rb and mix improve system-level safety, especially under severe weather. The main claim is that perception can be directly optimized for downstream safety outcomes even when the controller is non-differentiable.
Significance. If the claim were fully established, the contribution would be valuable: it provides a practical way to inject formal system-level safety specifications into perception training without requiring a differentiable controller, and it uses the rulebook formalism to weight perceptual errors by their downstream safety impact. The evaluation on a different CARLA map and under two weather conditions is a genuine strength, as is the use of a realistic, non-differentiable simulator in the training loop. However, the current evidence does not fully support the central claim: the safety metric used for evaluation is the same rulebook score used as the training reward, the load-bearing assumption that a ground-truth controller yields zero violations is not verified, and the empirical results have no statistical grounding.
major comments (4)
- [Section III-A4 and Section IV-D] Remark 1 is load-bearing and is not established for the evaluation distribution. The lemmas and corollary that would justify it are stated without proofs, and their hypotheses include that the set O of relevant objects remains constant over a 0.1 s time step, that the maximum response time is zero, that v_e <= v_lim, and that tau >= Delta t. In the CARLA evaluation, vehicles and pedestrians can enter the ego lane within a step and the controller is not response-time-free, so the lemmas do not apply. Without an empirical check that the exact controller with ground-truth perception achieves zero rule violations on the evaluation scenarios, the rulebook reward rrb confounds perception errors with controller and environment limitations. Please run that ground-truth controller experiment and report per-rule violation scores, or revise the reward design to account for nonzero baseline violations.
- [Section V-C and Section IV-E] The evaluation metric is essentially the same rulebook violation score that is used as the training reward. Tables I and II report r1-r4 totals, which are exactly the quantities optimized by the rb and mix training objectives. The reported improvement is therefore partly a direct consequence of reward optimization rather than an independent measure of system-level safety. To support the abstract's claim, the paper should include independent safety metrics (for example, collision and near-miss counts, time-to-collision, or a held-out rulebook not used during training), and should also report the original pre-trained model at epoch 0 as an additional baseline.
- [Algorithm 1 and Section IV-E] The gradient update as written is not the REINFORCE gradient for the policy that generated the trajectory. Algorithm 1 labels the summed log-probability term 'Loss' but performs theta = theta + alpha * grad(Loss), which is gradient ascent; if 'Loss' is meant to be a loss, the sign is inconsistent with gradient descent. More importantly, Section IV-E replaces the sampled token with the ground-truth token whenever the detection is wrong or the rulebook reward is nonzero, so the log-probability term is evaluated at actions that were not actually taken. This is not a standard policy-gradient estimator. Please state the exact objective being optimized and derive the update rule, or justify why the replacement rule is a valid estimator of that objective.
- [Section V] No number of evaluation episodes, random seeds, or error bars is reported for Tables I-III. Given that CARLA scenarios are randomly spawned and the violation scores vary widely across conditions (for example, r1 for pc increases from 42.786 under normal weather to 303.59 under severe weather), a single run cannot support the claimed improvement. Please report means and variances over multiple seeds and scenario counts, and ideally use paired same-scenario comparisons across the pc, rb, and mix models.
minor comments (3)
- [Algorithm 1] The symbols N, T, and n are not defined in the pseudocode; please define them and clarify whether the displayed quantity is a loss or a return.
- [Table III] The column layout of Table III is ambiguous: it is not immediately clear which accuracy values correspond to rb, pc, and mix under each weather condition. Use explicit subheaders or separate columns.
- [Section IV-C and Section IV-D] The numerical values of the rulebook parameters (abrake, tau, epsilon, vlim, amax, amin, r, and the IoU threshold) are not reported, so the violation scores cannot be reproduced from the text alone.
Circularity Check
The system-level safety metric is the same rulebook violation score used as the training reward, so the main result is partly a restatement of the optimization objective; held-out map/weather evaluation and the prioritized-object accuracy give independent evidence.
-
self definitional
[Section IV-E (Reward Design) and Section V-C (Evaluation, Tables I-II)]
"We have two parts of the reward: the perception reward rpc and rulebook rrb reward. So, the total reward for each token would be r = βrpc + (1 − β)rrb ... the rulebook reward is calculated from the violation scores. Given a bounding box and its associated realization x, if it is a wrong detection result and triggers the violation with rule ri, then rrbi = rbi(x); otherwise rrbi = 0 ... The total rulebook reward is rrb = PK i=1 rrbi (x). ... The tables compare models trained with perception loss only (pc), rulebook loss only (rb), and a combination of both (mix)."
The outcome reported as system-level safety is the sum of r1-r4 violation scores in Tables I-II, and the rulebook training reward rrb is defined as the sum of exactly those same rule violation scores rbi(x). Models labeled rb and mix are updated by policy gradients (Algorithm 1) to maximize rrb, equivalently to minimize the reported total violation score. Hence the lower violation totals for rb/mix relative to pc are, up to generalization across the new map and weather, a direct consequence of optimizing the evaluation quantity rather than an independent measurement. The different-map/weather evaluation and Table III's prioritized-object accuracy are separate evidence, so the circularity is partial.
full rationale
The paper's derivation is mostly a standard RL fine-tuning pipeline: a probabilistic detection model is treated as a policy, and the reward combines a perception IoU term with rulebook violation scores. The rulebook formalism [25] is cited from prior work including a coauthor, but it is an external published formalism and is not invoked as a uniqueness theorem or as the sole justification for the approach, so that self-citation is not load-bearing. The load-bearing assumption is Remark 1 (Section III-A4), supported by Lemmas 1-3 and Corollary 1 (Section IV-D); these are stated without proof and require conditions (constant relevant-object set, zero response time, ve ≤ vlim, τ ≥ Δt) that may not hold in the CARLA evaluation. This is a support gap and a validity threat, but it is not an equation-level circularity: the reward-vs-evaluation overlap is the only step that reduces by construction. Because the evaluation uses a different map, random object types, and unseen weather, and because Table III reports a separate prioritized-object accuracy measure, the central claim retains independent empirical content. Accordingly the circularity score is moderate, not extreme.
Assumptions & free parameters
free parameters (4)
- beta (reward balance)
- w_percp (perception reward weight)
- IoU threshold for perception reward
- Rulebook constants (abrake, tau, epsilon, r, amax, amin, v_lim)
assumptions (5)
- standard math The MDP formulation for perception-as-policy is valid.
- domain assumption The rulebook formalism from [25] accurately captures system-level safety.
- domain assumption Ground-truth perception yields zero rule violations (Remark 1).
- domain assumption Lemma assumptions: the set of relevant objects O stays constant over a time step, zero response time, etc.
- standard math Policy gradient estimator with per-token rewards is unbiased.
Cite this review
Pith. "Pith review of Incorporating System-level Safety Requirements in Perception Models via Reinforcement Learning." pith.science (2026). https://pith.science/paper/3LJRDBQP
@misc{pith2026241202951,
author = {Pith},
title = {Pith review of: Incorporating System-level Safety Requirements in Perception Models via Reinforcement Learning},
year = {2026},
howpublished = {\url{https://pith.science/paper/3LJRDBQP}},
note = {Machine review of arXiv:2412.02951}
}
read the original abstract
Perception components in autonomous systems are often developed and optimized independently of downstream decision-making and control components, relying on established performance metrics like accuracy, precision, and recall. Traditional loss functions, such as cross-entropy loss and negative log-likelihood, focus on reducing misclassification errors but fail to consider their impact on system-level safety, overlooking the varying severities of system-level failures caused by these errors. To address this limitation, we propose a novel training paradigm that augments the perception component with an understanding of system-level safety objectives. Central to our approach is the translation of system-level safety requirements, formally specified using the rulebook formalism, into safety scores. These scores are then incorporated into the reward function of a reinforcement learning framework for fine-tuning perception models with system-level safety objectives. Simulation results demonstrate that models trained with this approach outperform baseline perception models in terms of system-level safety.
Figures
Reference graph
Works this paper leans on
-
[1]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2016, pp. 770–778
2016
-
[2]
Yolov3: An incremental improvement,
J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” arXiv preprint arXiv:1804.02767 , 2018
arXiv 2018
-
[3]
Diffstack: A differentiable and modular control stack for autonomous vehicles,
P. Karkus, B. Ivanovic, S. Mannor, and M. Pavone, “Diffstack: A differentiable and modular control stack for autonomous vehicles,” in Proceedings of The 6th Conference on Robot Learning , ser. Proceedings of Machine Learning Research, K. Liu, D. Kulic, and J. Ichnowski, Eds., vol. 205. PMLR, 14–18 Dec 2023, pp. 2170–2180. [Online]. Available: https://proc...
work page 2023
-
[4]
Where’s Waldo? Sensor- based temporal logic motion planning,
H. Kress-Gazit, G. Fainekos, and G. Pappas, “Where’s Waldo? Sensor- based temporal logic motion planning,” in Proc. of IEEE International Conference on Robotics and Automation , April 2007, pp. 3116–3121
work page 2007
-
[5]
Valet parking without a valet,
D. Conner, H. Kress-Gazit, H. Choset, A. Rizzi, and G. Pappas, “Valet parking without a valet,” in Proc. of IEEE/RSJ International Conference on Intelligent Robots and Systems , 2007, pp. 572–577
work page 2007
-
[6]
A fully automated framework for control of linear systems from temporal logic specifications,
M. Kloetzer and C. Belta, “A fully automated framework for control of linear systems from temporal logic specifications,” IEEE Transactions on Automatic Control , vol. 53, no. 1, pp. 287–297, 2008
work page 2008
-
[7]
Receding horizon control for temporal logic specifications,
T. Wongpiromsarn, U. Topcu, and R. M. Murray, “Receding horizon control for temporal logic specifications,” in Proceedings of the 13th ACM international conference on Hybrid systems: computation and control, 2010, pp. 101–110
work page 2010
-
[8]
Linear time logic control of linear systems,
P. Tabuada and G. J. Pappas, “Linear time logic control of linear systems,” IEEE Transactions on Automatic Control , vol. 51, no. 12, pp. 1862–1877, 2006
work page 2006
Show all 33 references
-
[9]
Hierarchical control system design using approximate simulation,
A. Girard and G. J. Pappas, “Hierarchical control system design using approximate simulation,” Automatica, vol. 45, no. 2, pp. 566–571, 2009
2009
-
[10]
Sampling-based motion planning with deterministic µ-calculus specifications,
S. Karaman and E. Frazzoli, “Sampling-based motion planning with deterministic µ-calculus specifications,” in Proc. of the IEEE Confer- ence on Decision and Control (CDC) , 2009
2009
-
[11]
Incremental sampling-based algorithm for minimum- violation motion planning,
L. I. R. Castro, P. Chaudhari, J. Tumova, S. Karaman, E. Frazzoli, and D. Rus, “Incremental sampling-based algorithm for minimum- violation motion planning,” in 52nd IEEE Conference on Decision and Control, Dec 2013, pp. 3217–3224
2013
-
[12]
Minimum- violation planning for autonomous systems: Theoretical and practical considerations,
T. Wongpiromsarn, K. Slutsky, E. Frazzoli, and U. Topcu, “Minimum- violation planning for autonomous systems: Theoretical and practical considerations,” in 2021 American Control Conference, 2021, submit- ted
2021
-
[13]
Verifai: A toolkit for the formal design and analysis of artificial intelligence-based systems,
T. Dreossi, D. J. Fremont, S. Ghosh, E. Kim, H. Ravanbakhsh, M. Vazquez-Chanlatte, and S. A. Seshia, “Verifai: A toolkit for the formal design and analysis of artificial intelligence-based systems,” in International Conference on Computer Aided Verification . Springer, 2019, p...
2019
-
[14]
Formal analysis and redesign of a neural network-based aircraft taxiing system with verifai,
D. J. Fremont, J. Chiu, D. D. Margineantu, D. Osipychev, and S. A. Seshia, “Formal analysis and redesign of a neural network-based aircraft taxiing system with verifai,” in International Conference on Computer Aided Verification. Springer, 2020, pp. 122–134
2020
-
[15]
nureality: A vr environment for research of pedestrian and autonomous vehicle interactions,
P. Schmitt, N. Britten, J. Jeong, A. Coffey, K. Clark, S. S. Kothawade, E. C. Grigore, A. Khaw, C. Konopka, L. Pham et al. , “nureality: A vr environment for research of pedestrian and autonomous vehicle interactions,” arXiv preprint arXiv:2201.04742 , 2022
2022 arXiv
-
[16]
Verisig: verifying safety properties of hybrid systems with neural network controllers,
R. Ivanov, J. Weimer, R. Alur, G. J. Pappas, and I. Lee, “Verisig: verifying safety properties of hybrid systems with neural network controllers,” in Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control , 2019, pp. 169–178
2019
-
[17]
Semantic adversarial deep learning,
T. Dreossi, S. Jha, and S. A. Seshia, “Semantic adversarial deep learning,” in International Conference on Computer Aided Verification. Springer, 2018, pp. 3–26
2018
-
[18]
Leveraging clas- sification metrics for quantitative system-level analysis with temporal logic specifications,
A. Badithela, T. Wongpiromsarn, and R. M. Murray, “Leveraging clas- sification metrics for quantitative system-level analysis with temporal logic specifications,” in 2021 60th IEEE Conference on Decision and Control (CDC). IEEE, 2021, pp. 564–571
2021
-
[19]
R. S. Sutton and A. G. Barto, Reinforcement learning: An introduction. MIT press, 2018
2018
-
[20]
Playing atari with deep reinforcement learning,
V . Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” arXiv preprint arXiv:1312.5602 , 2013
2013 arXiv
-
[21]
Human-level control through deep reinforcement learning,
V . Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski et al. , “Human-level control through deep reinforcement learning,” nature, vol. 518, no. 7540, pp. 529–533, 2015
2015
-
[22]
Simple statistical gradient-following algorithms for connectionist reinforcement learning,
R. J. Williams, “Simple statistical gradient-following algorithms for connectionist reinforcement learning,” Machine learning , vol. 8, pp. 229–256, 1992
1992
-
[23]
Policy gradi- ent methods for reinforcement learning with function approximation,
R. S. Sutton, D. McAllester, S. Singh, and Y . Mansour, “Policy gradi- ent methods for reinforcement learning with function approximation,” Advances in neural information processing systems , vol. 12, 1999
1999
-
[24]
Policy gradient methods,
OpenAI, “Policy gradient methods,” Spinning Up in Deep RL, 2018, accessed: 2023-11-04. [Online]. Available: https://spinningup.openai. com/en/latest/spinningup/rl intro3.html
2018
-
[25]
Liability, ethics, and culture-aware behavior specification using rulebooks,
A. Censi, K. Slutsky, T. Wongpiromsarn, D. Yershov, S. Pendleton, J. Fu, and E. Frazzoli, “Liability, ethics, and culture-aware behavior specification using rulebooks,” in 2019 International Conference on Robotics and Automation (ICRA) , 2019, pp. 8536–8542
2019
-
[26]
Pix2seq: A language modeling framework for object detection,
T. Chen, S. Saxena, L. Li, D. J. Fleet, and G. Hinton, “Pix2seq: A language modeling framework for object detection,” arXiv preprint arXiv:2109.10852, 2021
2021 arXiv
-
[27]
Oneformer: One transformer to rule universal image segmentation,
J. Jain, J. Li, M. T. Chiu, A. Hassani, N. Orlov, and H. Shi, “Oneformer: One transformer to rule universal image segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 2989–2998
2023
-
[28]
An image is worth 16x16 words: Transformers for image recognition at scale,
A. Dosovitskiy, “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020
2010 arXiv
-
[29]
Tuning computer vision models with task rewards,
A. S. Pinto, A. Kolesnikov, Y . Shi, L. Beyer, and X. Zhai, “Tuning computer vision models with task rewards,” in International Confer- ence on Machine Learning . PMLR, 2023, pp. 33 229–33 239
2023
-
[30]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Doll ´ar, “Microsoft coco: Common objects in context,” 2015
2015
-
[31]
CARLA: An open urban driving simulator,
A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V . Koltun, “CARLA: An open urban driving simulator,” in Proceedings of the 1st Annual Conference on Robot Learning , 2017, pp. 1–16
2017
-
[32]
Responsibility-sensitive safety: an introduction with an eye to logical foundations and formalization,
I. Hasuo, “Responsibility-sensitive safety: an introduction with an eye to logical foundations and formalization,” arXiv preprint arXiv:2206.03418, 2022
2022 arXiv
-
[33]
Microsoft coco: Common objects in context,
T.-Y . Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Doll ´ar, and C. L. Zitnick, “Microsoft coco: Common objects in context,” in European conference on computer vision. Springer, 2014, pp. 740–755
2014
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.