RECORD
The Reasonable Crowd: Towards evidence-based and interpretable models of driving behavior
Machine Learning · Submitted 28 Jul 2021
abstract compiled · 2026-09-23 · no language-model review
The authors' main claim Second, we found that the rulebook provides high interpretability without excessively sacrificing performance.
The compiled record
The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.
Open the compiled record
The abstract, sentence by sentence
Autonomous vehicles must balance a complex set of objectives. There is no consensus on how they should do so, nor on a model for specifying a desired driving behavior. We created a dataset to help address some of these questions in a limited operating domain. The data consists of 92 traffic scenarios, with multiple ways of traversing each scenario. Multiple annotators expressed their preference between pairs of scenario traversals. We used the data to compare an instance of a rulebook, carefully hand-crafted independently of the dataset, with several interpretable machine learning models such as Bayesian networks, decision trees, and logistic regression trained on the dataset. To compare driving behavior, these models use scores indicating by how much different scenario traversals violate each of 14 driving rules. The rules are interpretable and designed by subject-matter experts. First, we found that these rules were enough for these models to achieve a high classification accuracy on the dataset. Second, we found that the rulebook provides high interpretability without excessively sacrificing performance. Third, the data pointed to possible improvements in the rulebook and the rules, and to potential new rules. Fourth, we explored the interpretability vs performance trade-off by also training non-interpretable models such as a random forest. Finally, we make the dataset publicly available to encourage a discussion from the wider community on behavior specification for AVs. Please find it at github.com/bassam-motional/Reasonable-Crowd.
background contribution method result interpretation main claim
| # | sentence | role | the authors' own work | states the main conclusion | strength |
|---|---|---|---|---|---|
| s0 | Autonomous vehicles must balance a complex set of objectives. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| s1 | There is no consensus on how they should do so, nor on a model for specifying a desired driving behavior. | gap 0.93 | 0.06 | 0.02 | asserted 1.00 |
| s2 | We created a dataset to help address some of these questions in a limited operating domain. | contribution 0.85 | 0.96 | 0.02 | asserted 0.98 |
| s3 | The data consists of 92 traffic scenarios, with multiple ways of traversing each scenario. | method 0.86 | 0.89 | 0.02 | asserted 1.00 |
| s4 | Multiple annotators expressed their preference between pairs of scenario traversals. | method 0.96 | 0.82 | 0.02 | asserted 1.00 |
| s5 | We used the data to compare an instance of a rulebook, carefully hand-crafted independently of the dataset, with several interpretable machine learning models such as Bayesian networks, decision trees, and logistic regression trained on the dataset. | method 0.93 | 0.96 | 0.04 | asserted 0.98 |
| s6 | To compare driving behavior, these models use scores indicating by how much different scenario traversals violate each of 14 driving rules. | method 1.00 | 0.67 | 0.03 | asserted 1.00 |
| s7 | The rules are interpretable and designed by subject-matter experts. | method 0.90 | 0.32 | 0.04 | asserted 1.00 |
| s8 | First, we found that these rules were enough for these models to achieve a high classification accuracy on the dataset. | result 1.00 | 0.97 | 0.24 | asserted 1.00 |
| s9 | Second, we found that the rulebook provides high interpretability without excessively sacrificing performance. | result 0.98 | 0.94 | 0.36 | asserted 1.00 |
| s10 | Third, the data pointed to possible improvements in the rulebook and the rules, and to potential new rules. | result 0.92 | 0.92 | 0.11 | hedged possibility 0.91 |
| s11 | Fourth, we explored the interpretability vs performance trade-off by also training non-interpretable models such as a random forest. | method 0.56 | 0.96 | 0.05 | asserted 0.96 |
| s12 | Finally, we make the dataset publicly available to encourage a discussion from the wider community on behavior specification for AVs. | contribution 0.78 | 0.90 | 0.04 | asserted 0.90 |
| s13 | Please find it at github.com/bassam-motional/Reasonable-Crowd. | contribution 0.86 | 0.55 | 0.02 | asserted 0.70 |
What was examined, and what was not
- arXiv listing abstract v1
- full text
- figures and tables
- references and citing sentences
- relations to other papers
- P abstract
- S sections
- T body sentences
- C citations
- R cross-paper relations
The paper
Original abstract
Autonomous vehicles must balance a complex set of objectives. There is no consensus on how they should do so, nor on a model for specifying a desired driving behavior. We created a dataset to help address some of these questions in a limited operating domain. The data consists of 92 traffic scenarios, with multiple ways of traversing each scenario. Multiple annotators expressed their preference between pairs of scenario traversals. We used the data to compare an instance of a rulebook, carefully hand-crafted independently of the dataset, with several interpretable machine learning models such as Bayesian networks, decision trees, and logistic regression trained on the dataset. To compare driving behavior, these models use scores indicating by how much different scenario traversals violate each of 14 driving rules. The rules are interpretable and designed by subject-matter experts. First, we found that these rules were enough for these models to achieve a high classification accuracy on the dataset. Second, we found that the rulebook provides high interpretability without excessively sacrificing performance. Third, the data pointed to possible improvements in the rulebook and the rules, and to potential new rules. Fourth, we explored the interpretability vs performance trade-off by also training non-interpretable models such as a random forest. Finally, we make the dataset publicly available to encourage a discussion from the wider community on behavior specification for AVs. Please find it at github.com/bassam-motional/Reasonable-Crowd.
Human review
How the literature uses it
1 arXiv paper cite this in 1 sentence
What it cites
Cites as background 2 papers
-
AVs balance a complex set of objectives, such as driving safely and comfortably, complying with numerous traffic laws, getting to a destination, and generally meeting ethical and cultural expectations of reasonable driving behavior [ref] .
On a Formal Model of Safe and Scalable Self-driving Cars · Introduction · 0.80
-
A closed-loop ML-based planning benchmark for autonomous driving from human demonstrations is under development [ref] .
NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles · Related Work · 0.93
Discussion
Know something about this paper? Leave a comment, or highlight any passage above to comment on it.