The Waymo Open Sim Agents Challenge

Alex Kuefler; Brandyn White; Cole Gulino; Dragomir Anguelov; John Lambert; Michelle Li; Nick Rhinehart; Nico Montali; Paul Mougin; Shimon Whiteson

RECORD

The Waymo Open Sim Agents Challenge

Nico Montali, John Lambert, Paul Mougin, Alex Kuefler, Nick Rhinehart, Michelle Li, Cole Gulino, Tristan Emrich

show 4 more authors

Zoey Yang, Shimon Whiteson, Brandyn White, Dragomir Anguelov

Computer Vision and Pattern Recognition · Submitted 19 May 2023

abstract compiled · 2026-09-23 · no language-model review

The authors' main claim The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.

how it is stated · asserted 1.00 read the record →

Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2305.12032 (version 4)

Pith IDLTSWFCW3, Pith's permanent record of this paper and its review

CitationAlex Kuefler, Brandyn White, Cole Gulino, Dragomir Anguelov, John Lambert, Michelle Li, Nick Rhinehart, Nico Montali, Paul Mougin, Shimon Whiteson, Tristan Emrich, Zoey Yang. "The Waymo Open Sim Agents Challenge." arXiv:2305.12032 (2023). https://pith.science/paper/LTSWFCW3

Claim paper

The compiled record

The paper's own sentences, arranged. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Open the compiled record

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 4,834 input tokens.

What the abstract itself states, as probabilities: claims first 0.92 · states limitation 0.25 · human participants 0.08 · controlled comparison 0.66 · randomized assignment 0.01 · states quantitative result 0.20 · compares with prior results 0.71 · mentions code or data availability 0.27.

The abstract, sentence by sentence

arXiv listing abstract v4. Results and the main claim are underlined; show every role.

Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#	sentence	role	the authors' own work	states the main conclusion	strength
s0	Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development.	background 1.00	0.05	0.04	asserted 1.00
s1	In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC).	contribution 1.00	0.90	0.06	asserted 0.99
s2	WOSAC is the first public challenge to tackle this task and propose corresponding metrics.	contribution 0.86	0.65	0.09	asserted 1.00
s3	The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving.	contribution 0.84	0.15	0.12	asserted 0.98
s4	We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023.	contribution 0.80	0.93	0.05	asserted 0.99
s5	The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.	future work 0.90	0.63	0.09	asserted 1.00

What kind of paper the classifier reports this is: dataset or benchmark 0.96 · evidence from observation or experiment 0.36.

What was examined, and what was not

sources read

arXiv listing abstract v4

not examined

full text
figures and tables
references and citing sentences
relations to other papers

levels run

P abstract

not run

S sections
T body sentences
C citations
R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

The paper

Original abstract

Simulation with realistic, interactive agents represents a key task for autonomous vehicle software development. In this work, we introduce the Waymo Open Sim Agents Challenge (WOSAC). WOSAC is the first public challenge to tackle this task and propose corresponding metrics. The goal of the challenge is to stimulate the design of realistic simulators that can be used to evaluate and train a behavior model for autonomous driving. We outline our evaluation methodology, present results for a number of different baseline simulation agent methods, and analyze several submissions to the 2023 competition which ran from March 16, 2023 to May 23, 2023. The WOSAC evaluation server remains open for submissions and we discuss open problems for the task.

Human review

How Pith checks papers → · Full audit of this paper →

How the literature uses it

14 arXiv papers cite this in 42 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 29 sentences without a confident label

Uses its method or tool 3 papers

In addition, following WOSAC [ref] we also report the mean average displacement error (ADE) over 32 rollouts and the minimum average displacement error (minADE) over 32 rollouts.

Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving · Waymo Open Sim Agents Challenge · 0.88
We employ established evaluation metrics [ref] and Sim Agents Challenge metrics [ref] to assess the realism and diversity of generated trajectories.

WcDT: World-centric Diffusion Transformer for Traffic Scene Generation · Experimental Setup · 0.93
Following [ref] we formulate the problem as a hidden Markov model [equation] where $𝒮$ is the set of true world states, $𝒪$ is the set of observations, $p(o_t |s_t)$ is the emission distribution, and $p(s_t |s_t-1)$ governs the latent transition.

A Generative Model for Closed-Loop Microsimulation of Signalized Intersections · Problem Definition · 0.91

Uses its data or values 2 papers

We train our method and baselines on the Waymo Open Motion Dataset (WOMD) [ref] and evaluate on the Waymo Open Sim Agents Challenge (WOSAC) benchmark [ref] .

Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving · Experiments · 0.91
We train and evaluate the traffic scene generation model using the Waymo Motion Prediction dataset [ref] , which provides agents' motion trajectories and high-definition map data.

TSDiT: Traffic Scene Diffusion Models With Transformers · Dataset · 0.98

Compares with it 3 papers

We include several notable baselines reported by [ref] on WOSAC.

Improving Agent Behaviors with RL Fine-tuning for Autonomous Driving · Waymo Open Sim Agents Challenge · 0.86
Similar to the results of Sim Agents [ref] , the ADE and FDE of MTR++ [ref] were significantly lower than the other methods in our experiments, and the ADE of TSDiT was slightly better than that of MTR++, but the FDE was increased compared to MTR++.

TSDiT: Traffic Scene Diffusion Models With Transformers · Comparison Results · 0.99
We compare our proposed WcDT model against several state-of-the-art benchmarks submitted to the Sim Agent Challenge[Sim Agent Leaderboard as of 02-04-2024: <https://waymo.com/open/challenges/2023/sim-agents/>] [ref] , including Random Agent, Constant Velocity [ref] , MTR+++ [ref] , WayFormer [ref] , MULTIPATH++ [ref] , MVTA [ref] , and MVTE [ref] .

WcDT: World-centric Diffusion Transformer for Traffic Scene Generation · Comparison with state-of-the-art methods · 0.93

Cites it as background 4 papers

WOSAC [ref] is the first public benchmark for the evaluation of simulation agents in the domain of autonomous driving, introducing new evaluation metrics and leveraging large-scale real-world logged data [ref] with a diverse set of scenarios and agent behaviors.

Multiverse Transformer: 1st Place Solution for Waymo Open Sim Agents Challenge 2023 · Introduction · 0.97
The Waymo Open Sim Agents Challenge (WOSAC) [ref] popularized distributional evaluation of simulated rollouts against real trajectories, complementing task-level measures such as collision rate or goal completion.

Heterogeneous Self-Play for Realistic Highway Traffic Simulation · Related Work · 0.87
Moreover, the Waymo Open Sim Agents Challenge (WOSAC) [ref] is the first public challenge to develop simulations with realistic and interactive agents.

A Survey on Multimodal Large Language Models for Autonomous Driving · Multimodal Language Models for Perception · 0.86

Discussion

Know something about this paper? Leave a comment, or highlight any passage above to comment on it.