Pith. sign in

REVIEW 3 major objections 7 minor 56 references

A new 2.85-million-frame simulation dataset for CARLA Leaderboard 2.0 ties perception and planning together.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 20:16 UTC pith:K74J4CGD

load-bearing objection TaCarla is a genuinely useful large Leaderboard 2.0 dataset with multi-task annotations and solid baselines; the planning-oracle question needs one clarifying sentence but is not a fatal flaw. the 3 major comments →

arxiv 2602.23499 v4 pith:K74J4CGD submitted 2026-02-26 cs.RO cs.AI

TaCarla: A comprehensive benchmarking dataset for end-to-end autonomous driving

Tugrul Gorgulu , Atakan Dag , M. Esat Kalfaoglu , Halil Ibrahim Kuru , Baris Can Cam , Halil Ibrahim Ozturk , Ozsel Kilinc This is my paper
classification cs.RO cs.AI
keywords autonomous drivingCARLA Leaderboard 2.0datasetplanningperceptionsensor configurationrarity scoresimulation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces TaCarla, a large-scale dataset collected in the CARLA simulation environment for the Leaderboard 2.0 autonomous driving challenge. It contains more than 2.85 million frames recorded with a NuScenes-style 360-degree sensor suite, designed to support dynamic object detection, lane and centerline detection, traffic light recognition, prediction, and planning in both open-loop and closed-loop settings. The authors argue that previous simulation datasets were limited either by restricted sensor configurations, oscillatory expert behavior, or a focus on planning alone. TaCarla aims to combine the strengths of those predecessors, using the rule-based PDM expert and adding richer annotations plus a rarity score to identify long-tail scenarios.

Core claim

TaCarla claims to be the largest publicly described dataset for the CARLA Leaderboard 2.0 challenge, providing over 2.85 million frames at 10 Hz, with six cameras, five radars, and one LiDAR arranged exactly as in NuScenes, along with auxiliary modalities such as BEV RGB, depth, and segmentation. The dataset includes seven dynamic object classes plus traffic light annotations, lane dividers, centerlines, ego waypoints, and rule-based text descriptions. The authors also propose a normalized rarity score derived from inverse document frequency to quantify how unusual each scene description is, and they report baseline results for 3D object detection, lane detection, traffic light detection, an

What carries the argument

The central mechanism is the pairing of the rule-based PDM-Lite expert policy with the NuScenes sensor configuration. The PDM expert provides stable, non-oscillatory driving behavior for generating ground-truth trajectories, while the NuScenes sensor suite gives 360-degree perception coverage, allowing the dataset to serve both planning and perception tasks. The rarity score, a normalized inverse document frequency over text annotations, is the identity that provides the dataset's uniqueness measure.

Load-bearing premise

The planning ground-truth labels assume that the PDM-Lite rule-based expert, which was originally front-sensor only, behaves correctly when the dataset provides 360-degree sensors; if the expert policy is not updated to use the full sensor suite, the planning labels in rear-centric scenarios may be suboptimal.

What would settle it

Inspect the PDM expert code or run it in the YieldToEmergencyVehicle scenario with the TaCarla sensor configuration and check whether the braking or yielding action occurs before the emergency vehicle becomes visible in the rear camera; if the expert never reacts to rear-approaching vehicles, the planning labels for that scenario do not reflect the intended 360-degree awareness.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If TaCarla is widely adopted, it could become a shared benchmark for closed-loop evaluation across perception and planning in the CARLA Leaderboard 2.0 environment.
  • The inclusion of rule-based text descriptions and a rarity score could enable the study of long-tail scenario coverage and its effect on model behavior.
  • Baselines for several tasks offer a reference point for comparing future end-to-end and modular driving models on the same data.
  • The NuScenes-style sensor layout may allow models pretrained or evaluated on nuScenes to be fine-tuned on TaCarla with minimal adaptation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct testable extension would be to rerun the PDM expert with a modified policy that explicitly uses rear cameras and fused LiDAR, and compare the resulting planning labels in YieldToEmergencyVehicle scenarios against the current dataset.
  • The rarity score could serve as a sampling weight during training to see whether upweighting rare scenes improves closed-loop performance on underrepresented scenarios.
  • The paper's claim that large datasets improve model performance suggests that scaling TaCarla to additional towns or weather conditions could further increase its value, though this would be a significant collection effort.
  • The planning ground truth relies on the PDM expert's decisions; a critical comparison against human or learned experts in edge cases would clarify the extent to which the dataset captures optimal behavior.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. The paper introduces TaCarla, a large-scale dataset collected in CARLA 0.9.15 for the Leaderboard 2.0 challenge. It contains over 2.85 million frames at 10 Hz, recorded with a NuScenes-style sensor configuration (6 cameras, 5 radars, 1 LiDAR) plus BEV RGB, depth, instance segmentation, and semantic segmentation. Annotations cover dynamic objects, lane dividers, centerlines, traffic lights, rule-based text descriptions, and a normalized rarity score. Data collection uses the PDM rule-based expert, with routes split at scenario trigger points. Baselines are reported for 3D object detection (RQR3D), lane/centerline detection (TopoBDA), traffic light detection (FCOS), and planning (Transfuser, DiffusionDrive, PlanT), including open-loop trajectory metrics and closed-loop Leaderboard 2.0 metrics in unseen Town13. The authors claim that TaCarla combines the robustness of PDM with the 360-degree coverage of NuScenes, providing the largest multi-task Leaderboard 2.0 dataset to date.

Significance. If the dataset and benchmarks are as described, TaCarla is a valuable resource for autonomous driving research: it is among the largest Leaderboard 2.0 datasets, provides multi-task annotations beyond planning, and includes both open-loop and closed-loop evaluation. The public dataset links and visualization code are a concrete strength, as are the rule-based text annotations and the proposed rarity score for long-tail analysis. The baseline suite spans perception and planning, and the use of a rule-based expert is a reasonable design choice relative to RL-based collectors. However, the reliability of the planning ground truth, the transparency of the training-filtering procedure, and the statistical basis of the baseline comparisons need to be strengthened before the benchmark claims are fully supported.

major comments (3)
  1. [§1 and §4.4] The paper motivates the NuScenes sensor configuration by observing that PDM-Lite's front-only setup has 'no input to detect an approaching emergency vehicle from behind' in YieldToEmergencyVehicle. It then states that TaCarla uses PDM as the expert policy and adopts NuScenes sensors. However, the manuscript never specifies whether the PDM expert was itself modified to consume the 360° sensor inputs, or whether it operates on privileged simulation state. If the expert still relies on front-only or privileged information, the planning labels in rear-approach scenarios may be suboptimal, and the claimed fix is only a property of the recorded sensor suite, not of the expert behavior. The driving-score filter in §4.4 implicitly concedes that some expert episodes are poor, but no per-scenario validation is provided. Please add an explicit description of the expert's inputs and per-scenario exp
  2. [§4.4, Table 7] The planning training set is filtered with a hand-chosen 'driving score > 70' threshold, but the paper gives no information about how many scenarios or frames are removed, whether the filter is applied per route or per frame, or whether the released labels are the filtered or unfiltered set. This matters because the open-loop metrics in Table 7 are computed on the validation set, and if the training distribution excludes a substantial fraction of hard scenarios, the reported performance is not a representative benchmark of the full planning task. Please report the size and composition of the filtered training set, the distribution of driving scores, and the effect of the threshold on performance.
  3. [Tables 3–8] All baseline results are reported as single numbers with no variance, seeds, or number of runs. For a dataset/benchmark paper, this makes it impossible to assess whether differences between models (e.g., Table 8: PlanT 52.95 vs. PlanT* 59.25; or DiffusionDrive vs. Transfuser in Table 7) are meaningful. Please provide mean and standard deviation over at least three seeds for the main perception and planning tables, or clearly state that these are single-run examples not intended as conclusive model comparisons.
minor comments (7)
  1. [Abstract/§1] The claim that TaCarla is 'the largest dataset in the literature' and the stated aim of an 'order of magnitude increase' are not supported by Table 2: Bench2Drive has 2M frames and PDM-Lite has 581k, so 2.85M frames is about 1.4x and 4.9x, not an order of magnitude. Please calibrate these claims.
  2. [Table 9] The AccidentTwoWays row for PlanT reports ADE 0.5810 with FDE 2.2346, AHE 0.1368, and FHE 0.1934, which exactly duplicate the Accident row for PlanT. This appears to be a copy-paste error; please check.
  3. [§4.3] The sentence 'Every individual image in the dataset consists of a traffic light instance with three distinct classes' is ambiguous. It should be clarified that each annotated image may contain multiple traffic lights, each labeled with one of the three classes red/yellow/green.
  4. [§4.4, Table 7] PlanT has missing values for H=4 and H=1 horizons. The paper should explain why these are missing (e.g., training details, evaluation protocol) rather than presenting them as empty cells.
  5. [Eq. (rarity score), §3] The min-max normalization in the rarity score makes scores comparable only within the corpus used to compute the extrema. If the dataset is extended or the score is applied to new samples, the normalized values will shift. This should be stated explicitly.
  6. [§4.5] The closed-loop evaluation is described as 'simplified validation dataset routes to 36 scenarios' but no details are given on how the routes were simplified, how many routes were used, or how they map to the official Leaderboard 2.0 evaluation. Please provide the exact route definitions or a more precise protocol.
  7. [Throughout] There are numerous typos and formatting issues (e.g., 'analyize', 'autonomus', 'Curvelanes', 'driving score ¿70'). A careful proofreading pass is needed.

Circularity Check

0 steps flagged

No significant circularity: dataset construction is self-contained; the few self-cited baselines are not load-bearing.

full rationale

TaCarla's central claim—a large, multi-task dataset for the CARLA Leaderboard 2.0 challenge—does not reduce to its inputs by construction. The data are generated with the external CARLA 0.9.15 simulator and Leaderboard 2.0 routes; the sensor rig is the nuScenes configuration (external corpus); and the planning ground truth comes from PDM-Lite, an external rule-based expert (OpenDriveLab), not from the authors' own models. The paper's own acknowledgment that PDM-Lite's front-only inputs are blind to rear-approaching actors in YieldToEmergencyVehicle (Section 1) is a genuine label-validity gap—the paper asserts that adopting the NuScenes 360° sensor configuration addresses it without stating that the expert policy itself was modified to consume those sensors. But this is a correctness/quality concern about the planning labels, not a circular derivation: the open-loop ADE/FDE metrics are computed against held-out expert trajectories, and the closed-loop numbers come from the official CARLA Leaderboard V2 metrics, both external yardsticks. The proposed rarity score is explicitly an adaptation of IDF with its formula given, so its behavior is definitional rather than a claimed first-principles prediction. The only self-citations—TopoBDA/TopoMaskV2 [19,20] and RQR3D [23] used as baseline architectures—are experimental tooling that demonstrate dataset usability; they are not premises of the dataset's validity, and the reported numbers are empirical measurements rather than fitted parameters renamed as predictions. No 'prediction' in the paper is equivalent by construction to a fitted quantity or to a self-citation chain, so the paper warrants a low score.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

The central dataset claim rests on simulator ground-truth fidelity, expert-policy quality, and cross-dataset compatibility. The free parameters listed affect only the baseline experiments, not the dataset itself.

free parameters (2)
  • Driving score filter threshold = 70
    Planning baselines are trained only on scenarios whose driving score exceeds 70 (Section 4.4). This hand-chosen threshold alters the training distribution and can make closed-loop results optimistic.
  • Lane-change command classification threshold = 2 m lateral at 15 m ahead
    Used in Section 4.4 to map lane-guidance waypoints to left/straight/right driving commands for Transfuser and DiffusionDrive; a design choice that affects planning inputs.
axioms (3)
  • domain assumption CARLA 0.9.15 provides ground-truth labels (3D boxes, lanes, traffic lights, depth) of sufficient accuracy for training and benchmarking.
    All dataset labels come from the CARLA simulator; no external validation or comparison against real-world annotations is provided.
  • domain assumption The PDM-Lite rule-based expert is a reliable oracle for planning labels across all 36 Leaderboard 2.0 scenarios.
    The planning ground truth consists of expert trajectories. The paper notes the expert's native front-only sensor setup is blind to rear-approaching actors, but does not demonstrate the expert handles those scenarios correctly after adopting the NuScenes sensor configuration.
  • domain assumption The NuScenes sensor configuration in CARLA is compatible with models trained on real nuScenes data.
    The paper claims 'seamless switch' between TaCarla and nuScenes, but does not run any zero-shot transfer or fidelity experiment to support cross-dataset compatibility.
invented entities (1)
  • Rarity score no independent evidence
    purpose: Normalized IDF-based metric to quantify scenario uniqueness and identify long-tail events
    The rarity score is introduced in this paper and normalized using min/max over the same corpus it describes; no external validation on other datasets or against human judgments is provided.

pith-pipeline@v1.3.0-alltime-deepseek · 16292 in / 12380 out tokens · 115455 ms · 2026-08-02T20:16:39.909153+00:00 · methodology

0 comments
read the original abstract

Collecting a high-quality dataset is a critical task that demands meticulous attention to detail, as overlooking certain aspects can render the entire dataset unusable. Autonomous driving challenges remain a prominent area of research, requiring further exploration to enhance the perception and planning performance of vehicles. However, existing datasets are often incomplete. For instance, datasets that include perception information generally lack planning data, while planning datasets typically consist of extensive driving sequences where the ego vehicle predominantly drives forward, offering limited behavioral diversity. In addition, many real datasets struggle to evaluate their models, especially for planning tasks, since they lack a proper closed-loop evaluation setup. The CARLA Leaderboard 2.0 challenge, which provides a diverse set of scenarios to address the long-tail problem in autonomous driving, has emerged as a valuable alternative platform for developing perception and planning models in both open-loop and closed-loop evaluation setups. Nevertheless, existing datasets collected on this platform present certain limitations. Some datasets appear to be tailored primarily for limited sensor configuration, with particular sensor configurations. To support end-to-end autonomous driving research, we have collected a new dataset comprising over 2.85 million frames using the CARLA simulation environment for the diverse Leaderboard 2.0 challenge scenarios. Our dataset is designed not only for planning tasks but also supports dynamic object detection, lane divider detection, centerline detection, traffic light recognition, prediction tasks and visual language action models . Furthermore, we demonstrate its versatility by training various models using our dataset. Moreover, we also provide numerical rarity scores to understand how rarely the current state occurs in the dataset.

Figures

Figures reproduced from arXiv: 2602.23499 by Atakan Dag, Baris Can Cam, Halil Ibrahim Kuru, Halil Ibrahim Ozturk, M. Esat Kalfaoglu, Ozsel Kilinc, Tugrul Gorgulu.

Figure 1
Figure 1. Figure 1: These images include views from 6 cameras, point cloud [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of the ego vehicle’s ground-truth location. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Outputs of the TaCarla Label Viewer To align our dataset with the nuScenes [4] benchmark, which provides annotated keyframes at 2 Hz, we downsam￾ple our original 10 Hz data to 2 Hz. This conversion ensures consistency in temporal resolution, facilitating fair compar￾isons and compatibility with existing evaluation protocols. Additionally, we select the scenarios whose name contains keywords such as acciden… view at source ↗
Figure 4
Figure 4. Figure 4: Bird’s Eye View (BEV) results demonstrating the per [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Outputs of the FCOS traffic light model 4.3. Traffic Light Detection The dataset we propose consists of 238,780 and 187,987 images containing traffic light instances in the training and validation sets, respectively. Every individual image in the dataset consists of a traffic light instance with three distinct classes: red, yellow, green. Every instance is labeled with its 2D bounding box and corresponding… view at source ↗
Figure 6
Figure 6. Figure 6: Waypoints from ground truth(Top Left), PlanT(Top [PITH_FULL_IMAGE:figures/full_fig_p008_6.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 7 linked inside Pith

  1. [1]

    A deep learning ap- proach to traffic lights: Detection, tracking, and classifica- tion

    Karsten Behrendt and Libor Novak. A deep learning ap- proach to traffic lights: Detection, tracking, and classifica- tion. InRobotics and Automation (ICRA), 2017 IEEE Inter- national Conference on. IEEE, 2017. 3

    2017

  2. [2]

    Unsupervised labeled lane markers using maps

    Karsten Behrendt and Ryan Soussan. Unsupervised labeled lane markers using maps. InProceedings of the IEEE Inter- national Conference on Computer Vision, 2019. 3

    2019

  3. [3]

    PDM-Lite: A rule-based planner for carla leaderboard 2.0.https : / / github

    Jens Beißwenger. PDM-Lite: A rule-based planner for carla leaderboard 2.0.https : / / github . com / OpenDriveLab/DriveLM/blob/DriveLM-CARLA/ docs/report.pdf, 2024. 1, 8

    2024

  4. [4]

    nuscenes: A multi- modal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Gi- ancarlo Baldan, and Oscar Beijbom. nuscenes: A multi- modal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020. 2, 3, 4, 6, 8

    2020

  5. [5]

    Addressing action oscillations through learning policy inertia, 2021

    Chen Chen, Hongyao Tang, Jianye Hao, Wulong Liu, and Zhaopeng Meng. Addressing action oscillations through learning policy inertia, 2021. 3

    2021

  6. [6]

    Persformer: 3d lane detection via perspective transformer and the openlane benchmark

    Li Chen, Chonghao Sima, Yang Li, Zehan Zheng, Jiajie Xu, Xiangwei Geng, Hongyang Li, Conghui He, Jianping Shi, Yu Qiao, and Junchi Yan. Persformer: 3d lane detection via perspective transformer and the openlane benchmark. InEu- ropean Conference on Computer Vision (ECCV), 2022. 3, 7

    2022

  7. [7]

    Transfuser: Imita- tion with transformer-based sensor fusion for autonomous driving.Pattern Analysis and Machine Intelligence (PAMI),

    Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, and Andreas Geiger. Transfuser: Imita- tion with transformer-based sensor fusion for autonomous driving.Pattern Analysis and Machine Intelligence (PAMI),

  8. [8]

    Sledge: Synthesizing driving environments with generative models and rule-based traffic, 2024

    Kashyap Chitta, Daniel Dauner, and Andreas Geiger. Sledge: Synthesizing driving environments with generative models and rule-based traffic, 2024. 3

    2024

  9. [9]

    Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmark- ing

    Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, and Kashyap Chitta. Navsim: Data-driven non- reactive autonomous vehicle simulation and benchmark- ing. InAdvances in Neural Information Processing Systems (NeurIPS), 2024. 7, 8

    2024

  10. [10]

    CARLA: An open urban driving simulator

    Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. InProceedings of the 1st Annual Conference on Robot Learning, pages 1–16, 2017. 8

    2017

  11. [11]

    Everingham, L

    M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge.International Journal of Computer Vision (IJCV), 88(2):303–338, 2010. 3

    2010

  12. [12]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. InConference on Computer Vision and Pattern Recog- nition (CVPR), 2012. 2

    2012

  13. [13]

    Springer International Publishing, 2020

    Yuliang Guo, Guang Chen, Peitao Zhao, Weide Zhang, Jing- hao Miao, Jingao Wang, and Tae Eun Choe.Gen-LaneNet: A Generalized and Scalable Approach for 3D Lane Detection, page 666–681. Springer International Publishing, 2020. 3, 7

    2020

  14. [14]

    Lvis: A dataset for large vocabulary instance segmentation

    Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. InThe IEEE Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2019. 3

    2019

  15. [15]

    Can vehicle motion planning generalize to realistic long-tail scenarios?, 2024

    Marcel Hallgarten, Julian Zapata, Martin Stoll, Katrin Renz, and Andreas Zell. Can vehicle motion planning generalize to realistic long-tail scenarios?, 2024. 3

    2024

  16. [16]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016. 5, 7

    2016

  17. [17]

    Bevdet4d: Exploit tempo- ral cues in multi-camera 3d object detection.arXiv preprint arXiv:2203.17054, 2022

    Junjie Huang and Guan Huang. Bevdet4d: Exploit tempo- ral cues in multi-camera 3d object detection.arXiv preprint arXiv:2203.17054, 2022. 5

    Pith/arXiv arXiv 2022

  18. [18]

    Bench2drive: Towards multi-ability bench- marking of closed-loop end-to-end autonomous driving

    Xiaosong Jia, Zhenjie Yang, Qifeng Li, Zhiyuan Zhang, and Junchi Yan. Bench2drive: Towards multi-ability bench- marking of closed-loop end-to-end autonomous driving. In NeurIPS 2024 Datasets and Benchmarks Track, 2024. 1, 3, 4, 8

    2024

  19. [19]

    Topobda: Towards bezier de- formable attention for road topology understanding.arXiv preprint arXiv:2412.18951, 2024

    Muhammet Esat Kalfaoglu, Halil Ibrahim Ozturk, Ozsel Kil- inc, and Alptekin Temizel. Topobda: Towards bezier de- formable attention for road topology understanding.arXiv preprint arXiv:2412.18951, 2024. 7

    arXiv 2024

  20. [20]

    Topomaskv2: Enhanced instance-mask-based formulation for the road topology prob- lem.arXiv preprint arXiv:2409.11325, 2024

    Muhammet Esat Kalfaoglu, Halil Ibrahim Ozturk, Ozsel Kilinc, and Alptekin Temizel. Topomaskv2: Enhanced instance-mask-based formulation for the road topology prob- lem.arXiv preprint arXiv:2409.11325, 2024. 7

    Pith/arXiv arXiv 2024

  21. [21]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models, 2020. 2

    2020

  22. [22]

    Towards learning- based planning: The nuplan benchmark for real-world au- tonomous driving

    Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, and Holger Caesar. Towards learning- based planning: The nuplan benchmark for real-world au- tonomous driving. In2024 IEEE International Conference on Robotics and Automation (I...

    2024

  23. [23]

    Rqr3d: Reparametrizing the regression targets for bev-based 3d object detection

    Ozsel Kilinc and Cem Tarhan. Rqr3d: Reparametrizing the regression targets for bev-based 3d object detection. 2025. 5

    2025

  24. [24]

    Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Ste- fan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. The open images dataset V4: unified image classification, object detection, and visual relationship detection at scale. arXiv e-prints:1811.00982, 2018. 3

    Pith/arXiv arXiv 2018

  25. [25]

    Hdmapnet: An online hd map construction and evaluation framework

    Qi Li, Yue Wang, Yilun Wang, and Hang Zhao. Hdmapnet: An online hd map construction and evaluation framework. In 2022 International Conference on Robotics and Automation (ICRA), pages 4628–4634. IEEE, 2022. 6

    2022

  26. [26]

    Graph-based topology reasoning for driv- ing scenes.arXiv preprint arXiv:2304.05277, 2023

    Tianyu Li, Li Chen, Huijie Wang, Yang Li, Jiazhi Yang, Xiangwei Geng, Shengyin Jiang, Yuting Wang, Hang Xu, Chunjing Xu, et al. Graph-based topology reasoning for driv- ing scenes.arXiv preprint arXiv:2304.05277, 2023. 6

    Pith/arXiv arXiv 2023

  27. [27]

    Zhiqi Li, Zhiding Yu, Shiyi Lan, Jiahan Li, Jan Kautz, Tong Lu, and Jose M. Alvarez. Is ego status all you need for open- loop end-to-end autonomous driving? In2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14864–14873, 2024. 4

    2024

  28. [28]

    Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving.arXiv preprint arXiv:2411.15139, 2024

    Bencheng Liao, Shaoyu Chen, Haoran Yin, Bo Jiang, Cheng Wang, Sixu Yan, Xinbang Zhang, Xiangyu Li, Ying Zhang, Qian Zhang, and Xinggang Wang. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving.arXiv preprint arXiv:2411.15139, 2024. 7, 8

    Pith/arXiv arXiv 2024

  29. [29]

    Maptrv2: An end-to-end framework for online vectorized hd map construction.International Journal of Computer Vision, pages 1–23, 2024

    Bencheng Liao, Shaoyu Chen, Yunchi Zhang, Bo Jiang, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Maptrv2: An end-to-end framework for online vectorized hd map construction.International Journal of Computer Vision, pages 1–23, 2024. 6

    2024

  30. [30]

    Microsoft COCO: Common Objects in Context

    Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll´ar, and C Lawrence Zitnick. Microsoft COCO: Common Objects in Context. InThe European Conference on Computer Vision (ECCV),

  31. [31]

    Regularizing action policies for smooth control with reinforcement learning, 2021

    Siddharth Mysore, Bassel Mabsout, Renato Mancuso, and Kate Saenko. Regularizing action policies for smooth control with reinforcement learning, 2021. 3

    2021

  32. [32]

    K-lane: Lidar lane dataset and benchmark for urban roads and highways

    Dong-Hee Paek, Seung-Hyun Kong, and Kevin Tirta Wijaya. K-lane: Lidar lane dataset and benchmark for urban roads and highways. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Work- shop on Autonomous Driving (WAD), 2022. 3

    2022

  33. [33]

    Spatial as deep: Spatial cnn for traffic scene understanding

    Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Spatial as deep: Spatial cnn for traffic scene understanding. InAAAI Conference on Artificial Intelligence (AAAI), 2018. 3

    2018

  34. [34]

    Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unproject- ing to 3d

    Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unproject- ing to 3d. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, Part XIV 16, pages 194–210. Springer, 2020. 5

    2020

  35. [35]

    Traffic light detection: A learning algorithm and evaluations on challenging dataset

    Mark Philip Philipsen, Morten Bornø Jensen, Andreas Møgelmose, Thomas B Moeslund, and Mohan M Trivedi. Traffic light detection: A learning algorithm and evaluations on challenging dataset. Inintelligent transportation systems (ITSC), 2015 IEEE 18th international conference on, pages 2341–2345. IEEE, 2015. 3

    2015

  36. [36]

    Designing network design spaces

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll ´ar. Designing network design spaces. InProceedings of the IEEE/CVF conference on com- puter vision and pattern recognition, pages 10428–10436,

  37. [37]

    Carllava: Vi- sion language models for camera-only closed-loop driving

    K Renz, L Chen, AM Marcu, J H ¨unermann, B Hanotte, A Karnsund, J Shotton, E Arani, and O Sinavski. Carllava: Vi- sion language models for camera-only closed-loop driving. arxiv 2024.arXiv preprint arXiv:2406.10165. 1

    Pith/arXiv arXiv 2024

  38. [38]

    Sophia Koepke, Zeynep Akata, and Andreas Geiger

    Katrin Renz, Kashyap Chitta, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata, and Andreas Geiger. Plant: Explainable planning transformers via object-level representations. InConference on Robotic Learning (CoRL),

  39. [39]

    A statistical interpretation of term specificity and its application in retrieval

    Karen Sp ¨arck Jones. A statistical interpretation of term specificity and its application in retrieval. InJournal of Doc- umentation, pages 11–21. Emerald Group Publishing, 1972. CiteSeerX 10.1.1.115.8343. 5

    1972

  40. [40]

    Car- lane: A lane detection benchmark for unsupervised do- main adaptation from simulation to multiple real-world do- mains

    Bonifaz Stuhr, Johann Haselberger, and Julian Gebele. Car- lane: A lane detection benchmark for unsupervised do- main adaptation from simulation to multiple real-world do- mains. InAdvances in Neural Information Processing Sys- tems, pages 4046–4058. Curran Associates, Inc., 2022. 3

    2022

  41. [41]

    Scalability in perception for autonomous driving: Waymo open dataset

    Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Et- tinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in percepti...

    2020

  42. [42]

    Efficientdet: Scalable and efficient object detection

    Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781–10790, 2020. 5

    2020

  43. [43]

    Fcos: Fully convolutional one-stage object detection

    Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. InThe IEEE International Conference on Computer Vision (ICCV), 2019. 7

    2019

  44. [44]

    Openlane-v2: A topology reasoning benchmark for unified 3d hd mapping.Advances in Neural Information Processing Systems, 36, 2024

    Huijie Wang, Tianyu Li, Yang Li, Li Chen, Chonghao Sima, Zhenbo Liu, Bangjun Wang, Peijin Jia, Yuting Wang, Shengyin Jiang, et al. Openlane-v2: A topology reasoning benchmark for unified 3d hd mapping.Advances in Neural Information Processing Systems, 36, 2024. 3, 6

    2024

  45. [45]

    Argoverse 2: Next generation datasets for self-driving perception and forecasting.arXiv preprint arXiv:2301.00493, 2023

    Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting.arXiv preprint arXiv:2301.00493, 2023. 3

    Pith/arXiv arXiv 2023

  46. [46]

    Topomlp: An simple yet strong pipeline for driving topology reasoning.ICLR, 2024

    Dongming Wu, Jiahao Chang, Fan Jia, Yingfei Liu, Tiancai Wang, and Jianbing Shen. Topomlp: An simple yet strong pipeline for driving topology reasoning.ICLR, 2024. 6

    2024

  47. [47]

    Curvelane-nas: Unifying lane- sensitive architecture search and adaptive point blending

    Hang Xu, Shaoju Wang, Xinyue Cai, Wei Zhang, Xiaodan Liang, Li, and Zhenguo. Curvelane-nas: Unifying lane- sensitive architecture search and adaptive point blending. In ECCV, 2020. 3

    2020

  48. [48]

    Once-3dlanes: Building monocular 3d lane detec- tion

    Fan Yan, Ming Nie, Xinyue Cai, Jianhua Han, Hang Xu, Zhen Yang, Chaoqiang Ye, Yanwei Fu, Michael Bi Mi, and Li Zhang. Once-3dlanes: Building monocular 3d lane detec- tion. InProceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, 2022. 3

    2022

  49. [49]

    Center- based 3d object detection and tracking

    Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center- based 3d object detection and tracking. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11784–11793, 2021. 5

    2021

  50. [50]

    End-to-end lane marker detection via row-wise classifica- tion, 2020

    Seungwoo Yoo, Heeseok Lee, Heesoo Myeong, Sungrack Yun, Hyoungwoo Park, Janghoon Cho, and Duck Hoon Kim. End-to-end lane marker detection via row-wise classifica- tion, 2020. 3

    2020

  51. [51]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. InIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020. 3

    2020

  52. [52]

    Bdd100k: A diverse driving dataset for heterogeneous multitask learning, 2020

    Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Dar- rell. Bdd100k: A diverse driving dataset for heterogeneous multitask learning, 2020. 3

    2020

  53. [53]

    Cafe-ad: Cross-scenario adaptive feature enhancement for trajectory planning in au- tonomous driving, 2025

    Junrui Zhang, Chenjie Wang, Jie Peng, Haoyu Li, Jianmin Ji, Yu Zhang, and Yanyong Zhang. Cafe-ad: Cross-scenario adaptive feature enhancement for trajectory planning in au- tonomous driving, 2025. 3

    2025

  54. [54]

    Lanevil: Benchmarking the robustness of lane detection to environmental illusions, 2024

    Tianyuan Zhang, Lu Wang, Hainan Li, Yisong Xiao, Siyuan Liang, Aishan Liu, Xianglong Liu, and Dacheng Tao. Lanevil: Benchmarking the robustness of lane detection to environmental illusions, 2024. 3

    2024

  55. [55]

    Vil-100: A new dataset and a baseline model for video instance lane de- tection, 2021

    Yujun Zhang, Lei Zhu, Wei Feng, Huazhu Fu, Mingqian Wang, Qingxia Li, Cheng Li, and Song Wang. Vil-100: A new dataset and a baseline model for video instance lane de- tection, 2021. 3

    2021

  56. [56]

    Annotation Details Samples in TaCarla were collected using the PlanT [38] data pipeline

    Supplementary 6.1. Annotation Details Samples in TaCarla were collected using the PlanT [38] data pipeline. An additional translation is applied to compensate for the virtual ego-vehicle center (1.3 m inx, 2.5 m inz). All labels are stored as Parquet files, which can be read with PySpark to reduce label loading time. In addition, we pro- vide code for rea...

    1934