{"id":"bb0b2325-909f-4764-9f7e-92931008cd7d","arxiv_id":"2602.23499","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"TaCarla releases 2.85M CARLA Leaderboard 2.0 frames with nuScenes-style sensors, multi-task annotations, planning baselines, and a text-based rarity score.","lead":"TaCarla is a 2.85-million-frame simulated driving dataset built from CARLA Leaderboard 2.0 scenarios, with nuScenes-style sensors and labels for perception and planning. It includes baselines for 3D detection, lane detection, traffic-light detection, and closed-loop planning, plus a text-based rarity score for long-tail events.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Planning labels may be unreliable if PDM-Lite expert doesn't use 360° sensors; paper doesn't validate expert behavior in rear-observer scenarios.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the planning ground-truth labels are produced by PDM-Lite, and the paper does not clarify whether the expert uses 360° sensors or how it handles rear-observer scenarios. This is the most critical gap because the dataset's central value is as a benchmark for planning; if the expert is blind to relevant actors, the labels themselves are wrong, invalidating any downstream imitation learning on those scenarios. Although PDM-Lite is a rule-based planner that may use simulation ground truth (in which case the reader's specific mechanism is not applicable), the paper does not say this, and the lack of per-scenario expert validation remains a real issue. The paper's own decision to filter training scenarios by driving score > 70 further indicates that some expert trajectories are of low quality, yet no such filter or per-scenario driving scores are provided to users of the dataset. Other potential concerns (e.g., baseline models using only front cameras despite the 360° dataset, and the simplified closed-loop evaluation routes) are secondary because they relate to the baselines rather than the dataset's core correctness. Given that the paper is otherwise substantial and reproducible, a conditional verdict is appropriate: the dataset may be valuable, but these open questions about label validity must be addressed before it can be fully trusted.","tokens_in":16574,"tokens_out":11272,"duration_ms":107751,"concrete_test":"Inspect the PDM-Lite expert code used for data collection (or run a minimal reproduction) to determine whether its decisions use camera imagery, LiDAR, or only privileged simulator ground truth. Then, for a sample of YieldToEmergencyVehicle episodes from the dataset, run the expert in a closed-loop CARLA Leaderboard 2.0 simulation with the emergency vehicle approaching from behind, and compare the expert's waypoints to the recorded dataset labels. If the expert fails to yield (e.g., continues without slowing or pulling over), the planning labels for that scenario are invalid.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that TaCarla is a comprehensive planning benchmark hinges on the planning ground-truth labels generated by the PDM-Lite expert. The paper explicitly states (Section 1) that PDM-Lite's native front-only sensor configuration is blind to rear-approaching actors in YieldToEmergencyVehicle scenarios, and that adopting the NuScenes 360° configuration addresses this. However, it never states whether the expert policy itself was modified to use the new 360° sensors or whether it still relies only on front cameras — or, alternatively, on privileged simulation ground truth. If the expert still uses only front-facing inputs, then in exactly those scenarios where the emergency vehicle approaches from behind, the planning labels will be suboptimal or incorrect: the ego vehicle will not yield or will yield too late. This directly corrupts the open-loop planning metrics (ADE/FDE) and the closed-loop evaluation for those scenarios, undermining the dataset's claim to support comprehensive planning across all Leaderboard 2.0 scenarios. The paper provides no per-scenario validation of the expert's behavior, and its own training-set filtering (driving score > 70) suggests that some scenarios contain poor expert trajectories, making the ambiguity about the expert's perception a load-bearing gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TaCarla, a large-scale dataset collected in CARLA 0.9.15 for the Leaderboard 2.0 challenge. It contains over 2.85 million frames at 10 Hz, recorded with a NuScenes-style sensor configuration (6 cameras, 5 radars, 1 LiDAR) plus BEV RGB, depth, instance segmentation, and semantic segmentation. Annotations cover dynamic objects, lane dividers, centerlines, traffic lights, rule-based text descriptions, and a normalized rarity score. Data collection uses the PDM rule-based expert, with routes split at scenario trigger points. Baselines are reported for 3D object detection (RQR3D), lane/centerline detection (TopoBDA), traffic light detection (FCOS), and planning (Transfuser, DiffusionDrive, PlanT), including open-loop trajectory metrics and closed-loop Leaderboard 2.0 metrics in unseen Town13. The authors claim that TaCarla combines the robustness of PDM with the 360-degree coverage of NuScenes, providing the largest multi-task Leaderboard 2.0 dataset to date.","tokens_in":16851,"tokens_out":5753,"duration_ms":55729,"significance":"If the dataset and benchmarks are as described, TaCarla is a valuable resource for autonomous driving research: it is among the largest Leaderboard 2.0 datasets, provides multi-task annotations beyond planning, and includes both open-loop and closed-loop evaluation. The public dataset links and visualization code are a concrete strength, as are the rule-based text annotations and the proposed rarity score for long-tail analysis. The baseline suite spans perception and planning, and the use of a rule-based expert is a reasonable design choice relative to RL-based collectors. However, the reliability of the planning ground truth, the transparency of the training-filtering procedure, and the statistical basis of the baseline comparisons need to be strengthened before the benchmark claims are fully supported.","major_comments":[{"comment":"The paper motivates the NuScenes sensor configuration by observing that PDM-Lite's front-only setup has 'no input to detect an approaching emergency vehicle from behind' in YieldToEmergencyVehicle. It then states that TaCarla uses PDM as the expert policy and adopts NuScenes sensors. However, the manuscript never specifies whether the PDM expert was itself modified to consume the 360° sensor inputs, or whether it operates on privileged simulation state. If the expert still relies on front-only or privileged information, the planning labels in rear-approach scenarios may be suboptimal, and the claimed fix is only a property of the recorded sensor suite, not of the expert behavior. The driving-score filter in §4.4 implicitly concedes that some expert episodes are poor, but no per-scenario validation is provided. Please add an explicit description of the expert's inputs and per-scenario exp","section":"§1 and §4.4"},{"comment":"The planning training set is filtered with a hand-chosen 'driving score > 70' threshold, but the paper gives no information about how many scenarios or frames are removed, whether the filter is applied per route or per frame, or whether the released labels are the filtered or unfiltered set. This matters because the open-loop metrics in Table 7 are computed on the validation set, and if the training distribution excludes a substantial fraction of hard scenarios, the reported performance is not a representative benchmark of the full planning task. Please report the size and composition of the filtered training set, the distribution of driving scores, and the effect of the threshold on performance.","section":"§4.4, Table 7"},{"comment":"All baseline results are reported as single numbers with no variance, seeds, or number of runs. For a dataset/benchmark paper, this makes it impossible to assess whether differences between models (e.g., Table 8: PlanT 52.95 vs. PlanT* 59.25; or DiffusionDrive vs. Transfuser in Table 7) are meaningful. Please provide mean and standard deviation over at least three seeds for the main perception and planning tables, or clearly state that these are single-run examples not intended as conclusive model comparisons.","section":"Tables 3–8"}],"minor_comments":[{"comment":"The claim that TaCarla is 'the largest dataset in the literature' and the stated aim of an 'order of magnitude increase' are not supported by Table 2: Bench2Drive has 2M frames and PDM-Lite has 581k, so 2.85M frames is about 1.4x and 4.9x, not an order of magnitude. Please calibrate these claims.","section":"Abstract/§1"},{"comment":"The AccidentTwoWays row for PlanT reports ADE 0.5810 with FDE 2.2346, AHE 0.1368, and FHE 0.1934, which exactly duplicate the Accident row for PlanT. This appears to be a copy-paste error; please check.","section":"Table 9"},{"comment":"The sentence 'Every individual image in the dataset consists of a traffic light instance with three distinct classes' is ambiguous. It should be clarified that each annotated image may contain multiple traffic lights, each labeled with one of the three classes red/yellow/green.","section":"§4.3"},{"comment":"PlanT has missing values for H=4 and H=1 horizons. The paper should explain why these are missing (e.g., training details, evaluation protocol) rather than presenting them as empty cells.","section":"§4.4, Table 7"},{"comment":"The min-max normalization in the rarity score makes scores comparable only within the corpus used to compute the extrema. If the dataset is extended or the score is applied to new samples, the normalized values will shift. This should be stated explicitly.","section":"Eq. (rarity score), §3"},{"comment":"The closed-loop evaluation is described as 'simplified validation dataset routes to 36 scenarios' but no details are given on how the routes were simplified, how many routes were used, or how they map to the official Leaderboard 2.0 evaluation. Please provide the exact route definitions or a more precise protocol.","section":"§4.5"},{"comment":"There are numerous typos and formatting issues (e.g., 'analyize', 'autonomus', 'Curvelanes', 'driving score ¿70'). A careful proofreading pass is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is a dataset contribution from an industry group and uses two self-authored baselines (TopoBDA and RQR3D); this is not a conflict, but the reviewers should verify that those baselines are not tuned to favor the dataset. The central dataset-collection effort appears genuine and the release is a strength, but the planning-label validity question is load-bearing enough to require a revision cycle. I would recommend checking the license and completeness of the HuggingFace release before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a real contribution. TaCarla gives the community a 2.85M-frame CARLA Leaderboard 2.0 dataset with NuScenes-style 360° sensors, multiple annotation types (3D boxes, lanes, centerlines, traffic lights, text descriptions), a rarity score, and baselines for perception and planning. That is more than Bench2Drive and PDM-Lite offer individually. The released links and the scenario counts make the central claim plausible, and the planning baselines (Transfuser, DiffusionDrive, PlanT) are sensible choices.\n\nThe best part is the breadth: one dataset for both modular perception and end-to-end planning, with a clean sensor suite that maps to nuScenes. The rarity score is a small but useful addition.\n\nNow the soft spots, in proportion.\n\nThe stress-test worry about the PDM-Lite expert being blind to rear-approaching vehicles does not land cleanly. PDM-Lite is a rule-based planner; nothing in the paper says it relies on the three front cameras for perception, and the sentence that 'the PDM-Lite dataset ... employs a sensor configuration' refers to the recorded sensors, not necessarily the planner's internal input. If the expert uses privileged state from CARLA, the rear-view issue is moot. That said, the paper never actually says this, and given that the motivation for switching to NuScenes sensors was exactly that failure mode, the authors need to state explicitly whether the expert policy uses the 360° sensors, privileged ground truth, or the original front-only inputs. One sentence in the experimental section would close this.\n\nMore concrete weaknesses: the baseline tables report single runs with no variance; the training-set filter (driving score >70) is applied without reporting how many scenarios are dropped; and closed-loop scores could be inflated by cherry-picking routes. PlanT* with rarity>0 is an unexplained extra condition. The writing also needs a cleanup (typos, 'PlanT data pipeline' in the supplement).\n\nOverall this paper deserves a serious referee. It is not a paradigm shift, but it is a solid dataset artifact with a plausible central claim and public artifacts. The authors should be asked to clarify the expert's perception, add error bars or at least multiple seeds, and transparently report the filtering. After those revisions, I would be comfortable using it as a benchmark.\n\nRecommendation: engage with review; treat the planning-oracle clarification as a necessary revision, not grounds for rejection.\n\nBest,\n[Your name]","headline":"TaCarla is a genuinely useful large Leaderboard 2.0 dataset with multi-task annotations and solid baselines; the planning-oracle question needs one clarifying sentence but is not a fatal flaw.","tokens_in":17343,"tokens_out":4380,"would_cite":true,"duration_ms":36978,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 2.85-million-frame simulation dataset for CARLA Leaderboard 2.0 ties perception and planning together.","keywords":["autonomous driving","CARLA Leaderboard 2.0","dataset","planning","perception","sensor configuration","rarity score","simulation"],"falsifier":"Inspect the PDM expert code or run it in the YieldToEmergencyVehicle scenario with the TaCarla sensor configuration and check whether the braking or yielding action occurs before the emergency vehicle becomes visible in the rear camera; if the expert never reacts to rear-approaching vehicles, the planning labels for that scenario do not reflect the intended 360-degree awareness.","tokens_in":16502,"feed_emoji":"🚗","tokens_out":1004,"duration_ms":11305,"temperature":0.7,"pith_summary":"The paper introduces TaCarla, a large-scale dataset collected in the CARLA simulation environment for the Leaderboard 2.0 autonomous driving challenge. It contains more than 2.85 million frames recorded with a NuScenes-style 360-degree sensor suite, designed to support dynamic object detection, lane and centerline detection, traffic light recognition, prediction, and planning in both open-loop and closed-loop settings. The authors argue that previous simulation datasets were limited either by restricted sensor configurations, oscillatory expert behavior, or a focus on planning alone. TaCarla aims to combine the strengths of those predecessors, using the rule-based PDM expert and adding richer annotations plus a rarity score to identify long-tail scenarios.","feed_headline":"2.85M frames tie AD perception to planning","feed_subtitle":"TaCarla dataset uses NuScenes sensors and a rule-based expert to enable closed-loop driving benchmarks.","key_machinery":"The central mechanism is the pairing of the rule-based PDM-Lite expert policy with the NuScenes sensor configuration. The PDM expert provides stable, non-oscillatory driving behavior for generating ground-truth trajectories, while the NuScenes sensor suite gives 360-degree perception coverage, allowing the dataset to serve both planning and perception tasks. The rarity score, a normalized inverse document frequency over text annotations, is the identity that provides the dataset's uniqueness measure.","core_discovery":"TaCarla claims to be the largest publicly described dataset for the CARLA Leaderboard 2.0 challenge, providing over 2.85 million frames at 10 Hz, with six cameras, five radars, and one LiDAR arranged exactly as in NuScenes, along with auxiliary modalities such as BEV RGB, depth, and segmentation. The dataset includes seven dynamic object classes plus traffic light annotations, lane dividers, centerlines, ego waypoints, and rule-based text descriptions. The authors also propose a normalized rarity score derived from inverse document frequency to quantify how unusual each scene description is, and they report baseline results for 3D object detection, lane detection, traffic light detection, an","pith_inferences":["A direct testable extension would be to rerun the PDM expert with a modified policy that explicitly uses rear cameras and fused LiDAR, and compare the resulting planning labels in YieldToEmergencyVehicle scenarios against the current dataset.","The rarity score could serve as a sampling weight during training to see whether upweighting rare scenes improves closed-loop performance on underrepresented scenarios.","The paper's claim that large datasets improve model performance suggests that scaling TaCarla to additional towns or weather conditions could further increase its value, though this would be a significant collection effort.","The planning ground truth relies on the PDM expert's decisions; a critical comparison against human or learned experts in edge cases would clarify the extent to which the dataset captures optimal behavior."],"forward_implications":["If TaCarla is widely adopted, it could become a shared benchmark for closed-loop evaluation across perception and planning in the CARLA Leaderboard 2.0 environment.","The inclusion of rule-based text descriptions and a rarity score could enable the study of long-tail scenario coverage and its effect on model behavior.","Baselines for several tasks offer a reference point for comparing future end-to-end and modular driving models on the same data.","The NuScenes-style sensor layout may allow models pretrained or evaluated on nuScenes to be fine-tuned on TaCarla with minimal adaptation."],"fun_headline_variants":["2.85M-frame CARLA dataset unifies perception and planning","TaCarla: 2.85M frames for end-to-end driving benchmarks","Largest CARLA Leaderboard 2.0 dataset ties perception to planning","CARLA dataset with 2.85M frames powers closed-loop AD testing"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The planning ground-truth labels assume that the PDM-Lite rule-based expert, which was originally front-sensor only, behaves correctly when the dataset provides 360-degree sensors; if the expert policy is not updated to use the full sensor suite, the planning labels in rear-centric scenarios may be suboptimal.","fun_headline_variants_meta":{"raw":{"variants":["2.85M-frame CARLA dataset unifies perception and planning","TaCarla: 2.85M frames for end-to-end driving benchmarks","Largest CARLA Leaderboard 2.0 dataset ties perception to planning","CARLA dataset with 2.85M frames powers closed-loop AD testing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1247,"prompt_tokens":811,"completion_tokens":436,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":555,"completion_tokens_details":{"reasoning_tokens":354}},"tokens_in":555,"tokens_out":436,"duration_ms":4237,"temperature":1.0,"reasoning_tokens":354,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T20:16:39.909153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Inspect the PDM expert code or run it in the YieldToEmergencyVehicle scenario with the TaCarla sensor configuration and check whether the braking or yielding action occurs before the emergency vehicle becomes visible in the rear camera; if the expert never reacts to rear-approaching vehicles, the planning labels for that scenario do not reflect the intended 360-degree awareness.","supporting_citations":[],"review_version":1}