{"id":"23d59cd7-043c-4b88-a27b-22a95e2fca19","arxiv_id":"2508.16962","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"HDSim injects driving styles into simulation by modifying the BEV perception inputs of background vehicles, yielding up to 68% more detected AD failures and style classifications that match real data.","lead":"HDSim generates background traffic for self-driving tests by giving each virtual car a human-like driving style, implemented as LLM-written code that changes what the vehicle's perception module sees. The paper reports that this stylized traffic exposes up to 68% more failures in tested autonomous-driving models than homogeneous traffic.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 68% failure-detection improvement in the abstract is not backed by any failure-count metric in §4; Tables 1–2 report DS/RC relative drops, so the headline quantitative claim currently rests on an undefined proxy.","rationale":"I read the paper as making two connected claims: that PMBI produces human-like stylized driving behavior, and that this behavior reveals up to 68% more safety-critical failures in tested AD models. The reader's weakest_assumption focused on the theoretical validity of PMBI as a proxy for human cognitive biases. That is a real concern, but it is downstream of a more immediately checkable problem: the paper's central quantitative claim is not operationalized. The experiments report DS and RC percentage drops, which are route-level aggregate scores, not failure counts. A 67.6% DS drop (the closest number to 68% in the tables) could arise from one early collision on a route, and it does not by itself establish that more safety-critical failures were 'detected.' Without a defined failure event, a counting procedure, and variance information, the headline number is unverifiable from the manuscript alone. This is not an internal inconsistency or a disagreement with consensus; it is a missing link between the reported data and the abstract's strongest assertion. The reader's rationale did flag this empirical gap, so there is partial agreement, but the reader's formal weakest_assumption was the PMBI proxy, which I would rank second. A conditional verdict is appropriate: the idea is plausible, the mechanism is clearly described, and the realism analysis is suggestive, but the central quantitative claim needs to be re-derived with an explicit failure-count metric and error bars before acceptance.","tokens_in":11347,"tokens_out":3629,"duration_ms":40499,"concrete_test":"Obtain or release raw CARLA logs for the baseline and HDSim conditions for at least one tested AD model (e.g., InterFuser, Table 1). Define a safety-critical failure explicitly, such as any route segment with a collision, a red-light violation, or an RC below a fixed threshold, and count failures per route over a fixed set of seeds. Compute the relative increase in failure counts from the baseline to the HDSim condition. If the increase does not reproduce the reported 68% figure, or if the DS drops are driven by one catastrophic event per route rather than multiple distinct failures, the headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantified claim, 'up to 68% more safety-critical failures' (Abstract and §1), is not supported by a direct failure-count metric anywhere in §4. Tables 1–2 report Driving Score (DS) and Route Compliance (RC) as relative percentage drops from a baseline, not as counts or rates of safety-critical failures such as collisions, red-light violations, or route-abandonment events. The largest tabled DS reduction is 67.6% (AIM, aggressive/normal/normal, Table 2), but a DS drop is not a 'failure case' count: a single collision on one route can reduce DS by a large fraction, while multiple independent failures could be conflated into one route-level score. No per-route failure definitions, no absolute counts, no variances, and no seed information are provided. Therefore the headline improvement, which is the main practical contribution of HDSim, is not derivable from the reported evidence. The PMBI mechanism's faithfulness is important, but the empirical gap in the central metric is more immediately load-bearing: if the 68% figure cannot be reproduced with a proper failure-count definition, the paper's core claim fails regardless of how plausible the perception-manipulation mechanism is.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces HDSim, a framework for generating human-like background traffic in CARLA for testing autonomous driving systems. HDSim models drivers through a hierarchical style model (L1 personality, L2 physiological, L3 attentional influences) and translates style descriptions into BEV-image manipulations via an LLM and a set of predefined APIs, a mechanism the authors call Perception-Mediated Behavior Influence (PMBI). The paper evaluates HDSim with the InterFuser model plus five other AD models, reporting Driving Score and Route Compliance under style-homogeneous, style-heterogeneous, and challenging traffic; it also compares simulated trajectory style realism against the INTERACTION dataset and reports system overhead. The central claims are that embedding HDSim improves detection of safety-critical AD failures by up to 68% and yields accident cases consistent with real-world NHTSA records.","tokens_in":11564,"tokens_out":6154,"duration_ms":59021,"significance":"If the quantitative claims were properly supported, HDSim would be a practically valuable contribution to AD testing: it adds semantically meaningful, composable driving-style diversity to a standard simulator without retraining the tested AD models or the background-driver policy, and it offers an interpretable perceptual mechanism plus efficiency data. The cognitive-hierarchy framing and the idea of using LLM-generated perception-modulation programs are interesting and go beyond simple parameter tweaks. The paper ships no code or benchmark artifacts, however, and the experimental support for the headline failure-detection gain is currently indirect, so the significance is conditional on a metric that directly measures failures.","major_comments":[{"comment":"The headline claim of \"up to 68% more\" safety-critical failures is not supported by the reported evidence. Tables 1 and 2 report relative drops in Driving Score and Route Compliance, not counts, rates, or any per-route definition of a failure event. A Driving Score drop such as the 67.6% row for AIM (Table 2) can be dominated by a single collision on one route and does not quantify how many additional distinct failure events were discovered. The abstract and introduction should either be rephrased in terms of Driving Score degradation or, preferably, augmented with a direct failure metric: e.g., numbers of collisions, red-light violations, off-road events, and route abandonments across the 10 routes, with absolute counts and per-seed variance.","section":"Abstract and §4.3, Tables 1–2"},{"comment":"The experiments are described as \"repeated three times to ensure statistical robustness,\" yet no table reports error bars, confidence intervals, standard deviations, or random seeds. Because the compared conditions differ by a few percentage points to tens of percentage points, it is impossible to tell which differences are real. Please report mean ± std over seeds and routes for DS/RC and for any new failure-count metric.","section":"§4.1 and Tables 1–2"},{"comment":"The PMBI mechanism assumes that modifying the BEV image seen by a competent AD model is a faithful proxy for how human cognitive biases alter perception. That assumption is load-bearing for the realism claims, and the current validation is indirect: Table 3 and Figure 6 show that the resulting trajectories are classified as style-consistent, but they do not establish that the perceptual-manipulation route itself is what makes them human-like, as opposed to simple action-level distortions that happen to be correlated with style. Please add an ablation that isolates the perception channel (e.g., compare PMBI against the same policy written directly at the action level, or against direct parameter changes) and, if possible, a sanity check that the modified BEV corresponds to the intended subjective interpretation rather than to corrupted sensor data.","section":"§3.3 and §4.4"}],"minor_comments":[{"comment":"The citation for Tesla deployment status points to a company accessories blog with a misspelled domain (teslaacessories.com); please replace it with an official or peer-reviewed source.","section":"§1 and References"},{"comment":"The realism comparison is not fully apples-to-apples: CARLA has no entries for distracted or fatigued styles, and the ProSim F1 for the distracted style is 0.00. The reported average improvements over CARLA and ProSim should be restricted to the styles where both baselines are defined, or the per-style gaps should be reported separately.","section":"§4.4, Table 3"},{"comment":"The claim of strong alignment with real-world accidents rests on three NHTSA case IDs, but the paper does not describe the matching procedure or criteria. Please specify how the simulated cases were selected and how correspondence was judged, ideally with a small protocol or a blinded comparison.","section":"§4.4"},{"comment":"The style-realistic annotations were produced by five drivers, but no inter-annotator agreement is reported; since these labels train the classifiers used in Table 3, reporting Cohen's kappa or a similar measure would strengthen the validation.","section":"§4.4"},{"comment":"The text says runtime \"remains comparable\" to rule-based CARLA simulation, but the driver step is 0.0229 s versus 0.0057 s for the rule simulator at 30 agents, roughly a 4x gap; please either rephrase the claim or report the overhead ratio explicitly.","section":"§4.5, Table 5"},{"comment":"Table 2 appears to contain an incomplete row: \"normal fatigued distracted\" has no DS/RC entries, and the \"Heter.\" label is placed on the following row. Please check the table layout and include all values.","section":"All Tables, especially Table 2"},{"comment":"There are several typos and formatting issues: \"aggressiv\" in Section 2, \"Incretmental\" in Figure 3, \"indiretly\" in Figure 4, \"Performanc\" in Table 4, and \"AP Is\" in Algorithm 1; the reference to \"Zhan and et al.\" should be \"Zhan et al.\"","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The core idea is timely and could appeal to the autonomous-driving simulation community, but the manuscript currently overstates its main result. No code, benchmark scripts, or seed data are provided, which makes the 68% figure even harder to verify; I would encourage the editor to treat code/data release as part of the revision if the authors can comply. The realism validation via INTERACTION-derived classifiers is a reasonable external anchor, though the small annotation pool and lack of agreement statistics limit it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nWorth knowing: the core idea here is genuinely fresh. The authors build a simulator in which an LLM generates code that modifies the BEV input to an AD model, effectively biasing perception rather than low-level control. That is not what ProSim or CARLA-style parameter tweaks do, and it's a clever way to inject style without touching the policy. The hierarchical three-layer style model (personality, physiological, attentional) is also a sensible way to think about composable driver traits.\n\nThat said, the headline number is not supported. The abstract and intro claim 'up to 68% more safety-critical failures,' but §4 reports only relative Driving Score and Route Compliance drops. A DS reduction is not a failure count. The largest tabled drop is 67.6% in AIM's DS, which could reflect one collision on one route. There are no absolute failure counts, no per-route definitions, no error bars or confidence intervals despite three route repeats. This is load-bearing: the paper's practical contribution is better failure detection, and that specific claim is currently unmeasured.\n\nOther soft spots are more minor. The realism validation relies on five human annotators labeling INTERACTION trajectories, with no inter-annotator agreement, split details, or classification protocol reported. The 'distracted' F1 of 45% is low and gets a passing mention at best. The PMBI fidelity assumption—that BEV alteration is a valid proxy for human cognitive bias—is an assumption, though the paper is transparent about it. The efficiency and robustness tables are welcome, and the failure case studies do align with NHTSA-style incidents, though only for three examples.\n\nCredit where due: the paper ships no code, but the mechanism is described in enough detail to be reimplemented, the related work is handled fairly, and the authors do not oversell the mechanism in the body even if the abstract oversells the results. The central argument holds up as a direction; it just needs a much sharper evaluation to back the specific claim.\n\nI'd send this to a serious referee. The idea deserves a close look, but the revision needs to add absolute failure rates, variance, and an unambiguous definition of 'failure detection.' Without that, the headline claim should be softened.","headline":"Original idea with a real experimental gap: the headline '68% more failures' isn't supported by the reported metrics.","tokens_in":12113,"tokens_out":2157,"would_cite":false,"duration_ms":22667,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HDSim embeds human cognitive driving styles into simulation by having an LLM rewrite the perception of background vehicles, and reports exposing up to 68% more safety-critical AD failures than conventional traffic.","keywords":["human-like traffic simulation","autonomous driving testing","large language models","perception-mediated behavior influence","driver style modeling","safety-critical failure detection","CARLA simulator","cognitive driver model"],"falsifier":"Run the reported AD models in the same CARLA scenarios with the same style triplets, but replace PMBI's BEV edits with equivalent action-level perturbations matched to the same style descriptors; if the detected failure counts do not drop when the BEV edits are removed, perception mediation is not the causal driver of the reported 68% improvement.","tokens_in":11124,"feed_emoji":"🚗","tokens_out":8899,"duration_ms":84240,"temperature":0.7,"pith_summary":"This paper tries to establish that the missing ingredient in self-driving simulators is human cognitive style, and that style can be injected by changing what a background driver perceives rather than by reprogramming its actions. HDSim wraps a competent driving core in three stackable influence layers -- personality, physiological state, and attention -- and uses an LLM to translate a natural-language style description into small edits of the bird's-eye-view image the core consumes. In CARLA experiments, the authors report that this perception-mediated approach exposes up to 68% more safety-critical failures in tested self-driving systems than homogeneous rule-based background traffic, while the induced accident patterns match real-world reports. If the claim holds, AD testers get a plug-and-play way to fill simulators with diverse, interpretable human drivers without retraining the systems under test.","feed_headline":"HDSim uncovers up to 68% more self-driving failures in traffic tests","feed_subtitle":"Injected by editing what background vehicles perceive, simulated human styles expose hidden self-driving blind spots.","key_machinery":"The load-bearing mechanism is Perception-Mediated Behavior Influence (PMBI): an LLM converts a style description into a policy set and then into executable calls against 16 perception-modulation APIs that redraw the BEV image fed to the agent's driving model -- changing object distances, sizes, positions, traffic-light states, and lane geometry. The same off-the-shelf AD model supplies the agent's basic driving competence, so style emerges from how the model reacts to its edited subjective view rather than from hand-coded action rules. A hierarchical driver style model organizes which edits are active -- stable personality edits, periodically updated physiological edits, and stochastically triggered attentional edits -- and Weber-Fechner-style parameter smoothing keeps the illusion coherent across steps.","core_discovery":"The paper's central claim is that human driving style can be modeled as a hierarchy of cognitive influences and realized inside a simulator by biasing perception rather than commands. The authors define a Driving Capability Layer, implemented by an off-the-shelf AD model shared by all agents, surrounded by three Style Influence Layers for personality, physiological state, and attention. An LLM, prompted with the style description and supplied with 16 predefined BEV-manipulation APIs plus 62 handcrafted examples, writes code that alters the subjective image -- making a leading vehicle look farther away, lane marks look curved, or a green light look yellow -- and the AD model's own planner then produces the stylized action. The paper reports that in style-homogeneous traffic the tested AD models' Driving Scores drop by up to roughly 65%, that selected challenging scenarios reveal up to 68% more failures than the normal baseline, and that simulated trajectory statistics and accident cases align with human-labeled real-world data better than the compared baselines.","pith_inferences":["If perception edits are the true cause of the reported gains, a direct testable extension is applying PMBI to other perceptual modalities such as LiDAR point clouds or language-conditioned observations; success there would extend stylized traffic to sensor-fusion AD stacks without retraining.","The realism experiments compare simulated trajectories to human-labeled styles, but they do not verify that the perceptual edits correspond to actual human visual biases; a psychophysical validation (e.g., measuring distance misestimation in fatigued drivers) would either support or undermine the cognitive grounding.","The Weber-Fechner consistency rule implies a concrete prediction: the same style should produce scaled rather than constant perceptual edits as speed and distance change, which can be checked by inspecting the LLM's API parameters across contexts.","A natural limitation to watch: AD models that plan from high-level abstractions rather than raw BEV images may be less steerable by PMBI, so the 68% figure may depend on the BEV-centric architecture of the tested systems."],"forward_implications":["Embedding HDSim into existing CARLA-style test pipelines can reveal hidden failure modes that homogeneous traffic hides, with the paper reporting up to 68% more detected safety-critical failures.","Because style is applied only to perception inputs, the same driving model can serve as the core for many stylized background agents without retraining, making new styles a matter of writing a new natural-language description.","Each style-induced accident can be traced to a concrete perceptual edit (e.g., underestimating following distance), which gives testers an interpretable explanation of why the AD system failed.","Stylized background agents keep near-baseline route completion with full route compliance in single-style tests, indicating that detected failures reflect weaknesses of the tested system rather than degenerate background drivers.","The framework's runtime stays practical -- 30 to 70 agents with roughly linear resource growth and only a few LLM invocations per route -- so large-scale diverse traffic tests are feasible."],"supporting_citations":[{"why":"Supplies the CARLA simulator platform, its BEV rendering, and the Leaderboard metrics used in all experiments.","marker":"Dosovitskiy et al. [2017]"},{"why":"Provides the Roach expert policy used to implement the background SimAgents' driving capability layer.","marker":"Zhang et al. [2021]"},{"why":"InterFuser is the representative state-of-the-art AD model primarily tested under HDSim traffic.","marker":"Shao et al. [2023]"},{"why":"ProSim is the LLM-promptable closed-loop traffic simulation baseline used for realism comparison.","marker":"Tan et al. [2024]"},{"why":"INTERACTION dataset supplies real human trajectories for training style classifiers and validating simulated style realism.","marker":"Zhan and et al. [2019]"},{"why":"NMVCCS accident records anchor the claim that style-induced simulated failures align with real-world crashes.","marker":"National Highway Traffic Safety Administration [2024]"},{"why":"Psychological account of individual differences in driving informs the hierarchical style model's structure.","marker":"Elander et al. [1993]"},{"why":"Attention Resource Theory grounds the attentional influence layer's stochastic triggers.","marker":"Kahneman [1973]"},{"why":"Weber-Fechner law is used to maintain temporal consistency of the perceptual adjustments across simulation steps.","marker":"Fechner [1860]"}],"fun_headline_variants":["HDSim: LLM-driven traffic that reveals 68% more self-driving blind spots","Simulated human quirks expose 68% more self-driving failures","Perception-hacked traffic finds 68% more self-driving test failures","LLM-tweaked perception reveals hidden self-driving mistakes","Human-like traffic simulation uncovers 68% more safety-critical errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on the assumption that editing the bird's-eye-view image seen by a competent driving model is a faithful way to reproduce how human cognitive states bias perception, so that the model's resulting actions genuinely mimic human style rather than simply reacting to artificial sensor corruption.","fun_headline_variants_meta":{"raw":{"variants":["HDSim: LLM-driven traffic that reveals 68% more self-driving blind spots","Simulated human quirks expose 68% more self-driving failures","Perception-hacked traffic finds 68% more self-driving test failures","LLM-tweaked perception reveals hidden self-driving mistakes","Human-like traffic simulation uncovers 68% more safety-critical errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3258,"prompt_tokens":925,"completion_tokens":2333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":2238}},"tokens_in":541,"tokens_out":2333,"duration_ms":17889,"temperature":1.0,"reasoning_tokens":2238,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:08:56.535153+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the reported AD models in the same CARLA scenarios with the same style triplets, but replace PMBI's BEV edits with equivalent action-level perturbations matched to the same style descriptors; if the detected failure counts do not drop when the BEV edits are removed, perception mediation is not the causal driver of the reported 68% improvement.","supporting_citations":[],"review_version":2}