{"id":"25cd2839-6661-4801-b7de-73f870f5c264","arxiv_id":"2506.08963","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Trajectory prediction models that pass standard accuracy metrics still generate red-light violations, illegal stops, and near-collisions when evaluated online in a microsimulator, and new intersection-specific metrics expose these failures.","lead":"The paper proposes new metrics to catch unsafe driving behavior in AI traffic models at intersections, such as running red lights and stopping in the middle of the road. It shows that a generative model trained on simulated traffic can produce rule-breaking trajectories even when its usual error scores look fine.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'ideally-behaved trajectories' premise is never tested: the proposed metrics are not reported for the SUMO training data, so the model's rule-breaking could be faithful learning rather than a generative-model failure.","rationale":"I agree with the reader's weakest assumption: the evaluation ground truth is the SUMO training data, and its rule-compliance is untested. The load-bearing role: the abstract's 'despite' formulation contrasts ideal inputs and low reconstruction errors with observed violations. If the input is not ideal, the contrast is not established. The concrete test is straightforward and requires no new methodology: apply the paper's own metrics to its own training data and report the error metrics. I also noticed that the per-cluster vehicle counts differ between Table 2 and Table 3 (e.g., T on EBT: 57 vs 27) despite both claiming a total of 1026 vehicles; this suggests either different simulation runs or an inconsistency, and warrants a check, though it is secondary to the main premise. Since the reader already conditions acceptance on adding missing metrics/data, my read does not change the verdict.","tokens_in":8545,"tokens_out":7615,"duration_ms":76538,"concrete_test":"Run the metric pipeline of Section 3 on the 36-hour SUMO training set (or a held-out split) and report the violation counts for red-light, mid-intersection stoppage, pre-stopbar stoppage, and TTC encounters. Also report ADE/minADE, FDE/minFDE, and KDE-NLL on the same evaluation used for Tables 2 and 3. If the training-data red-light violation rate is near zero and the ADE/FDE values are low, the central claim is supported. If the training data already contains comparable violation rates, the 'ideally-behaved' premise is refuted and the claim needs to be restated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (Abstract) is that 'despite using ideally-behaved trajectories as input, and achieving low trajectory reconstruction errors, the generated trajectories show behaviors that break traffic rules.' The first premise is asserted without evidence. Section 4 describes a 36-hour SUMO dataset, calibrated to a real intersection, but never reports the Section 3 metrics on that dataset. SUMO is a microsimulator with its own car-following and signal-logic models; vehicles can enter the intersection on yellow and be in the box when the phase changes, and actuated signal timing can produce boundary cases. If the training trajectories already contain a non-negligible red-light violation rate, the model's 14.8% rate in Table 2 may simply reflect the empirical distribution it was trained on, not a failure to internalize traffic rules. The phrase 'ideally-behaved' would then be false, and the 'despite' part of the central claim would collapse. The second premise—'low trajectory reconstruction errors'—is also unsupported: Section 4 only says training reached 'reasonable accuracy based on pre-existing metrics,' with no ADE/FDE or KDE-NLL numbers reported anywhere. Without these two baselines, the paper has not demonstrated that low reconstruction error coexists with rule violations; it has only demonstrated that this particular model, on this synthetic dataset, produces some rule-violating trajectories.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an evaluation framework for deep generative vehicle-trajectory models at signalized intersections, adding four intersection-aware metrics (red-light violation, mid-intersection stoppage, pre-stopbar stoppage, and time-to-collision encounters) to standard reconstruction-based metrics such as ADE/FDE/KDE-NLL. The authors train a CVAE-based multi-vehicle trajectory model with an intersection position encoder and signal timing input on 36 hours of SUMO-simulated data calibrated to a real intersection, then evaluate the model online in SUMO via TRACI on 4,000 seconds of unseen traffic for 1,026 vehicles. The central claim is that, despite using 'ideally-behaved trajectories' as input and achieving low trajectory reconstruction errors, the generated trajectories exhibit traffic-rule violations, and that the proposed metrics reveal these failures.","tokens_in":8827,"tokens_out":4422,"duration_ms":43507,"significance":"If substantiated, the paper's central message is important: reconstruction-based metrics commonly used in trajectory prediction are insufficient to certify safety or rule compliance at signalized intersections, and traffic-engineering-specific metrics can expose failures that ADE/FDE would miss. The proposed metric set is a useful contribution, and the simulation-in-loop evaluation protocol is a promising direction. The paper also demonstrates a concrete architectural improvement: the intersection position encoding appears to nearly eliminate mid-intersection stoppages relative to the authors' baseline. However, the current manuscript does not provide the evidence needed to establish the central claim: the 'ideally-behaved' premise is asserted rather than measured, and no reconstruction-error numbers are reported. These omissions are load-bearing and require additional experiments and reporting.","major_comments":[{"comment":"The abstract's first premise, 'ideally-behaved trajectories as input,' is never verified. The paper does not report the Section 3 metrics on the training dataset. SUMO is a microsimulator with its own car-following, lane-changing, and signal-logic models; vehicles can enter the intersection on yellow and be inside the box when the phase changes, and actuated signal control can produce boundary cases. If the training trajectories already contain red-light violations or other rule-breaking behavior, the 14.8% red-light violation rate in Table 2 may simply reflect the empirical training distribution rather than a generative-model failure. Please report the four proposed metrics on the training and validation trajectories, and justify concretely that the training data is indeed 'ideally behaved.'","section":"Section 4, dataset description"},{"comment":"The paper's second premise, 'achieving low trajectory reconstruction errors,' is unsupported. Section 4 states only that the model was 'trained to convergence (with reasonable accuracy based on pre-existing metrics in Table 1),' but no ADE, FDE, or KDE-NLL numbers appear anywhere in the manuscript. Without these values on a held-out test set, the 'despite' part of the central claim is not demonstrated. Please report the standard reconstruction metrics for both models, ideally for the same online evaluation window used in Tables 2 and 3.","section":"Section 4, training and evaluation"},{"comment":"The coupling between the learned model and SUMO is under-specified. The paper says the first 2 seconds of ground-truth trajectory are provided and then 'the model predicts the rest of the trajectory till the vehicle exits the simulation,' but it does not explain how the predicted trajectories are injected through TRACI, whether they override SUMO's own vehicle dynamics, how conflicts between the model's planned position and SUMO's car-following/right-of-way logic are resolved, or how often predictions are updated. This matters because red-light violations and TTC encounters could be artifacts of the trajectory-injection mechanism rather than properties of the generated trajectories. Please provide a precise description of the closed-loop protocol, including pseudocode or a state diagram.","section":"Section 4, online evaluation protocol"},{"comment":"The per-cluster counts differ between the Improved Model and Baseline tables even though both report a total of 1,026 vehicles. For example, 'T on EBT' is 57 in Table 2 but 27 in Table 3, 'T on EBTR' is 61 versus 91, and 'T on WBTR' is 79 versus 54. Since each vehicle belongs to exactly one trajectory cluster, the cluster totals should be identical across the two models if the same simulation run is used. Please explain whether the two models were evaluated on different traffic draws, and if so, how the comparison in Section 4 is controlled; otherwise, correct the tables.","section":"Tables 2 and 3"},{"comment":"Several metric thresholds are defined without sensitivity analysis: the 5 mph / 2 s criterion for mid-intersection stoppage in Section 3.2, the 0.5 m/s queue-discharge rate in Section 3.3, and the 4 s TTC threshold in Section 3.4. Because these metrics are the paper's main contribution, the reported violation rates may depend strongly on these choices. Please either perform a sensitivity analysis over reasonable threshold ranges or provide stronger empirical justification for the chosen values, and state whether the qualitative conclusions (e.g., 'Improved Model virtually eliminates Mid-Intersection Stoppage') are robust to those choices.","section":"Section 3, metric thresholds"}],"minor_comments":[{"comment":"References [8] and [9] appear to be the same paper (Huang et al., 'Multimodal trajectory prediction: A survey' and 'A survey'); please merge or distinguish them.","section":"References"},{"comment":"There is a typo in the sentence introducing X and Y: 'HereX consists of...' and 'HereY consists of...' are run together without punctuation.","section":"Section 2.1"},{"comment":"The statement 'right turn on red is often allowed at many intersections... in our experiments, this sort of right turn on red light is not allowed' should clarify how SUMO's configuration enforces this prohibition, since the simulation setup is central to the red-light violation metric.","section":"Section 3.1"},{"comment":"The KDE-NLL expression would benefit from a definition of the kernel density estimate and the notation P(Y_i^t | KDE(...)); as written it is not self-contained.","section":"Table 1"},{"comment":"No code or data availability statement is included. For an evaluation-framework paper, releasing the analytics tool and configuration files would substantially improve reproducibility.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The missing baselines are fixable within the paper's scope: the authors need to report the proposed metrics on the training data and give explicit reconstruction-error numbers, plus clarify the SUMO coupling. The cluster-count discrepancy in Tables 2 and 3 is a data-integrity issue that must be resolved. The paper is not ready for acceptance, but the core idea is sound and worth pursuing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know: this paper introduces four intersection-specific evaluation metrics (red-light violation, mid-intersection stoppage, pre-stopbar stoppage, TTC encounters) and evaluates a generative trajectory model online in a SUMO microsimulator. That is a genuinely new and useful contribution for traffic authorities who need to screen models for safety-critical behaviors. The metrics are sensible, and the stratified results in Tables 2/3 show that the model does produce rule-breaking trajectories.\n\nWhat it does well: the online evaluation setup—freezing the model and unrolling trajectories in a live SUMO scenario—is a step beyond static test-set benchmarking, and the metrics target real concerns (red-light compliance, stopping in the box, sluggish queue discharge) that ADE/FDE and similar distance-based metrics ignore. The idea that low reconstruction error can coexist with rule violations is worth testing.\n\nThe soft spot is the central claim. The abstract says 'despite using ideally-behaved trajectories as input, and achieving low trajectory reconstruction errors.' Neither premise is shown. Section 4 reports 'reasonable accuracy based on pre-existing metrics' but no ADE/FDE/KDE-NLL numbers appear anywhere. And the training data's own violation rate is never reported. SUMO's calibrated simulation is not guaranteed to be 'ideally-behaved'—vehicles can enter on yellow and be in the intersection when the phase changes, and actuated signals create edge cases. If the training data already contains a 15% red-light rate, the model's 14.8% may just be faithful learning, not a generative-model failure. The stress-test note has this right, and it lands directly on the abstract.\n\nAlso worth noting: the baseline is the authors' own earlier model; Trajectron++ is cited but not compared. No code or data is released, so the claims are not independently checkable. These are addressable, not fatal. A revision should add the offline metrics, report the training-data violation counts, describe how the model's trajectories override or interact with SUMO's own vehicle logic, and release code and data.\n\nThe paper deserves a serious referee because the metrics and the evaluation protocol are real contributions, and the central question—do reconstruction metrics miss safety-critical violations?—matters. A reviewer can push for the missing evidence. I would bring it to a reading group if the offline numbers and training-data context get added, but as it stands the conclusion is asserted rather than demonstrated.\n\nRecommendation: engage with it, but require the missing baselines before taking the central claim at face value.","headline":"A useful set of intersection-specific safety metrics, undermined by missing baseline numbers and an untested 'ideally-behaved' training-data claim.","tokens_in":9356,"tokens_out":1777,"would_cite":false,"duration_ms":17883,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a generative vehicle trajectory model can score low on reconstruction error while still producing trajectories that run red lights, stop mid-intersection, and create low-time-to-collision encounters, so standard…","keywords":["trajectory prediction","generative models","traffic intersection safety","red-light violation","microsimulation evaluation","conditional variational autoencoder","time-to-collision","traffic engineering metrics"],"falsifier":"Re-run the red-light and stoppage metrics on the 36-hour training dataset itself; if that \"ideally behaved\" data already contains red-light crossings or slow green-light starts, the reported rule-breaking by the learned model could come from the data or the evaluation harness instead of the model, while a clean training set with persistent violations would support the paper's central claim.","tokens_in":8331,"feed_emoji":"🚦","tokens_out":12270,"duration_ms":125769,"temperature":0.7,"pith_summary":"The paper asks whether standard reconstruction-based metrics can certify that a generative trajectory model is safe to use at a signalized intersection. It trains a multi-vehicle trajectory forecasting model on a large dataset drawn from a calibrated microsimulation of a real intersection, then runs the trained model live in the simulator under traffic conditions it has not seen. The central finding is that the model achieves low trajectory reconstruction error yet produces frequent rule-breaking behavior: about 15 percent of generated vehicle trajectories run red lights, around 20 percent show pre-stopbar stoppage, and the runs record over a thousand low-time-to-collision encounters. To make these failures visible, the paper introduces four intersection-aware metrics: red-light violation, mid-intersection stoppage, pre-stopbar stoppage, and time-to-collision encounters. If the finding is right, trajectory prediction models for intersections need to be evaluated for traffic-rule compliance and safety, not only for positional accuracy.","feed_headline":"Generated vehicles run red lights despite low trajectory error","feed_subtitle":"A live microsimulation exposes rule-breaking that standard reconstruction metrics miss.","key_machinery":"The argument rests on two pieces. The first is the generative model itself: a conditional variational autoencoder with an LSTM history encoder, a directed-graph interaction encoder over vehicles within an attention radius, and an intersection-aware position embedding that locates each vehicle relative to a static endpoint in one of three trajectory regions, together with the traffic-signal state. The second is a set of four evaluation metrics that translate rule-breaking into countable events: a red-light violation is counted when a vehicle crosses the stop bar during a red phase, mid-intersection stoppage when a vehicle stays nearly stopped inside the crosswalk-bounded area for at least two seconds, pre-stopbar stoppage when a vehicle fails to discharge from a queue at a low-end rate during green, and time-to-collision encounters when the simulator's conflict module reports a value below four seconds. These metrics, computed from live simulation logs rather than from pre-recorded test error, are what expose the gap between reconstruction accuracy and safety.","core_discovery":"On its own terms, the paper establishes a mismatch between predictive accuracy and rule compliance in generative vehicle trajectory models. The authors train a conditional variational autoencoder that predicts multi-modal future trajectories from vehicle histories, neighbor interactions, traffic-signal state, and a new intersection-aware position encoding that references region endpoints. They then unroll predictions for 1,026 vehicles through a microscopic traffic simulator, using two seconds of ground-truth history as the initial condition, and count four types of traffic-engineering violations in the simulation logs. Both the improved model and the baseline violate traffic rules at rates that standard error metrics would not flag, with the improved model, for example, producing 152 red-light violations (14.8 percent) and 1,047 conflict encounters. Their conclusion is that low reconstruction error does not imply safe or legal behavior and that the proposed metrics provide a way to detect this.","pith_inferences":["If the same mismatch appears in other generative trajectory models, the practical standard for approving such models would need to include explicit rule-compliance checks rather than reconstruction error.","The four metrics could be computed directly on real intersection video or LiDAR trajectory logs, which would also test whether the violations are a simulator artifact or a property of the learned model.","A natural next step is to make these violation counts differentiable or part of the training objective, so that models are optimized for compliance instead of only for positional accuracy.","The cluster-level pattern, with red-light violations concentrated on particular approaches and turning movements, suggests that encoding the signal-phase and intersection geometry more carefully could remove a large share of the violations."],"forward_implications":["A trajectory model can pass standard error-based evaluation while frequently violating traffic signals, so accuracy alone should not be treated as a safety certificate.","The four metrics give traffic authorities a concrete checklist for screening generative models before deployment at a signalized intersection.","Adding signal-state and intersection-geometry information to a trajectory model removes dangerous mid-intersection stops but does not by itself fix red-light compliance or collision risk.","Live microsimulation evaluation under unseen traffic conditions reveals failures that a pre-recorded test set cannot expose.","Because the training inputs are described as ideally behaved, the reported violations count as behaviors introduced by the model rather than inherited from the data."],"supporting_citations":[{"why":"Supplies the base trajectory-prediction architecture and the simulation-in-loop unrolling scheme this work extends with intersection encodings.","marker":"[18]"},{"why":"Provides the probabilistic multi-agent forecasting baseline whose conditional variational formulation the model builds on and compares against.","marker":"[15]"},{"why":"Supplies the conditional variational autoencoder objective used for training the multi-modal trajectory distribution.","marker":"[16]"},{"why":"Provides the graph-structured interaction encoding and node future encoder used in the model.","marker":"[11]"},{"why":"Supplies the conditional variational autoencoder mechanism used to sample plausible future trajectories.","marker":"[13]"},{"why":"Gives the queue-discharge rate used to define a pre-stopbar stoppage violation.","marker":"[17]"},{"why":"Provides the four-second time-to-collision threshold used to flag conflict encounters.","marker":"[14]"},{"why":"Provides the movement-data library used to compute the paper's metrics from simulation logs.","marker":"[4]"}],"fun_headline_variants":["Low error doesn't stop AI traffic from running reds","Trajectory models ace accuracy, fail traffic rules","Generative traffic models break rules despite low error","AI vehicles run red lights with low prediction error","Traffic AI: accurate but lawless in simulation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the microsimulation that produced the training data is itself realistic and rule-compliant, so every violation counted in the generated trajectories can be attributed to the learned model rather than to artifacts in the simulator or violations already present in the training data.","fun_headline_variants_meta":{"raw":{"variants":["Low error doesn't stop AI traffic from running reds","Trajectory models ace accuracy, fail traffic rules","Generative traffic models break rules despite low error","AI vehicles run red lights with low prediction error","Traffic AI: accurate but lawless in simulation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000153,"raw_usage":{"total_tokens":1198,"prompt_tokens":924,"completion_tokens":274,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":199}},"tokens_in":540,"tokens_out":274,"duration_ms":3979,"temperature":1.0,"reasoning_tokens":199,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:58:14.214744+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the red-light and stoppage metrics on the 36-hour training dataset itself; if that \"ideally behaved\" data already contains red-light crossings or slow green-light starts, the reported rule-breaking by the learned model could come from the data or the evaluation harness instead of the model, while a clean training set with persistent violations would support the paper's central claim.","supporting_citations":[{"cited_title":"Conditional Variational Autoencoder for Neural Machine Translation","cited_arxiv_id":"1812.04405","evidence_quote":"Supplies the conditional variational autoencoder mechanism used to sample plausible future trajectories."},{"cited_title":"Pamukkale University Journal of Engineering Sciences27(06 2020)","cited_arxiv_id":null,"evidence_quote":"Provides the four-second time-to-collision threshold used to flag conflict encounters."},{"cited_title":"GI_Forum – Journal of Geographic Information Sci- ence7(1), 54–68 (2019)","cited_arxiv_id":null,"evidence_quote":"Provides the movement-data library used to compute the paper's metrics from simulation logs."}],"review_version":1}