{"id":"c696af13-3693-49e8-ac5e-04948b76881e","arxiv_id":"2508.04642","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A simulation-to-real pipeline (HASS synthetic hard cases, scenario-aware prompts, and an image-to-ego geometry encoder) improves an MLLM's open-loop planning on nuScenes, especially in hard scenarios.","lead":"RoboTron-Sim trains a multimodal language model on simulated hard-case driving scenarios from CARLA and then evaluates it on real-world nuScenes data, reporting roughly 50% improvement on challenging planning metrics. It matters because rare, dangerous driving situations are underrepresented in real-world datasets, and simulated data could fill that gap if the transfer works.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Hard-scenario improvement (~50%) rests on a missing Table 2 and no ablation isolating HASS; the central Sim2Real attribution is currently unverifiable.","rationale":"The reader's verdict was CONDITIONAL, citing the missing hard-scenario table, lack of ablations, and ambiguous hard-scenario definition. My stress-test converges on the same conclusion but identifies a slightly different primary weak point: the missing attribution of the improvement to HASS specifically, rather than the model architecture or additional data volume. The paper's strongest claim is explicitly about improving real-world driving via simulated hard cases; to support this, one must show that the simulated hard-case data is the cause of the improvement. The provided evidence (Table 3) shows overall SOTA but does not isolate the HASS contribution. The absence of Table 2 makes the headline hard-case numbers impossible to verify. These concerns do not refute the approach, but they make the central claim currently unverifiable, so the verdict should remain CONDITIONAL pending the missing data and ablations. I agree partially with the reader's weakest_assumption: representativeness of HASS is relevant, but even the most representative simulation would not help if the paper cannot demonstrate that HASS is the causal factor in the observed gains.","tokens_in":10524,"tokens_out":2740,"duration_ms":29313,"concrete_test":"Ask authors to release Table 2 with exact definition of the hard-scenario subset (e.g., which nuScenes scenes/frames, criteria, counts). Then re-run with three training configurations: (1) nuScenes only, (2) nuScenes + HASS, (3) nuScenes + HASS without SPE/I2E (or with SPE/I2E on nuScenes-only). If (1) and (2) are equal, the Sim2Real claim fails; if (2) improves but (3) matches (2), the improvement is from architecture, not HASS.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the ~50% improvement in hard scenarios on nuScenes (Abstract; Sec. 1). The only support is Table 2, which is absent from the manuscript; Table 3 reports only overall nuScenes metrics. Moreover, the evaluation protocol for 'hard scenarios' is not defined (no subset criteria, sample size, or per-category breakdown). Even if the table were provided, the claim that HASS causes the improvement requires ablations: RoboTron-Sim is trained on 28,130 nuScenes + 47,553 HASS samples (Sec. 4.1.1), so gains over baselines could come from the SPE/I2E architecture, the MLLM backbone, or the extra data volume rather than from simulated hard-case content. The paper only reports that VAD with mixed data gains ~1% (Sec. 3.2), not a RoboTron-Sim without-HASS control. Thus the load-bearing inference 'HASS improves real-world hard scenarios' is unsupported by current evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RoboTron-Sim, a multimodal large language model (MLLM) based planning framework that augments real-world nuScenes training data with a CARLA-generated synthetic dataset, HASS, containing 13 hard-case and long-tail scenario categories. Two components are introduced: Scenario-aware Prompt Engineering (SPE), which conditions the model on data provenance (simulation/real) and geographic context, and an Image-to-Ego (I2E) Encoder, which injects camera intrinsic/extrinsic geometry. The model is evaluated on nuScenes open-loop planning, and the paper claims roughly 50% improvement in hard scenarios (48.1% L2, 45.8% collision) over baselines, plus state-of-the-art overall performance.","tokens_in":10736,"tokens_out":3863,"duration_ms":43776,"significance":"If the claims hold, the paper would make a useful contribution by demonstrating that carefully curated simulated hard cases can transfer to real-world MLLM driving, and the SPE/I2E components are well-motivated and clearly described. The HASS dataset and generation pipeline are concrete assets. However, the headline result is not currently verifiable from the submitted manuscript: the supporting table is absent, the hard-scenario evaluation protocol is undefined, and no ablation isolates the contribution of HASS from the extra data volume or the new architectural modules. The claimed 50% improvement is not reflected in the overall nuScenes table, which shows smaller average gains.","major_comments":[{"comment":"The central claim—'~50% improvement' in hard scenarios, specified as 48.1% L2 and 45.8% collision improvements—is not supported by any table in the submitted text. Table 2, which would presumably contain these numbers, is missing entirely. Table 3 reports only overall nuScenes metrics, where RoboTron-Sim improves over OmniDrive by about 33% in L2 (0.84 vs 0.56) and 38% in collision (0.94 vs 0.58) without ego pose, and 30% and 13% with ego pose—not the stated 50%. The 'hard scenarios' evaluation subset is never defined: no selection criteria, sample size, or per-category breakdown. Please restore Table 2 (or an equivalent) and define the hard-scenario subset precisely.","section":"Abstract; Sec. 1 (contributions); Table 3"},{"comment":"No ablation isolates the contribution of HASS. RoboTron-Sim is trained on 28,130 nuScenes + 47,553 HASS samples (Sec. 4.1.1). The only control mentioned is VAD with mixed data, which gains ~1% L2 in HD scenarios (Sec. 3.2); there is no RoboTron-Sim trained without HASS, nor a variant trained on an equal-sized synthetic dataset without hard-case composition, nor ablations removing SPE/I2E. Consequently, the observed gains could stem from the MLLM backbone, the extra training data volume, SPE, or I2E rather than from the specifically hard-case content. Please provide ablations that separate these factors, e.g., nuScenes-only, nuScenes + equal-sized non-hard synthetic data, and nuScenes + HASS without SPE/I2E.","section":"Sec. 4.1.1; Sec. 3.2"},{"comment":"The evaluation protocol is incomplete in the manuscript. Both the dataset description and the metric details are deferred with 'Please refer to the supplementary material for more details,' but the supplementary material is not part of the reviewed manuscript. Without the hard-scenario definition, the trajectory-frequency details, and the exact evaluation split, the reported numbers cannot be reproduced or independently checked. Please include the full protocol in the main text or make the supplementary available for review.","section":"Sec. 4.1.1; Sec. 4.1.2"},{"comment":"No error bars, confidence intervals, or statistical significance tests are reported for any metric. Since the hard-scenario subset is likely small, the claimed 48.1%/45.8% improvements may be based on a handful of scenarios. Please report the number of evaluation scenarios in the hard subset and, if possible, variance across multiple training seeds or by bootstrapping over scenes.","section":"General; Table 3"}],"minor_comments":[{"comment":"Typographical issues: 'V AD' and 'LLaV A-OneVision' appear with inconsistent spacing throughout the text and figures; please standardize to 'VAD' and 'LLaVA-OneVision'.","section":"Throughout"},{"comment":"Reference [22] duplicates [12] (same title/venue), and reference [26] appears twice with different page ranges. Please deduplicate and unify citation entries.","section":"References"},{"comment":"The caption should state that 'Real Scenario' refers to nuScenes and 'Simulated Scenario' to HASS, and clarify whether the counts include both E2D and H2D subsets. This would aid readability.","section":"Table 1"},{"comment":"The claim of being the 'first in-depth investigation' is difficult to verify and unnecessary; please soften to avoid an unsubstantiated novelty assertion.","section":"Sec. 1, contributions"},{"comment":"The paper lacks a limitations and future-work section. In particular, it should discuss the absence of closed-loop evaluation, the open-loop-only nature of the nuScenes benchmark, and potential biases in the hard-scenario selection.","section":"Discussion/Limitations"}],"recommendation":"major_revision","confidential_remarks":"The submitted manuscript is missing Table 2 and the definition of the hard-scenario evaluation subset, which are essential to the paper's central claim. Please ensure that the full manuscript—including all tables and supplementary material—is complete in future submissions, as the current text cannot support the headline result."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper's headline result — roughly 50% improvement on hard nuScenes scenarios — is not verifiable from the manuscript as it stands. Table 2, which should support it, is missing, and there is no ablation that isolates the HASS dataset from the other changes. But the dataset itself and the domain-adaptation recipe (SPE + I2E) are worthwhile and deserve proper referee scrutiny.\n\nWhat's actually new: HASS is a 47,553-sample CARLA dataset covering 13 long-tail categories with deliberately balanced day/night and sunny/rainy splits, built with a Think2Drive teacher and a six-camera rig mirroring nuScenes. The authors also align the coordinate systems between CARLA and nuScenes, which is the kind of unglamorous detail that often breaks sim-to-real transfer. On the modeling side, SPE tells the MLLM whether it is looking at simulation or real data and which city/traffic rules apply, and the I2E encoder injects camera intrinsics/extrinsics through an MLP adapter. These are sensible, concrete ideas. The overall nuScenes numbers in Table 3 are strong: with ego pose, RoboTron-Sim reports avg L2 0.23 m vs OmniDrive's 0.33, and lower collision and boundary rates.\n\nSoft spots, in proportion. The load-bearing claim of ~48% L2 and ~46% collision improvement on hard scenarios is supported only by a table that is not in the manuscript. The evaluation protocol for 'hard scenarios' is also not defined — no subset criteria, no per-category counts. Even if the table appears, the attribution to HASS is not established without ablations: the model is trained on 28k real + 47.5k sim samples, so the gain could come from extra data volume, the SPE/I2E modules, or the MLLM backbone rather than the simulated hard-case content. The paper reports only that VAD with mixed data gains ~1%, which is not a substitute for a RoboTron-Sim-without-HASS control. No error bars or multiple seeds are reported, and no code or data is released. None of this is fatal to the underlying idea; the pieces are plausible. But the current evidence cannot support the headline.\n\nWho it's for: people working on MLLM-based planning and sim-to-real data augmentation. The dataset could be a useful community resource if released. It deserves a serious referee, not a desk reject, but the authors should be asked to supply the missing table and ablations before it is accepted.","headline":"Missing table and ablations undercut the ~50% hard-scenario claim, but the HASS dataset and SPE/I2E adaptation recipe are solid enough for serious peer review.","tokens_in":11290,"tokens_out":2146,"would_cite":false,"duration_ms":23100,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RoboTron-Sim claims that training an MLLM planner on simulated hard cases generated in CARLA, with scenario-aware prompts and camera-geometry conditioning, improves real-world nuScenes open-loop planning by about half in hard scenarios.","keywords":["autonomous driving","multimodal large language models","sim-to-real transfer","synthetic hard cases","open-loop planning","CARLA","nuScenes","end-to-end planning"],"falsifier":"Run a controlled nuScenes experiment with three training sets: real data only, real data plus HASS, and real data plus HASS with the SPE provenance token removed or the I2E encoder disabled. If hard-scenario L2 distance and collision rate do not improve in the second condition over the first, or if removing either adapter does not degrade the result, the paper's claim that simulated hard cases and the proposed alignment mechanisms drive the gain is falsified.","tokens_in":10384,"feed_emoji":"🚗","tokens_out":8401,"duration_ms":96712,"temperature":0.7,"pith_summary":"RoboTron-Sim sets out to prove that rare, dangerous driving scenarios do not have to be collected in the real world to be learned: they can be synthesized in simulation and transferred to real-world planning. The paper builds HASS, a CARLA dataset of 47,553 samples spanning 13 high-risk edge-case categories with balanced day/night and sunny/rainy conditions, and trains a multimodal large language model (MLLM) planner on it together with real nuScenes data. To bridge the sim-to-real gap, it adds scenario-aware prompts that label data provenance and city, plus an Image-to-Ego encoder that feeds each camera's geometry into the model. On the nuScenes validation set, the method reports 48.1% lower L2 distance and 45.8% lower collision rate in hard-to-drive scenarios in open-loop planning (trajectory prediction without a driving loop), while keeping routine-scenario performance. If this holds, synthetic data becomes a practical way to cover long-tail driving events without dangerous data collection.","feed_headline":"Simulated hard-case data cuts real-world driving errors by ~50%","feed_subtitle":"A CARLA-trained driving LLM improves nuScenes hard-scenario L2 distance and collision rate by about half.","key_machinery":"The engine is HASS (Hard-case Augmented Synthetic Scenarios), generated in CARLA with a Think2Drive teacher and a six-camera 900x1600 sensor suite: it balances day/night and sunny/rainy conditions and oversamples 13 long-tail edge-case categories so rare events dominate training. Two adapters carry the Sim2Real argument: Scenario-aware Prompt Engineering (SPE) prepends data-provenance and city tokens so the LLM can treat simulated and real inputs differently, and the Image-to-Ego (I2E) encoder, a two-layer MLP, maps each camera's image-to-ego transformation into embeddings that normalize sensor geometry across domains. The supporting piece is coordinate alignment: CARLA's left-handed frame w","core_discovery":"The paper's central claim is that an MLLM can learn transferable real-world driving skills from simulated hard cases, provided the data is balanced and the model is told how to interpret the simulation. HASS supplies the data: 13 procedurally generated long-tail categories (jaywalking pedestrians, sudden cut-ins, near-collision events) plus routine hard-to-drive cases, built in CARLA by a Think2Drive teacher with a six-camera rig and aligned to nuScenes coordinates. On the model side, Scenario-aware Prompt Engineering tells the LLM whether the input is simulated or real and which city it is in, so its commonsense knowledge can adapt the policy; the Image-to-Ego Encoder injects per-view camer","pith_inferences":["The recipe could become a closed loop: mine failures from a real-world validation set, synthesize matching CARLA scenarios, retrain, and repeat, making edge-case coverage an iterative process rather than a static dataset.","The success of categorical provenance prompts suggests that simulator photorealism may matter less than explicit metadata; a quantitative version of this idea would condition on measured domain similarity instead of city labels.","If the open-loop gains reflect genuine behavioral transfer, closed-loop evaluation on the same hard scenarios should show corresponding improvements in safety-critical metrics."],"forward_implications":["The 13 HASS edge-case categories can be generated at scale, making rare-event coverage a data-generation problem rather than a data-collection problem.","Because the I2E encoder consumes camera parameters directly, planners trained this way should transfer across different sensor rigs without retraining on each camera setup.","Balanced environmental sampling (58.65% day/41.35% night, 48.38% sunny/51.61% rainy) gives the model exposure to conditions that are underrepresented in real datasets.","RoboTron-Sim outperforms prior MLLM planners and traditional planners on open-loop planning on nuScenes, including on collision and boundary metrics."],"supporting_citations":[{"why":"Supplies the simulator in which the HASS hard-case scenarios are generated.","marker":"[10]"},{"why":"Serves as the teacher model that generates HASS data using privileged simulator information.","marker":"[8]"},{"why":"Provides the real-world training and validation data, including the hard-scenario evaluation used for the headline numbers.","marker":"[9]"},{"why":"Supplies the open-loop evaluation protocol (L2 distance, collision rate, boundary violation) and the ego-status baseline.","marker":"[3]"},{"why":"Is the traditional end-to-end baseline that, when trained on mixed real and simulated data, shows only ~1% gains, motivating the MLLM design.","marker":"[49]"},{"why":"Is the MLLM baseline whose generalization is compared with RoboTron-Sim's, showing that naive MLLM transfer does not close the Sim2Real gap.","marker":"[14]"},{"why":"Is a strong MLLM planner baseline that RoboTron-Sim outperforms on nuScenes open-loop planning.","marker":"[25]"}],"fun_headline_variants":["Simulated hard cases improve real-world driving by ~50%","Driving model learns from simulated edge cases, cuts errors by half","RoboTron-Sim uses synthetic hard cases to improve challenging driving","Synthetic hard-case training reduces real-world driving mistakes by 50%"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that the HASS scenarios generated in CARLA by the Think2Drive teacher approximate the visual and behavioral diversity of real-world nuScenes hard cases closely enough that a model given only categorical provenance prompts and camera-geometry conditioning can transfer what it learns; if that representativeness fails, the reported gains may reflect dataset statistics rather than genuine sim-to-real transfer.","fun_headline_variants_meta":{"raw":{"variants":["Simulated hard cases improve real-world driving by ~50%","Driving model learns from simulated edge cases, cuts errors by half","RoboTron-Sim uses synthetic hard cases to improve challenging driving","Synthetic hard-case training reduces real-world driving mistakes by 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00092,"raw_usage":{"total_tokens":3790,"prompt_tokens":757,"completion_tokens":3033,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":501,"completion_tokens_details":{"reasoning_tokens":2958}},"tokens_in":501,"tokens_out":3033,"duration_ms":24224,"temperature":1.0,"reasoning_tokens":2958,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:49:25.498965+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a controlled nuScenes experiment with three training sets: real data only, real data plus HASS, and real data plus HASS with the SPE provenance token removed or the I2E encoder disabled. If hard-scenario L2 distance and collision rate do not improve in the second condition over the first, or if removing either adapter does not degrade the result, the paper's claim that simulated hard cases and the proposed alignment mechanisms drive the gain is falsified.","supporting_citations":[{"cited_title":"Carla: An open urban driving simulator,","cited_arxiv_id":null,"evidence_quote":"Supplies the simulator in which the HASS hard-case scenarios are generated."},{"cited_title":"Think2drive: Efficient reinforcement learning by thinking with latent world model for autonomous driving (in carla-v2),","cited_arxiv_id":null,"evidence_quote":"Serves as the teacher model that generates HASS data using privileged simulator information."},{"cited_title":"nuscenes: A multimodal dataset for autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Provides the real-world training and validation data, including the hard-scenario evaluation used for the headline numbers."},{"cited_title":"Is ego status all you need for open-loop end-to-end au- tonomous driving?","cited_arxiv_id":null,"evidence_quote":"Supplies the open-loop evaluation protocol (L2 distance, collision rate, boundary violation) and the ego-status baseline."},{"cited_title":"Vad: Vector- ized scene representation for efficient autonomous driving,","cited_arxiv_id":null,"evidence_quote":"Is the traditional end-to-end baseline that, when trained on mixed real and simulated data, shows only ~1% gains, motivating the MLLM design."}],"review_version":1}