{"id":"55e66a40-2de3-4fff-a3e3-1a237dc2ac99","arxiv_id":"2506.14100","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hierarchical real-world testing platform for VLM-integrated autonomous driving is demonstrated on a by-wire vehicle, with a cloud GPT-4 agent making high-level decisions inside an Autoware stack.","lead":"This paper introduces a vehicle-mounted test platform that lets researchers plug different vision-language models into a classical autonomous driving stack and run repeatable real-world scenarios on a closed track. A case study with a cloud-hosted GPT-4 agent shows the decision loop can run at low latency, though the validation metrics are mostly self-consistency checks rather than independent ground truth.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 5.1's accuracy metric scores deterministic middleware modules against outputs derived from the same inputs, and the VLM under test never appears in Table 2, so the central validation claim is unsupported.","rationale":"The reader and I identify the same weakest assumption: the accuracy metric in Section 5.1 is circular for the modules that carry the reported near-100% correctness, and no independent ground truth is provided for the VLM's actual decisions. I read the paper in good faith and credit the real engineering content — a by-wire Lexus with Autoware, GPT-4, Whisper, and closed-track operation is a plausible physical testbed, and the latency measurements demonstrate that the middleware itself can run at the stated rates. However, the platform's stated purpose is to validate VLM-based driving systems, and that purpose requires a correctness oracle for the VLM's behavior selection, scene understanding, and control-parameter choices. The paper does not provide such an oracle; Table 2 evaluates only peripheral deterministic modules. This is not a disagreement with external consensus but an internal validity problem in the evaluation section. Since the reader already assigned CONDITIONAL on exactly this basis, my stress-test does not move the verdict; it reinforces it with a concrete independent-annotation check that would settle whether the accuracy claim is genuinely informative.","tokens_in":22118,"tokens_out":4265,"duration_ms":48786,"concrete_test":"Re-score the recorded case-study prompts independently: have two annotators, blind to the paper's expected outputs and to GPT-4's responses, label the correct behavior and the correct direction of control-parameter adjustment for each logged prompt using only the raw camera image, vehicle state, and passenger command. Then compute annotator-annotator agreement, annotator-VLM agreement, and agreement between all annotators and the paper's 'expected outputs' (e.g., Cohen's kappa). If the paper's expected outputs exactly coincide with the VLM outputs by construction, or if independent annotators do not agree with the VLM substantially above chance, then the Section 5.1 accuracy claim is self-referential and the validation conclusion should be weakened to a demonstration of middleware consistency and latency.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the platform can 'test and validate VLM-based driving systems' in controlled, reproducible real-world scenarios. That claim requires a meaningful correctness signal for the VLM's decisions. Section 5.1 defines accuracy as 'comparing the generated outputs with the expected outputs based on the input messages,' and Table 2 reports near-100% correctness for eight modules. But none of those eight modules is the VLM: the Strategic Driving Intelligence Layer — the system under test — is not scored. The scored modules are deterministic reformatters (Prompt Generation Interface, Action Interface, Driving Behavior Selection, Motion Control Refinement) or extractors of data already produced by the classical stack (Vision Perception Aggregator, Localization State Adapter, Planning & Navigation Acquisition). For those modules, the 'expected output' is the structured representation the module itself was designed to produce from the same input, so near-perfect accuracy largely measures formatting and aggregation consistency, not whether the VLM chose a safe or correct behavior. The speech processor's 91.78% is the only result with an essentially independent ground truth (the transcribed command). Consequently, the conclusion that the framework 'maintains high accuracy' and supports VLM validation is not established: the key correctness question — whether GPT-4's selected behaviors, control parameters, and scene interpretations are correct in each scenario — is never directly evaluated. This gap is load-bearing because the paper's contribution as a validation platform depends on distinguishing correct from incorrect VLM decisions; without an external oracle or human ground-truth labels for VLM outputs, the platform can only demonstrate that messages are moved, formatted, and executed with low latency. Section 4.2's two trips per environment further limit the evidence, but the circular accuracy metric is the more fundamental issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a hierarchical real-world test platform for VLM-integrated autonomous driving, comprising a modular low-latency on-vehicle middleware, a clearly separated perception-planning-control architecture that can host VLM-based and conventional modules, and a configurable closed-track scenario suite. The authors describe a case study on a by-wire modified Lexus using Autoware as the backbone and GPT-4 as the VLM, and they report latency, accuracy, and computational-resource measurements across highway, intersection, and parking-lot scenarios. The central claim is that the platform can test and validate VLM-based driving systems in controlled, authentic, and reproducible real-world conditions prior to road deployment.","tokens_in":22330,"tokens_out":3654,"duration_ms":34265,"significance":"If substantiated, the platform would fill a genuine gap: simulation-based and logged-data benchmarks cannot provide controllable closed-loop real-world evaluation of VLM-integrated driving. The paper's strengths are its real-vehicle deployment details, the concrete hardware and middleware descriptions, and the independently measured latency and resource-utilization figures, which are credible and useful to the community. However, the validation of VLM decision correctness is currently missing, and the accuracy metric used is largely circular. Because the central claim is about testing and validating VLM-based driving systems, the current evidence supports only the claim that classical pipeline modules run with low latency and small footprint, not that the VLM's decisions or scene understanding are correct. The platform concept is valuable, but the paper's validation falls short of its stated goal.","major_comments":[{"comment":"The accuracy metric is defined as 'comparing the generated outputs with the expected outputs based on the input messages.' For the Prompt Generation Interface, Action Interface, Driving Behavior Selection, and Motion Control Refinement, the expected output is essentially a deterministic reformatting or relay of the same structured input data, so near-100% accuracy is largely tautological and does not indicate whether the VLM selected a safe or correct behavior. Table 2 does not score the VLM itself, and the only accuracy result with an independent ground truth is the Speech Command Processor (91.78%, against the transcribed command). Consequently, the conclusion in Section 6 that the framework 'maintains high accuracy' and can 'validate VLM-integrated autonomous driving' is not established. Please provide an independent correctness signal for VLM decisions and scene understanding, such as expert annotations, an established benchmark, or a quantitative comparison against a rule-based or classical baseline.","section":"Section 5.1, Section 5.2, Section 6"},{"comment":"Each scenario is demonstrated with only two trips (e.g., highway: 'traffic too slow' vs. 'drive safely'; intersection: 'catch a flight' vs. 'keep safe'), and no repeated runs are reported. The manuscript claims 'reproducible' testing and 'robust experimentation under diverse conditions,' but with two trials per scenario and no variance or error analysis, these claims are not supported. Please report multiple repetitions per scenario and condition, with per-trial results and summary statistics, and state the total number of runs in the validation results.","section":"Section 4.2, Section 5.2"},{"comment":"The reported latency and resource-utilization numbers are absolute and lack any comparison baseline. To demonstrate that the platform enables meaningful evaluation of VLM-integrated systems, the case study should compare the VLM-enabled pipeline against a comparable conventional autonomy stack (e.g., the same Autoware setup without the VLM, or a simple rule-based decision maker). Without such a baseline, one cannot determine whether the VLM improves, degrades, or leaves unchanged the driving performance, and the claim that the framework can 'validate' VLM-based driving systems remains unsupported.","section":"Section 5.2"}],"minor_comments":[{"comment":"The heading 'Tesing Vehicle Architecture' contains a typo and should read 'Testing Vehicle Architecture.'","section":"Section 3.1"},{"comment":"Section 5.2 reports accuracy numbers (mostly 100%, and 91.78% for the Speech Command Processor) in the text, but Table 2 shows only latency and resource utilization; add an accuracy column with per-module values and explicitly describe how each accuracy value was computed.","section":"Section 5.2, Table 2"},{"comment":"The subsection heading 'Time Effeciency' should be 'Time Efficiency.'","section":"Section 5.1"},{"comment":"The latency for the Speech Command Processor is reported inconsistently: Table 2 lists 0.24 ms, the text mentions 167.95 ms for the full end-to-end speech recognition process, and Section 4.2 states command processing times from 0.5 s to 1.9 s; please clarify which quantity is being reported in each place.","section":"Section 4.2.3, Section 5.2"},{"comment":"The ACM Reference Format and copyright notice cite 2018 and ACM Transactions on Internet Things, which does not match the current arXiv submission date or venue; update these fields.","section":"Title page"},{"comment":"The driving behavior set B is reported as [overtake, yield, following] for the highway scenario and later for the intersection scenario has only [yield, following]; make the scenario definitions consistent across the case study.","section":"Section 4.2.1, Section 4.2.2"}],"recommendation":"major_revision","confidential_remarks":"The circularity concern is real and is the core issue: the validation section does not score the VLM, so the paper's headline claim about validating VLM-based driving systems is not yet supported. This is fixable with additional experiments (independent ground truth, repeated trials, and a baseline), so I recommend major revision rather than rejection. The platform infrastructure itself appears sound and could be a useful contribution if the evaluation is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTwo things you should know. First, this is a real, drivable VLM-in-the-loop platform: a by-wire Lexus running Autoware with ChatGPT-4 issuing high-level actions on a closed track, with per-module latencies reported. That part is genuine and useful. Second, the paper's central validation claim — that the platform validates VLM decisions with \"high accuracy\" — collapses under inspection, because the accuracy metric never scores the VLM. Section 5.1 defines accuracy as comparing module outputs with expected outputs derived from the same input messages, and Table 2 reports near-100% correctness for eight middleware modules. None of them is GPT-4. The scored modules are deterministic reformatters (Prompt Generation Interface, Action Interface, Driving Behavior Selection, Motion Control Refinement) and aggregators of data already produced by the classical stack (Vision Perception Aggregator, Localization State Adapter, Planning & Navigation Acquisition). For those, the expected output is the structured representation the module itself was designed to produce, so near-perfect correctness is largely tautological. The one semi-independent signal is the Speech Command Processor at 91.78%, which at least has a ground-truth text string.\n\nWhat is new: the hierarchical middleware architecture, the separation of strategic, prompt/action, information/execution, autonomy, and hardware layers, and the reusable scenario catalog (highway, intersection, parking lot with weather, traffic, and passenger-command variations) deployed on an actual vehicle. The group has previously reported field experiments (refs 14, 15), so this is an extension of that program rather than a from-scratch result. That is fine, but it means novelty is incremental.\n\nThe soft spots beyond the circular metric: exactly two trips per environment, no baseline against existing evaluation methods, no released code or data, and the printed latencies for the Action Interface (2.48e-3 ms) are implausibly small, suggesting a measurement or unit issue. Also, the \"less than 20 ms\" claim in the conclusion refers to the middleware, not to VLM reasoning (cloud calls are 3 s apart).\n\nWho is this for: researchers building or using real-world testbeds for VLM driving. The architecture description alone is worth a read. But as a validation paper, it needs major work: independent labels or an oracle for VLM decisions, at least one baseline comparison, more trials, and artifact release. I'd send it to peer review — a serious referee could push it into a solid platform paper — but I would not accept it as is.\n\nRecommendation: major revision with real validation of the VLM's decisions, not just the middleware.","headline":"A genuinely useful real-world testbed description for VLM-integrated driving, but the accuracy metric never scores the VLM, so the central validation claim does not land.","tokens_in":23032,"tokens_out":2630,"would_cite":true,"duration_ms":25603,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A hierarchical real-world test platform evaluates VLM-equipped vehicles in repeatable closed-track scenarios before road deployment.","keywords":["vision-language models","autonomous driving","real-world testing","closed-loop evaluation","hierarchical architecture","validation platform","domain shift","modular middleware"],"falsifier":"Run the recorded scenario videos past independent human expert drivers and ask them to label the correct maneuver at each decision point; if the VLM's selected behavior agrees with the experts in materially fewer than the reported near-universal cases, the validation would be measuring self-consistency, not driving competence.","tokens_in":21862,"feed_emoji":"🚗","tokens_out":6466,"duration_ms":63284,"temperature":0.7,"pith_summary":"The paper is trying to establish that VLM-integrated autonomous driving can and should be validated as a complete physical system, not just in simulators or on logged datasets. It proposes a hierarchical real-world platform in which a vision-language model sits on top of a modular perception-planning-control stack and makes driving decisions that are executed by a real by-wire vehicle on a closed test track. The platform's value claim is that it combines the controllability of simulation, with configurable road geometry, road conditions, weather, and traffic participants, with the authenticity and closed loop of physical driving. A case study on highway, intersection, and parking-lot scenarios reports that the loop can run with sub-20 ms message latency and high module-level correctness, which the paper offers as evidence that such a testbed is practical before road deployment.","feed_headline":"Closed-track platform tests VLM self-driving before the road","feed_subtitle":"Modular vehicle stack stages repeatable highway, intersection, and parking scenarios for any vision-language model.","key_machinery":"The load-bearing object is the hierarchical vehicle architecture, whose layers separate strategic intelligence (the VLM) from prompting, information processing, autonomy, and hardware. The mechanism that carries the argument is the structured driving-state vector $V_s$ and action vector $V_a$: the middleware summarizes perception feed, vehicle state, planned behaviors, and passenger commands into a prompt, the VLM returns a high-level action, and the execution layer validates it against safety constraints before adjusting control parameters and executing scripts. The second half of the machinery is the scenario suite, where road geometry, road conditions, weather, and traffic participants are varied on a closed track, which is the paper's answer to repeatability and experimental control.","core_discovery":"The paper claims that VLM-integrated autonomous driving can be tested as a whole, in the physical world, before deployment: its platform replaces the usual choice between closed-loop simulation and open-road testing with a closed track where road geometry, road conditions, weather, and traffic participants are configurable. The vehicle side is organized into stacked layers, from the VLM as strategic intelligence through a prompting/action interface, an information-processing/execution layer that builds a driving-state vector and turns VLM actions into executable scripts, a modular autonomy layer, and the hardware layer, so the VLM's decisions close the loop through real sensors and actuators. A case study with a by-wire SUV, a cloud-hosted VLM, and an open-source autonomy stack runs the vehicle through highway, intersection, and parking-lot scenarios under varied conditions, reporting sub-20 ms message latencies and near-universal module-level correctness.","pith_inferences":["A direct extension would run the same scenario suite across several VLMs and report inter-model decision agreement; the paper tests only a single VLM.","The self-referential accuracy metric means that the high-correctness claim should be read as a wiring and formatting check; an independent human-annotated ground truth for scene understanding would be needed to support driving-competence claims.","The same harness could probe causal grounding, for example by removing the camera image from the prompt and observing whether the selected behavior changes; the platform already exposes the prompt-construction point where such ablations are easy.","Closed-track repeatability makes it feasible to measure response distributions rather than single runs, giving statistical confidence for rare-case behaviors."],"forward_implications":["Different VLMs can be swapped into the strategic intelligence layer and compared under identical, repeatable real-world conditions.","The separated perception-planning-control interfaces let one component be replaced while the rest of the stack is held fixed, isolating where an error enters the system.","Because scenarios are configurable, staged near-crash and low-visibility conditions can be repeated many times to check decision consistency.","The reported low latencies indicate that a cloud-hosted VLM can close the decision loop in a physical vehicle without breaking real-time constraints.","The platform supports controlled study of domain shift by varying weather, road condition, and traffic density on the same closed track."],"supporting_citations":[{"why":"supplies the open-source modular autonomy backbone used as the Autonomous Driving Layer.","marker":"[4]"},{"why":"supplies the cloud-hosted VLM whose decisions are under test.","marker":"[36]"},{"why":"provides the object-detection perception results bundled into the VLM prompt.","marker":"[53]"},{"why":"provides the LiDAR scan-matching localization that feeds the vehicle-state vector.","marker":"[5]"},{"why":"provides the speech-recognition engine for the passenger command channel.","marker":"[11]"},{"why":"represents the simulation approach whose sim-to-real gap motivates the real-world platform.","marker":"[16]"},{"why":"represents the logged-dataset benchmark that cannot support closed-loop repeatable scenario changes.","marker":"[46]"},{"why":"demonstrates prior on-board VLM motion-control validation that the platform extends to full-stack evaluation.","marker":"[15]"}],"fun_headline_variants":["VLM self-driving gets a closed-track stress test","Real-world track puts VLM driving AI through its paces","Before the road: VLM autonomous driving tested on closed track","Closed track runs VLM cars through repeatable scenarios","VLM driving gets repeatable real-world trials on closed track"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline accuracy results assume that each module's expected output is the correct answer; for behavior selection and prompt/action formatting the expected output is the same structured data the system itself encoded, so near-perfect correctness mostly reflects self-consistency rather than whether the VLM made the right driving decision.","fun_headline_variants_meta":{"raw":{"variants":["VLM self-driving gets a closed-track stress test","Real-world track puts VLM driving AI through its paces","Before the road: VLM autonomous driving tested on closed track","Closed track runs VLM cars through repeatable scenarios","VLM driving gets repeatable real-world trials on closed track"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000166,"raw_usage":{"total_tokens":1240,"prompt_tokens":920,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":239}},"tokens_in":536,"tokens_out":320,"duration_ms":3279,"temperature":1.0,"reasoning_tokens":239,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:54:00.370639+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the recorded scenario videos past independent human expert drivers and ask them to label the correct maneuver at each decision point; if the VLM's selected behavior agrees with the experts in materially fewer than the reported near-universal cases, the validation would be measuring self-consistency, not driving competence.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the open-source modular autonomy backbone used as the Autonomous Driving Layer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the cloud-hosted VLM whose decisions are under test."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the object-detection perception results bundled into the VLM prompt."},{"cited_title":"Chidhambararajan, Aman Rangapur, S","cited_arxiv_id":null,"evidence_quote":"provides the speech-recognition engine for the passenger command channel."}],"review_version":1}