Pith. sign in

REVIEW 5 major objections 4 minor 72 references

A dual-process VLM planner that routes simple scenes to fast prediction and complex scenes to verified slow reasoning claims 80.14% accuracy, 97.20% consistency, and 17.39% lower latency.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 13:08 UTC pith:BNP2BFD4

load-bearing objection A useful dual-process routing architecture and an honest long-tail evaluation, but the headline 97.20% LCS is the optimizer scoring itself on the same rule set it was trained against, so the consistency claim needs independent verification. the 5 major comments →

arxiv 2607.19194 v2 pith:BNP2BFD4 submitted 2026-07-21 cs.RO cs.CV

Cognitive Dual-Process Planning for Autonomous Driving with Structured Scene Knowledge and Verifiable Reasoning-Action Consistency

Zhongyao Yang , Haoyu Li , Yu Yan , Zhuangxuan Yu , Jiangfeng Nan , Jinrui Nan This is my paper
classification cs.RO cs.CV
keywords autonomous drivingvision-language modeldual-process planningstructured chain-of-thoughtreasoning-action consistencyrule-based verificationadaptive routingGRPO
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that high-level driving planning can be broken into a fast path and a slow path, with a lightweight visual arbiter deciding before any text is generated whether a scene is simple enough for direct action prediction or complex enough to warrant structured reasoning. The structured reasoning is written in a machine-parsable schema (S-CoT) and checked by deterministic rules that flag contradictions between stated scene factors and the chosen action; these checks provide rewards for policy optimization. On 574 manually verified test scenes, the planner reaches 80.14% planning accuracy and a 97.20% Logical Consistency Score, while cutting mean latency by 17.39% compared with reasoning on every scene. The reader should care because this is a concrete recipe for getting verifiable, explainable reasoning from vision-language models in a latency-constrained setting without hand-annotated rationales.

Core claim

The paper's central claim is that routing complexity estimation before language decoding, and then optimizing the slow reasoning path with deterministic rule-based consistency rewards, produces a planner that is simultaneously more accurate and more internally consistent than either always-fast or always-slow inference. The consistency score is computed by a fixed set of traffic rules that check parsed fields such as 'critical object' and 'environmental condition' against the chosen meta-action. The ablation indicates the logical-consistency reward is the main driver: it raises the consistency score from 62.15% to 97.20% and improves planning accuracy from 70.73% to 80.14%. The paper further

What carries the argument

The framework rests on three components. First, a Structured Chain-of-Thought (S-CoT) schema: machine-parsable fields for environmental condition, critical objects, lane availability, action planning, and reasoning, with discrete uncertainty labels. Second, a decoupled visual Arbiter: a lightweight transformer that pools multilevel vision-encoder features into a scene-complexity probability before language decoding and routes each input to the fast or slow path. Third, a deterministic rule-based validator: a fixed set of traffic rules (e.g., an immediate obstacle on the ego path requires decelerate or stop) that parses the S-CoT fields and checks them against the final meta-action; this vali

Load-bearing premise

The central claim rests on the assumption that training on action-anchored labels (expert first picks the action, then writes the rationale to match) and on rule-based rewards teaches the model genuine reasoning–action alignment, rather than the ability to produce post-hoc, rule-satisfying rationales for any action it chooses; if that assumption fails, the reported 97.20% consistency score would overstate the planner's true internal consistency.

What would settle it

An independent human audit of the test-set outputs, looking specifically for cases where the stated rationale does not genuinely justify the chosen action but still passes all validator rules (for example, a hazard that the rule set does not encode), would settle whether the consistency score is real. A cheaper experiment: retrain without the action-anchored reverse-order annotation; if the consistency gain persists, the gain comes from the validator rewards alone, not from the label distribution, which would weaken the authors' mechanism.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, VLM planners can expose a continuous latency–safety trade-off through the arbiter threshold, letting deployment choose how many scenes receive slow reasoning.
  • The deterministic validator provides a scalable source of training reward that does not require a learned reward model, making it cheap to extend to new traffic rules.
  • The automated data engine removes the need to hand-annotate rationale supervision, lowering the cost of building structured driving datasets.
  • Selective routing beats always-slow reasoning in the paper's experiments, indicating that 'over-reasoning' is a genuine failure mode that adaptive schemes can avoid.
  • The external long-tail subsets show the approach degrades most under low visibility and traffic-sign reasoning, giving a concrete target for future work.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because the consistency score is computed with a fixed rule set, a natural stress test would replace the rule set with stricter or different traffic rules and measure how much LCS drops; that would reveal whether the model has learned the specific rules or a deeper notion of scene–action consistency.
  • The reported false-negative rates across arbiter thresholds suggest that in a real deployment the threshold could be calibrated against a safety-critical false-negative budget rather than a fixed value, making the arbiter a safety dial.
  • The action-anchored annotation procedure is essentially a data-generation intervention; injecting deliberate rationale–action mismatches into training and checking whether the policy learns to flag them would test whether the consistency signal is robust or merely mirrors the training distribution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 4 minor

Summary. The paper proposes a dual-process planning framework for high-level autonomous driving. An automated data engine builds machine-parsable structured chain-of-thought (S-CoT) annotations using perception foundation models and an expert VLM. A decoupled visual Arbiter estimates scene complexity before language decoding and routes each input either to fast meta-action prediction or to slow structured reasoning. For slow-path outputs, a deterministic rule-based validator checks consistency between parsed S-CoT fields and the final meta-action and supplies rewards for GRPO. The authors report 91.8% CoT accuracy and 98.5% LCS on a 195-scene manual audit of the data engine, and 80.14% planning accuracy, 97.20% LCS, and 17.39% latency reduction versus static slow reasoning on 574 manually verified NAVSIM test samples. Ablations and external long-tail subsets are also reported, with candid recognition of degradation in low-visibility and traffic-sign scenarios.

Significance. If the central claims are accepted, the paper makes a useful contribution: it demonstrates a pre-decoding routing mechanism that is separate from autoregressive generation and exposes a threshold-controlled operating point; it introduces a machine-parsable S-CoT schema for high-level planning; and it applies deterministic rule-based rewards within GRPO to improve planning accuracy and reasoning-action consistency. The paper's strengths include a held-out test split, a manual audit of the data engine, an ablation study of the reward components, and an external long-tail evaluation that honestly reports failure modes. However, the most distinctive claim—verifiable reasoning-action consistency—is currently not independently supported because the evaluation metric is the same rule-based function used as the training reward and because the training-data construction removes the very contradictions the metric is intended to detect.

major comments (5)
  1. [§4.1 Eq. (11), §3.5 Eq. (8), Table 2] LCS is computed by the same deterministic validator (Table 1) that defines R_logic during GRPO. Since GRPO maximizes R_logic, the reported jump from 62.15% to 97.20% (Table 2; ablation config 1→2 in Table 5: 71.50→96.96) is in substantial part the policy optimizing its own evaluation metric. The 195-scene manual audit (§4.2) validates the validator on data-engine outputs, not on the final policy's test-set outputs. To support the central consistency claim, the authors should provide an independent evaluation: human audit of final-policy outputs, a held-out validator not used in training, or a constructed set of contradiction probes with precision/recall against known contradictions.
  2. [§3.2 Action-Anchored Reverse-Order Annotation] The annotation procedure first predicts the meta-action and then writes the rationale with that action fixed. Every SFT label is therefore consistent by construction, and the policy never sees a counterexample in which a rationale conflicts with an action. Combined with the same-validator issue, the large LCS increase is consistent with the model learning post-hoc rationalizations that satisfy the Table 1 trigger conditions, rather than learning to detect genuine reasoning-action contradictions. The paper should either include contradictory examples in the SFT/RLVR data or validate on examples that were not constructed to be consistent.
  3. [§4.1 Eq. (11), Fig. 7] The LCS denominator is ambiguous for routed models. If complex scenes misrouted to the fast path (FNR = 6.65% at the default threshold, roughly 28 samples on the 574-sample set) are excluded from N_complex because they produce no S-CoT, the reported 97.20% LCS is conditional on correct routing and is inflated relative to an unconditional consistency measure. Clarify whether LCS is computed on the full complex-scene set with forced slow reasoning, or only on scenes actually routed to the slow path, and report both variants.
  4. [Table 3, Table 2] The headline efficiency comparison (Dual-Process 80.14% vs Slow-Think 77.87%) is a 2.27-pp difference, which corresponds to roughly 13 correctly classified samples on 574; no confidence intervals or significance tests are reported. The paper should provide bootstrap confidence intervals or a paired significance test (e.g., McNemar's test) for planning accuracy, LCS, and routing F1/FNR, including the sensitivity curves in Fig. 7.
  5. [§4.1 External subsets] The external long-tail evaluation is difficult to assess because the subsets include 'non-public real-world cases' with no composition, labeling protocol, or source details. At a minimum, the authors should describe how these scenes were collected, how ground-truth actions were obtained, and how the action space was aligned with NAVSIM. Without this, the external robustness claims are not reproducible.
minor comments (4)
  1. [§3.1] The sentence beginning 'SFTestablishes the two output modes' appears to contain a typo; it should likely read 'SFT establishes the two output modes.'
  2. [§4.1 Metrics] The LCS definition in Eq. (11) says it is computed on the complex-scene evaluation set, but it is not stated whether the slow path is forced for all complex scenes during evaluation of non-routed baselines. Please make the evaluation protocol explicit.
  3. [Fig. 7] The threshold-sensitivity panel reports planning accuracy and FNR at five thresholds, but no variance or number of runs is given. Since the default threshold is selected on this curve, a small number of runs could materially affect the chosen operating point.
  4. [General] No code or data availability statement is included. Given that the data engine and external subsets are partly non-public, a release plan for code and the NAVSIM-derived splits would materially improve reproducibility.

Circularity Check

2 steps flagged

LCS gains reflect optimizing the same rule-based validator used as the metric, with contradictions removed from training labels by construction.

specific steps
  1. fitted input called prediction [Sec. 3.5 (Deterministic Consistency Validator) and Sec. 4.1 (Metrics, Eq. 11)]
    "During RLVR, the same rules compute R_logic for sampled outputs. ... The Logical Consistency Score (LCS) is computed using the validator in Table 1: LCS = N_pass/N_complex, where N_pass denotes the number of samples in the complex-scene evaluation set that pass all logical-consistency rules."

    The GRPO reward R_logic (Eq. 8) is computed by the same deterministic rule-based validator that defines LCS (Eq. 11). The policy is optimized with GRPO to maximize R_logic, and then LCS is reported as the evaluation metric. The 35.05-point LCS increase (62.15%→97.20%) is therefore in substantial part the optimizer improving its own training objective, not an externally grounded measurement of reasoning-action consistency. The held-out test set gives this some generalization content, but the metric and the training signal are the same function, so the central consistency claim is partially self-referential.

  2. self definitional [Sec. 3.2, Action-Anchored Reverse-Order Annotation]
    "Unlike online inference, offline annotation first requires the expert VLM to predict the discrete meta-action and then generate an action-conditioned rationale while keeping the selected action fixed."

    The S-CoT supervision is generated by first fixing the action and then writing a rationale conditioned on it, so reasoning-action contradictions are absent from the label distribution by construction. The data-engine LCS of 98.5% and the final policy LCS of 97.20% are measured against a distribution in which the target property was baked into the annotation procedure. The model is never trained on counterexamples where a rationale conflicts with an action, so the high LCS supports learning to reproduce the action-anchored generation pattern rather than learning to detect genuine reasoning-action inconsistencies.

full rationale

The planning-accuracy claim (80.14%) and the latency reduction (17.39%) are grounded in ground-truth NAVSIM meta-actions and are not circular. The circularity concern is specific to the reasoning-action consistency metric. Section 3.5 defines R_logic using the Table 1 validator, and Section 4.1 defines LCS with the same validator; the reported LCS improvement is largely the optimizer maximizing its own reward. Additionally, the action-anchored reverse-order annotation removes contradictory action-rationale pairs from training data by construction, so the model is trained to imitate a procedure where the action is fixed before the rationale is written. The manual audit in Section 4.2 validates the validator on data-engine outputs, not on the final GRPO policy's test-set outputs, so the 97.20% test LCS lacks an independent human check. Because the held-out test set and external long-tail subsets (LCS drops to 85.18%) show some generalization, the circularity is partial rather than total. No load-bearing self-citation chain or uniqueness argument is present, so the score is 6 rather than 8.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 1 invented entities

The central claims rest on a hand-defined task decomposition (meta-actions, complexity labels, validator rules) and on a data-generation procedure that builds consistency into the labels. No physical entity is invented; the free parameters are thresholds and reward weights tuned on validation plus several unreported hyperparameters.

free parameters (6)
  • Arbiter routing threshold tau = 0.7
    Selected on the validation set (Figure 7); directly sets the operating point for routing F1, FNR, latency, and planning accuracy.
  • Arbiter risk penalty weight w_r = not reported
    Hand-set in Eq. 3; the paper does not give its value or sensitivity.
  • Cost coefficients alpha and gamma in C_r = alpha = gamma = 1
    Hand-set in Eq. 5; C_r is computed offline with the frozen SFT planner.
  • GRPO reward weights for R_logic and R_risk = default 1.0
    Tuned on validation; Figure 7 reports sensitivity and notes that overweighting induces conservative behavior.
  • SFT segment weights w_t and global coefficient lambda = not reported
    Hand-set weights in Eq. 7; no values are given.
  • GRPO group size G, KL coefficient beta, clip epsilon = not reported
    Standard GRPO hyperparameters used in Eq. 9-10 but not reported in the text.
axioms (5)
  • domain assumption The rule-based validator in Table 1 captures the traffic rules needed to judge reasoning-action consistency.
    LCS is defined via Eq. 11 using this validator; if the rules are incomplete or wrong, the consistency score overstates true safety alignment.
  • ad hoc to paper Action-anchored reverse-order annotation produces trustworthy consistency labels.
    Section 3.2 fixes the meta-action before generating the rationale, so contradictions are removed from the label distribution by construction; the paper does not measure what this hides.
  • ad hoc to paper Scene-complexity labels (environmental, density, strategic-interaction risk) are a valid ground truth for routing.
    Section 3.2 hand-defines simple/complex; the Arbiter is trained against these labels, so routing performance is bounded by that definition.
  • domain assumption The NAVSIM-derived discrete meta-action space is a meaningful high-level planning target.
    Exact-match accuracy over 1-of-3 lateral and 1-of-4 longitudinal actions is treated as planning performance; no trajectory-level or closed-loop validation is claimed in the Conclusion.
  • domain assumption Offline privileged perception (future ego trajectory, SAM, Depth Anything) transfers to non-privileged deployment inputs.
    Section 3.2 uses future trajectory and offline perception models to build supervision, but these are unavailable at deployment; the transfer gap is not quantified.
invented entities (1)
  • S-CoT schema (structured chain-of-thought fields: environmental condition, critical objects, lane availability, action planning, brief reasoning) no independent evidence
    purpose: Machine-parsable intermediate representation for supervision and rule-based validation.
    The schema is an authored annotation format with no external, falsifiable handle beyond the authors' manual audit and the paper's own validator.

pith-pipeline@v1.3.0-alltime-deepseek · 16283 in / 13210 out tokens · 137691 ms · 2026-08-01T13:08:36.321626+00:00 · methodology

0 comments
read the original abstract

High-level planning for autonomous driving is a knowledge-intensive engineering decision task that requires accurate scene understanding, timely inference, and internally consistent action selection. Vision-language models (VLMs) can make intermediate reasoning explicit, but their use in deployed planners is constrained by costly structured supervision, unnecessary reasoning in routine scenes, and possible inconsistencies between generated rationales and driving actions. We present a cognitive dual-process planning framework that represents planning-relevant scene knowledge in a machine-parsable structured chain-of-thought (S-CoT) schema. An automated data engine integrates perception foundation models, critical-path filtering, and an expert VLM to generate S-CoT supervision without manual annotation of individual rationales. A lightweight visual Arbiter estimates scene complexity from multilevel vision-encoder features before language decoding and routes each input to either fast meta-action prediction or slow structured reasoning. For slow-path outputs, a deterministic rule-based validator checks whether the parsed S-CoT fields are consistent with the final meta-action and provides verifiable rewards for Group Relative Policy Optimization (GRPO). In a 195-scene manual audit, the generated annotations achieve 91.8\% CoT accuracy and a 98.5\% Logical Consistency Score (LCS). On 574 manually verified NAVSIM test samples, the planner achieves 80.14\% planning accuracy and 97.20\% LCS while reducing average latency by 17.39\% relative to applying slow reasoning to every scene. Evaluation on external long-tail subsets further identifies conditions under which routing and planning performance degrade. Together, these results show how explicit scene knowledge can be operationalized through adaptive reasoning and rule-based verification to support high-level VLM planning decisions.

Figures

Figures reproduced from arXiv: 2607.19194 by Haoyu Li, Jiangfeng Nan, Jinrui Nan, Yu Yan, Zhongyao Yang, Zhuangxuan Yu.

Figure 1
Figure 1. Figure 1: Overview of the proposed dual-process architecture. The planner takes front-view images, navigation commands, and ego-vehicle states as input. A lightweight Arbiter estimates scene complexity and routes each input to fast meta-action prediction or slow structured reasoning. The automated data engine provides structured chain-of-thought (S-CoT) supervision, while rule-based verification supplies rewards tha… view at source ↗
Figure 2
Figure 2. Figure 2: Automated data engine and hierarchical training workflow. The offline pipeline uses perception foundation models and an expert vision-language model (VLM) in sequence to generate structured chain-of-thought (S-CoT) annotations and scene-complexity labels. Training comprises a backbone supervised fine-tuning (SFT) warm-up followed by independent Arbiter calibration and Group Relative Policy Optimization (GR… view at source ↗
Figure 3
Figure 3. Figure 3: reports CoT accuracy and LCS for annota￾tions generated by the three pipelines. The single-pass AR pipeline achieves a CoT accuracy of 80.5% and an LCS of 88.7%; these lower scores are consistent with its suscepti￾bility to perception hallucinations. The modular-perception pipeline improves both metrics, but both remain below those achieved by the complete data engine. The complete data engine achieves 91.… view at source ↗
Figure 4
Figure 4. Figure 4: Qualitative comparison of reasoning–action consis￾tency in a complex scene. The supervised fine-tuning (SFT) baseline produces an action that contradicts the perceived haz￾ard, whereas the Group Relative Policy Optimization (GRPO)- aligned policy produces an action consistent with its stated reasoning. The verifiable reinforcement-learning stage strengthens this alignment. Relative to the Qwen3-VL SFT base… view at source ↗
Figure 5
Figure 5. Figure 5: Qualitative example of over-reasoning in a simple scene. In this example, full structured reasoning produces a hallucinated risk and an unnecessary conservative maneuver, whereas dynamic routing avoids invoking the slow reasoning path [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Qualitative routing analysis in out-of-distribution (OOD) and long-tail driving scenarios. The left panel shows correctly routed examples, with simple scenes assigned to the fast path and complex scenes assigned to the slow path. The right panel shows false-negative cases in which complex scenes receive low complexity scores and are incorrectly assigned to the fast path. With the routing threshold set to 𝜏… view at source ↗
Figure 7
Figure 7. Figure 7: Sensitivity to the Arbiter threshold and verifiable-reward weights. The left panel reports planning accuracy and false￾negative rate (FNR) at five thresholds; the dotted line marks the default setting (𝜏 = 0.7), selected based on the joint trade-off among planning accuracy, FNR, routing F1, and latency. In the right panel, each verifiable-reward weight is varied while the other remains at its default value… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

72 extracted references · 20 linked inside Pith

  1. [1]

    Qwen3-vl technical report

    Bai, S., Cai, Y., Chen, R., Chen, K., Chen, X., Cheng, Z., Deng, L., Ding, W., Gao, C., Ge, C., et al., 2025a. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631

    Pith/arXiv arXiv

  2. [2]

    Qwen2.5-vl technical report

    Bai,S.,Chen,K.,Liu,X.,Wang,J.,Ge,W.,Song,S.,Dang,K.,Wang, P., Wang, S., Tang, J., et al., 2025b. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.13923

    Pith/arXiv arXiv

  3. [3]

    Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L.D., Monfort, M., Muller, U., Zhang, J., et al.,

  4. [4]

    Arobustlayeredcontrolsystemforamobilerobot

    Brooks,R.,1986. Arobustlayeredcontrolsystemforamobilerobot. IEEE journal on robotics and automation 2, 14–23

    1986

  5. [5]

    nuplan:Aclosed-loopml- based planning benchmark for autonomous vehicles

    Caesar, H., Kabzan, J., Tan, K.S., Fong, W.K., Wolff, E., Lang, A., Fletcher,L.,Beijbom,O.,Omari,S.,2021. nuplan:Aclosed-loopml- based planning benchmark for autonomous vehicles. arXiv preprint arXiv:2106.11810

    Pith/arXiv arXiv 2021

  6. [6]

    Driving by the rules: A benchmark for integrating traffic sign regulations into vectorized hd map, in: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Chang, X., Xue, M., Liu, X., Pan, Z., Wei, X., 2025. Driving by the rules: A benchmark for integrating traffic sign regulations into vectorized hd map, in: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 6823–6833

    2025

  7. [7]

    End-to-end autonomous driving: Challenges and frontiers

    Chen, L., Wu, P., Chitta, K., Jaeger, B., Geiger, A., Li, H., 2024a. End-to-end autonomous driving: Challenges and frontiers. IEEE Transactions on Pattern Analysis and Machine Intelligence

  8. [8]

    Solve: Synergyoflanguage-visionandend-to-endnetworksforautonomous driving, in: Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp

    Chen, X., Huang, L., Ma, T., Fang, R., Shi, S., Li, H., 2025. Solve: Synergyoflanguage-visionandend-to-endnetworksforautonomous driving, in: Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp. 12068–12077

    2025

  9. [9]

    Internvl: Scaling up vision foundationmodelsandaligningforgenericvisual-linguistictasks,in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al., 2024b. Internvl: Scaling up vision foundationmodelsandaligningforgenericvisual-linguistictasks,in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185–24198

  10. [10]

    Asurveyoftransformerarchi- tectures for autonomous driving

    Chu,F.,Li,H.,Xie,L.,Zhao,J.,2025. Asurveyoftransformerarchi- tectures for autonomous driving. Expert Systems with Applications , 130338

    2025

  11. [11]

    Codevilla, F., Müller, M., López, A., Koltun, V., Dosovitskiy, A.,

  12. [12]

    Navsim: Data-driven non-reactive autonomous vehicle simulation and bench- marking

    Dauner,D.,Hallgarten,M.,Li,T.,Weng,X.,Huang,Z.,Yang,Z.,Li, H., Gilitschenski, I., Ivanovic, B., Pavone, M., et al., 2024. Navsim: Data-driven non-reactive autonomous vehicle simulation and bench- marking. Advances in Neural Information Processing Systems 37, 28706–28719

    2024

  13. [13]

    Driverx: A vision-language reasoning model for cross-task autonomous driving

    Diao, M., Yang, L., Yin, H., Wang, Z., Wang, Y., Tian, D., Liang, K., Ma, Z., 2025. Driverx: A vision-language reasoning model for cross-task autonomous driving. arXiv preprint arXiv:2505.20665

    arXiv 2025

  14. [14]

    Performanceandlightweightcoexistence:Vision-languagemodelfor visual scene question answering in autonomous driving

    Gan, L., Chen, Z., Hu, Q., Tang, X., Luo, X., Chu, W., Li, G., 2026. Performanceandlightweightcoexistence:Vision-languagemodelfor visual scene question answering in autonomous driving. Advanced Engineering Informatics 71, 104424

    2026

  15. [15]

    Large models for intelligenttransportationsystemsandautonomousvehicles:Asurvey

    Gan, L., Chu, W., Li, G., Tang, X., Li, K., 2024. Large models for intelligenttransportationsystemsandautonomousvehicles:Asurvey. Advanced Engineering Informatics 62, 102786

    2024

  16. [16]

    Pal: Program-aided language models, in: International Conference on Machine Learning, PMLR

    Gao, L., Madaan, A., Zhou, S., Alon, U., Liu, P., Yang, Y., Callan, J., Neubig, G., 2023. Pal: Program-aided language models, in: International Conference on Machine Learning, PMLR. pp. 10764– 10799

    2023

  17. [17]

    Explainable ai for en- gineering design: A unified approach of systems engineering and component-based deep learning demonstrated by energy-efficient building design

    Geyer, P., Singh, M.M., Chen, X., 2024. Explainable ai for en- gineering design: A unified approach of systems engineering and component-based deep learning demonstrated by energy-efficient building design. Advanced Engineering Informatics 62, 102843

    2024

  18. [18]

    Ghosh, A., Zheng, S., Tamburo, R., Vuong, K., Alvarez-Padilla, J., Zhu, H., Cardei, M., Dunn, N., Mertz, C., Narasimhan, S.G.,

  19. [19]

    Deepseek-r1:Incentivizingrea- soning capability in llms via reinforcement learning

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma,S.,Wang,P.,Bi,X.,etal.,2025. Deepseek-r1:Incentivizingrea- soning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    Pith/arXiv arXiv 2025

  20. [20]

    Planning-oriented autonomous driving, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp

    Hu, Y., Yang, J., Chen, L., Li, K., Sima, C., Zhu, X., Chai, S., Du, S., Lin, T., Wang, W., et al., 2023. Planning-oriented autonomous driving, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17853–17862

    2023

  21. [21]

    Senna: Bridging large vision- languagemodelsandend-to-endautonomousdriving

    Jiang, B., Chen, S., Liao, B., Zhang, X., Yin, W., Zhang, Q., Huang, C., Liu, W., Wang, X., 2024. Senna: Bridging large vision- languagemodelsandend-to-endautonomousdriving. arXivpreprint arXiv:2410.22313

    Pith/arXiv arXiv 2024

  22. [22]

    Vad: Vectorized scene representationforefficientautonomousdriving,in:Proceedingsofthe IEEE/CVF International Conference on Computer Vision, pp

    Jiang, B., Chen, S., Xu, Q., Liao, B., Chen, J., Zhou, H., Zhang, Q., Liu, W., Huang, C., Wang, X., 2023. Vad: Vectorized scene representationforefficientautonomousdriving,in:Proceedingsofthe IEEE/CVF International Conference on Computer Vision, pp. 8340– 8350

    2023

  23. [23]

    Alphadrive: Unleashing the power of vlms in autonomous driving via reinforce- ment learning and reasoning

    Jiang, B., Chen, S., Zhang, Q., Liu, W., Wang, X., 2025. Alphadrive: Unleashing the power of vlms in autonomous driving via reinforce- ment learning and reasoning. arXiv preprint arXiv:2503.07608

    Pith/arXiv arXiv 2025

  24. [24]

    General lane-changing model mobil for car-following models

    Kesting, A., Treiber, M., Helbing, D., 2007. General lane-changing model mobil for car-following models. Transportation Research Record 1999, 86–94

    2007

  25. [25]

    Segment anything, in: Proceedings of the IEEE/CVF international conference on computer vision, pp

    Kirillov,A.,Mintun,E.,Ravi,N.,Mao,H.,Rolland,C.,Gustafson,L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al., 2023. Segment anything, in: Proceedings of the IEEE/CVF international conference on computer vision, pp. 4015–4026

    2023

  26. [26]

    Robo-detr: Robustness-aware depth-guided trans- formerformonocular3dobjectdetectionunderadversevisualcondi- tions

    Li, H., Zhang, X., Zhong, J., Chen, Z., Zhao, Y., Wang, B., Leng, J., Sun, C., 2026. Robo-detr: Robustness-aware depth-guided trans- formerformonocular3dobjectdetectionunderadversevisualcondi- tions. IEEE Internet of Things Journal

    2026

  27. [27]

    Delving into the secrets of bev 3d object detection in autonomous driving: a comprehensive survey

    Li, H., Zhao, Y., Zhong, J., Wang, B., Sun, C., Sun, F., 2025. Delving into the secrets of bev 3d object detection in autonomous driving: a comprehensive survey. IEEE Transactions on Intelligent Transportation Systems 27, 119–144

    2025

  28. [28]

    Hydra-mdp:End-to-endmultimodalplanning withmulti-targethydra-distillation

    Li, Z., Li, K., Wang, S., Lan, S., Yu, Z., Ji, Y., Li, Z., Zhu, Z., Kautz, J.,Wu,Z.,etal.,2024. Hydra-mdp:End-to-endmultimodalplanning withmulti-targethydra-distillation. arXivpreprintarXiv:2406.06978

    Pith/arXiv arXiv 2024

  29. [29]

    Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving, in: Proceedings of the Computer Vision and Pattern Recognition Conference, pp

    Liao, B., Chen, S., Yin, H., Jiang, B., Wang, C., Yan, S., Zhang, X., Li, X., Zhang, Y., Zhang, Q., et al., 2025. Diffusiondrive: Truncated diffusion model for end-to-end autonomous driving, in: Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 12037–12047

    2025

  30. [30]

    Basic problems in stability and design of switched systems

    Liberzon, D., Morse, A.S., 1999. Basic problems in stability and design of switched systems. IEEE control systems magazine 19, 59– 70. Z. Yang & H. Li et al.:Preprint submitted to ElsevierPage 11 of 12

    1999

  31. [31]

    Visual instruction tuning

    Liu, H., Li, C., Wu, Q., Lee, Y.J., 2023. Visual instruction tuning. Advancesinneuralinformationprocessingsystems36,34892–34916

    2023

  32. [32]

    Grounding dino: Marrying dino withgroundedpre-trainingforopen-setobjectdetection,in:European conference on computer vision, Springer

    Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Jiang, Q., Li, C., Yang, J., Su, H., et al., 2024. Grounding dino: Marrying dino withgroundedpre-trainingforopen-setobjectdetection,in:European conference on computer vision, Springer. pp. 38–55

    2024

  33. [33]

    Adathinkdrive: Adaptive thinking via reinforcement learning for autonomous driving

    Luo, Y., Li, F., Xu, S., Lai, Z., Yang, L., Chen, Q., Luo, Z., Xie, Z., Jiang, S., Liu, J., et al., 2025. Adathinkdrive: Adaptive thinking via reinforcement learning for autonomous driving. arXiv preprint arXiv:2509.13769

    arXiv 2025

  34. [34]

    Meng, L., Yang, J., Tian, R., Dai, X., Wu, Z., Gao, J., Jiang, Y.G.,

  35. [35]

    Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving, in: European Conference on Computer Vision, Springer

    Nie,M.,Peng,R.,Wang,C.,Cai,X.,Han,J.,Xu,H.,Zhang,L.,2024. Reason2drive: Towards interpretable and chain-based reasoning for autonomous driving, in: European Conference on Computer Vision, Springer. pp. 292–308

    2024

  36. [36]

    Training language models to follow instructions with human feedback

    Ouyang,L.,Wu,J.,Jiang,X.,Almeida,D.,Wainwright,C.,Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al., 2022. Training language models to follow instructions with human feedback. Ad- vances in neural information processing systems 35, 27730–27744

    2022

  37. [37]

    Fasionad:Fastandslowfusionthinking systems for human-like autonomous driving with adaptive feedback

    Qian, K., Ma, Z., He, Y., Luo, Z., Shi, T., Zhu, T., Li, J., Wang, J., Chen,Z.,He,X.,etal.,2024. Fasionad:Fastandslowfusionthinking systems for human-like autonomous driving with adaptive feedback. arXiv preprint arXiv:2411.18013

    Pith/arXiv arXiv 2024

  38. [38]

    Simlingo: Vision- only closed-loop autonomous driving with language-action align- ment, in: Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pp

    Renz, K., Chen, L., Arani, E., Sinavski, O., 2025. Simlingo: Vision- only closed-loop autonomous driving with language-action align- ment, in: Proceedings of the Computer Vision and Pattern Recogni- tion Conference, pp. 11993–12003

    2025

  39. [39]

    Gaia-2: A controllable multi-view gen- erative world model for autonomous driving

    Russell, L., Hu, A., Bertoni, L., Fedoseev, G., Shotton, J., Arani, E., Corrado, G., 2025. Gaia-2: A controllable multi-view gen- erative world model for autonomous driving. arXiv preprint arXiv:2503.20523

    Pith/arXiv arXiv 2025

  40. [40]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., Klimov, O.,

  41. [41]

    Reasonnet: End-to-end driving with temporal and global reasoning, in:ProceedingsoftheIEEE/CVFconferenceoncomputervisionand pattern recognition, pp

    Shao, H., Wang, L., Chen, R., Waslander, S.L., Li, H., Liu, Y., 2023. Reasonnet: End-to-end driving with temporal and global reasoning, in:ProceedingsoftheIEEE/CVFconferenceoncomputervisionand pattern recognition, pp. 13723–13733

    2023

  42. [42]

    Deepseekmath: Pushing the limits of mathematical reasoning in open language models

    Shao,Z.,Wang,P.,Zhu,Q.,Xu,R.,Song,J.,Bi,X.,Zhang,H.,Zhang, M., Li, Y., Wu, Y., et al., 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    Pith/arXiv arXiv 2024

  43. [43]

    Learning to summarize with human feedback

    Stiennon,N.,Ouyang,L.,Wu,J.,Ziegler,D.,Lowe,R.,Voss,C.,Rad- ford, A., Amodei, D., Christiano, P.F., 2020. Learning to summarize with human feedback. Advances in neural information processing systems 33, 3008–3021

    2020

  44. [44]

    Sparsedrive: End-to-end autonomous driving via sparse scene repre- sentation, in: 2025 IEEE International Conference on Robotics and Automation (ICRA), IEEE

    Sun, W., Lin, X., Shi, Y., Zhang, C., Wu, H., Zheng, S., 2025. Sparsedrive: End-to-end autonomous driving via sparse scene repre- sentation, in: 2025 IEEE International Conference on Robotics and Automation (ICRA), IEEE. pp. 8795–8801

    2025

  45. [45]

    Drivevlm: The convergence of autonomousdrivingandlargevision-languagemodels.arXivpreprint arXiv:2402.12289

    Tian, X., Gu, J., Li, B., Liu, Y., Wang, Y., Zhao, Z., Zhan, K., Jia, P., Lang, X., Zhao, H., 2024. Drivevlm: The convergence of autonomousdrivingandlargevision-languagemodels.arXivpreprint arXiv:2402.12289

    Pith/arXiv arXiv 2024

  46. [46]

    Congestedtrafficstates in empirical observations and microscopic simulations

    Treiber,M.,Hennecke,A.,Helbing,D.,2000. Congestedtrafficstates in empirical observations and microscopic simulations. Physical review E 62, 1805

    2000

  47. [47]

    Attention is all you need

    Vaswani,A.,Shazeer,N.,Parmar,N.,Uszkoreit,J.,Jones,L.,Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017. Attention is all you need. Advances in neural information processing systems 30

    2017

  48. [48]

    A process knowledge representation approach for decision support in design of complex engineered systems

    Wang, R., Nellippallil, A.B., Wang, G., Yan, Y., Allen, J.K., Mistree, F., 2021. A process knowledge representation approach for decision support in design of complex engineered systems. Advanced Engi- neering Informatics 48, 101257

    2021

  49. [49]

    Omnidrive: A holistic vision-language datasetforautonomousdrivingwithcounterfactualreasoning,in:Pro- ceedings of the computer vision and pattern recognition conference, pp

    Wang, S., Yu, Z., Jiang, X., Lan, S., Shi, M., Chang, N., Kautz, J., Li, Y., Alvarez, J.M., 2025a. Omnidrive: A holistic vision-language datasetforautonomousdrivingwithcounterfactualreasoning,in:Pro- ceedings of the computer vision and pattern recognition conference, pp. 22442–22452

  50. [50]

    Drivecot:Integrating chain-of-thought reasoning with end-to-end driving

    Wang,T.,Xie,E.,Chu,R.,Li,Z.,Luo,P.,2024. Drivecot:Integrating chain-of-thought reasoning with end-to-end driving. arXiv preprint arXiv:2403.16996

    Pith/arXiv arXiv 2024

  51. [51]

    Self-consistency improves chain of thought reasoning in language models

    Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., Zhou, D., 2022. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171

    Pith/arXiv arXiv 2022

  52. [52]

    arXiv preprint arXiv:2511.00088

    Wang, Y., Luo, W., Bai, J., Cao, Y., Che, T., Chen, K., Chen, Y., Diamond,J.,Ding,Y.,Ding,W.,etal.,2025b.Alpamayo-r1:Bridging reasoningandactionpredictionforgeneralizableautonomousdriving in the long tail. arXiv preprint arXiv:2511.00088

    Pith/arXiv arXiv

  53. [53]

    Chain-of-thought prompting elicits reasoning in large language models

    Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al., 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems 35, 24824–24837

    2022

  54. [54]

    Dilu: A knowledge-driven approach to autonomous driving with large language models

    Wen, L., Fu, D., Li, X., Cai, X., Ma, T., Cai, P., Dou, M., Shi, B., He, L., Qiao, Y., 2023. Dilu: A knowledge-driven approach to autonomous driving with large language models. arXiv preprint arXiv:2309.16292

    Pith/arXiv arXiv 2023

  55. [55]

    Para- drive: Parallelized architecture for real-time autonomous driving, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Weng,X.,Ivanovic,B.,Wang,Y.,Wang,Y.,Pavone,M.,2024. Para- drive: Parallelized architecture for real-time autonomous driving, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15449–15458

    2024

  56. [56]

    Xing, S., Qian, C., Wang, Y., Hua, H., Tian, K., Zhou, Y., Tu, Z.,

  57. [57]

    Towards human-centric autonomous driving: A fast-slow architecture inte- grating large language model guidance with reinforcement learning

    Xu, C., Liu, J., Guo, Y., Zhang, Y., Hang, P., Sun, J., 2025. Towards human-centric autonomous driving: A fast-slow architecture inte- grating large language model guidance with reinforcement learning. arXiv preprint arXiv:2505.06875

    Pith/arXiv arXiv 2025

  58. [58]

    Vlm-ad: End-to-end autonomous driving through vision-language model supervision

    Xu, Y., Hu, Y., Zhang, Z., Meyer, G.P., Mustikovela, S.K., Srinivasa, S., Wolff, E.M., Huang, X., 2024a. Vlm-ad: End-to-end autonomous driving through vision-language model supervision. arXiv preprint arXiv:2412.14446

    Pith/arXiv arXiv

  59. [59]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model

    Xu, Z., Zhang, Y., Xie, E., Zhao, Z., Guo, Y., Wong, K.Y.K., Li, Z., Zhao, H., 2024b. Drivegpt4: Interpretable end-to-end autonomous driving via large language model. IEEE Robotics and Automation Letters

  60. [60]

    Yang, L., Kang, B., Huang, Z., Zhao, Z., Xu, X., Feng, J., Zhao, H., 2024.Depthanythingv2.AdvancesinNeuralInformationProcessing Systems 37, 21875–21911

    2024

  61. [61]

    Tree of thoughts: Deliberate problem solving withlargelanguagemodels

    Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., Narasimhan, K., 2023. Tree of thoughts: Deliberate problem solving withlargelanguagemodels. Advancesinneuralinformationprocess- ing systems 36, 11809–11822

    2023

  62. [62]

    1001–1009

    Openemma: Open-source multimodal model for end-to-end autonomous driving, in: Proceedings of the Winter Conference on Applications of Computer Vision, pp. 1001–1009

  63. [63]

    Knowledge transfer from simple to complex: A safe and efficient reinforcement learning framework for autonomous driving decision- making

    Zhou, R., Huang, J., Li, M., Li, H., Cao, H., Song, X., 2025a. Knowledge transfer from simple to complex: A safe and efficient reinforcement learning framework for autonomous driving decision- making. Advanced Engineering Informatics 65, 103188

  64. [64]

    Open- drivevla: Towards end-to-end autonomous driving with large vision language action model

    Zhou, X., Han, X., Yang, F., Ma, Y., Knoll, A.C., 2025b. Open- drivevla: Towards end-to-end autonomous driving with large vision language action model. arXiv preprint arXiv:2503.23463

    arXiv

  65. [65]

    Vision language models in autonomous driving: A survey and outlook

    Zhou, X., Liu, M., Yurtsever, E., Zagar, B.L., Zimmer, W., Cao, H., Knoll, A.C., 2024. Vision language models in autonomous driving: A survey and outlook. IEEE Transactions on Intelligent Vehicles

    2024

  66. [66]

    Autovla: A vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine- tuning

    Zhou, Z., Cai, T., Zhao, S.Z., Zhang, Y., Huang, Z., Zhou, B., Ma, J., 2025c. Autovla: A vision-language-action model for end-to-end autonomous driving with adaptive reasoning and reinforcement fine- tuning. arXiv preprint arXiv:2506.13757 . Z. Yang & H. Li et al.:Preprint submitted to ElsevierPage 12 of 12

    Pith/arXiv arXiv

  67. [68]

    Think-driver: From driving- sceneunderstandingtodecision-makingwithvisionlanguagemodels, in: European Conference on Computer Vision Workshop

    Zhang, Q., Zhu, M., Yang, H.F., 2024. Think-driver: From driving- sceneunderstandingtodecision-makingwithvisionlanguagemodels, in: European Conference on Computer Vision Workshop

    2024

  68. [2016]

    arXiv preprint arXiv:1604.07316

    End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316

    Pith/arXiv arXiv

  69. [2017]

    arXiv preprint arXiv:1707.06347

    Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    Pith/arXiv arXiv

  70. [2018]

    End-to-enddrivingviaconditionalimitationlearning,in:2018 IEEE international conference on robotics and automation (ICRA), IEEE. pp. 4693–4700

    2018

  71. [2024]

    Advances in Neural Information Processing Systems 37, 23464–23487

    Deepstack:Deeplystackingvisualtokensissurprisinglysimple and effective for lmms. Advances in Neural Information Processing Systems 37, 23464–23487

  72. [2025]

    6132– 6142

    Roadwork:Adatasetandbenchmarkforlearningtorecognize, observe,analyzeanddrivethroughworkzones,in:Proceedingsofthe IEEE/CVF International Conference on Computer Vision, pp. 6132– 6142