Pith. sign in

REVIEW 4 major objections 5 minor 14 references

A parameter-efficient LLaVA system, enriched with depth and key-object metadata and fused across models, achieves a validation score of 0.7799 and first place on the DriveLM-nuScenes driving QA leaderboard.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 17:06 UTC pith:VL5YX7QC

load-bearing objection The 0.7799 headline score comes from oracle fusion on validation labels, so it is an upper bound, not a system performance; the single-system results are the only honest contribution. the 4 major comments →

arxiv 2509.11071 v1 pith:VL5YX7QC submitted 2025-09-14 cs.CV cs.AIcs.CL

The System Description of CPS Team for Track on Driving with Language of CVPR 2024 Autonomous Grand Challenge

Jinghan Peng , Jingwen Wang , Xing Yu , Dehui Du This is my paper
classification cs.CV cs.AIcs.CL
keywords Vision language modelsAutonomous drivingDriveLM-nuScenesParameter-efficient fine-tuningLoRADepth estimationChain-of-thoughtAnswer fusion
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper describes a vision-language system for the Driving with Language track of a 2024 autonomous-driving challenge. The authors claim their system, built by fine-tuning LLaVA-1.5-7B and LLaVA-NeXT-7B with LoRA and DoRA and enriching inputs with depth-derived distance and key-object descriptions, achieved a validation score of 0.7799 and ranked first. The result matters because it shows that a comparatively modest, parameter-efficient recipe can outperform larger or more specialized approaches on a challenging driving QA benchmark.

Core claim

The central claim is that the proposed inference pipeline—extracting key-object tags, estimating object depth with an open-source depth model, converting depth to textual distance, and prepending object descriptions and states to the question—combined with a multi-system fusion rule, yields the top validation score of 0.7799 on DriveLM-nuScenes. The paper further claims that LLaVA-NeXT-7B fine-tuned with LoRA performs best among individual systems, and that fusing answers by question type (voting for multiple-choice and yes/no, highest evaluation score for open-ended questions) improves the final result.

What carries the argument

The central mechanism is the prompt design module: it takes the scene image and question, uses the key-object tags to locate objects, computes a representative depth per object (75th percentile of depth estimates from Depth Anything), converts that to a textual distance description, and generates key-object descriptions and states via a separate VLM query. These pieces are concatenated into a single prompt fed to the fine-tuned LLaVA model. The second mechanism is the fusion rule: for each question, the final answer is picked from several systems' outputs—voting for discrete-answer questions, and the response with the highest validation metric for open-ended questions.

Load-bearing premise

The reported score depends on choosing answers using the validation set's own evaluation metrics, so the result measures fit to the validation set rather than a fair out-of-sample prediction.

What would settle it

Hold out a random 20% of validation questions, apply the fusion rule using only the other 80% to pick answers, then score the held-out 20%; if that score drops well below 0.7799, the reported result is inflated by validation-set selection.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • LoRA fine-tuning outperforms DoRA on both LLaVA-1.5-7B and LLaVA-NeXT-7B for this driving QA task.
  • LLaVA-NeXT-7B with LoRA surpasses LLaVA-1.5-7B with either method on accuracy and language metrics.
  • Answer fusion across multiple systems improves the final score over any single system (0.7799 vs 0.7710).
  • One epoch of fine-tuning is enough to reach top performance without overfitting on this dataset.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The fusion rule selects open-ended answers by validation-set scores, so part of the reported improvement may reflect fitting the validation set rather than a genuine gain in generalization; an independent evaluation would be needed to confirm.
  • The approach suggests that injecting geometric priors (distance descriptions from depth maps) into the VLM prompt is a low-cost alternative to architectural changes for driving perception tasks.
  • The reported failure of few-shot CoT implies that generic reasoning prompts can suppress domain-specific knowledge; task-aware prompt construction may matter more than the prompting technique itself.
  • A 7B model with LoRA matching or beating bespoke models suggests the DriveLM benchmark can be approached without large-scale custom pretraining.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper is a system description for the DriveLM-nuScenes 'Driving with Language' track of the CVPR 2024 Autonomous Grand Challenge. The authors fine-tune LLaVA-1.5-7B and LLaVA-NeXT-7B with LoRA and DoRA, enrich inputs with key-object descriptions and depth estimates from Depth Anything, and build an inference pipeline that incorporates these cues into the prompt. They also attempt Chain-of-Thought prompting for multiple-choice and yes/no questions. On the validation set they report three individual system scores (0.7329, 0.7177, 0.7710) and a fused score of 0.7799, claiming 1st place on the validation leaderboard. The central claim is the fused score, which is obtained by selecting, for open-ended questions, the answer with the highest validation-set metric score among the individual systems.

Significance. The work demonstrates a practical recipe of parameter-efficient fine-tuning and multimodal prompting for driving question answering. The individual model comparison is informative: LLaVA-NeXT-7B+LoRA substantially outperforms LLaVA-1.5-7B variants, and the LoRA-vs-DoRA comparison is a useful data point. The paper honestly reports the validation protocol and does not claim a test-set result. However, the reported headline score is not a faithful measure of a deployed system because the fusion stage uses validation ground-truth metrics to choose per-question outputs; it is an oracle upper bound. The best honest single-system score is 0.7710. The paper also lacks code/checkpoints and gives an ambiguous account of CoT, which limits reproducibility. If the fusion section is corrected and the claims are revised accordingly, the paper can be a useful challenge report.

major comments (4)
  1. [§6, fusion paragraph] 'For other types of questions, we choose the answer that achieves the highest evaluation score across the relevant evaluation metrics as the final answer.' Selecting the best-scoring answer per question requires the validation ground-truth references (BLEU/ROUGE/CIDEr are all computed against references). Therefore the reported 0.7799 is an oracle upper bound, not the score of any coherent system that can be run on unseen data. This invalidates the abstract's 'top score of 0.7799... ranking 1st' as a system performance claim. The paper should use a fixed fusion rule that does not consult validation labels (e.g., majority vote for all question types) and report the resulting score, or explicitly label the 0.7799 as an oracle/upper-bound analysis and remove it from the abstract/conclusion as the headline result.
  2. [§5, CoT paragraph] The text first says the approach combines Zero-shot CoT and Few-shot CoT 'to enhance response accuracy,' then reports that application of few-shot CoT 'decreases model performance during evaluations' and 'may have inadvertently constrained the model's inherent reasoning abilities.' The final pipeline in Fig. 3 does not include a CoT stage, and no ablation quantifies the effect. The abstract nonetheless credits CoT. Please specify whether the submitted validation runs used CoT, remove the ambiguity with an explicit protocol statement, and include with/without CoT results for the final systems or explicitly state that CoT was removed.
  3. [§6, Table 1 and fusion details] The fusion uses 'various individual systems' beyond the three listed, but the number of systems, their individual scores, and the exact voting rule are not reported. If the voting for multiple-choice/yes-no questions uses any validation metric for tie-breaking, it suffers from the same oracle issue as the open-ended selection. Please give the full system inventory, the fusion rule, and, for the open-ended selection, the evaluator's identity (e.g., which metric) and the number of systems considered per question. Without this information the fusion is not reproducible and the contribution of each component cannot be assessed.
  4. [Abstract and §7] The rank-1 claim is made for the validation leaderboard, but the abstract omits 'validation,' and no link or external leaderboard snapshot is provided. In a setting where the final score is obtained with validation-set oracle selection, the stand-alone phrase 'ranking 1st on the leaderboard' is misleading. Please clarify the evaluation set, give the leaderboard reference (URL/date), and ensure that the rank claim is attached to a valid (non-oracle) score or is explicitly qualified.
minor comments (5)
  1. [Throughout] 'LLaV A' should be 'LLaVA' (e.g., abstract, Sec. 3).
  2. [Fig. 3 title] 'PROMPT DESING' is a typo; should be 'PROMPT DESIGN'.
  3. [§2.2] 'an 11×11 rectangular frame' should be 'an 11×11 window'.
  4. [Table 1] Use consistent metric capitalization: 'Bleu1' → 'BLEU-1', 'ROUGE L' → 'ROUGE-L', etc.
  5. [Reproducibility] No code, model checkpoints, or inference scripts are provided. Please add a reproducibility statement or a link to the code.

Circularity Check

1 steps flagged

Oracle-style fusion using validation-set metrics makes the headline 0.7799 a fitted upper bound rather than an out-of-sample system score.

specific steps
  1. fitted input called prediction [Section 6 (Results), fusion description and Table 1]
    "For other types of questions, we choose the answer that achieves the highest evaluation score across the relevant evaluation metrics as the final answer. Ultimately, we achieve an optimal final score of 0.7799."

    The 'fusion' system selects, for each open-ended question, the answer that scores best on the validation-set evaluation metrics (BLEU/ROUGE/CIDEr etc.). The reported final score of 0.7799 is therefore not an out-of-sample evaluation of a fixed, coherent system; it is the result of an oracle selection procedure that uses the validation ground-truth labels as a search signal. In other words, the evaluation metric is both the selection criterion and the reported outcome, so the headline score is by construction an optimized value rather than a prediction of a single model's performance. This is equivalent to fitting the final answer set to the validation target and then presenting the resulting score as the system's achievement.

full rationale

The individual system scores (LLaVA-1.5-7B+LoRA, LLaVA-1.5-7B+DoRA, LLaVA-NeXT-7B+LoRA) appear to be honest single-model evaluations on the validation set, and the paper's training/inference pipeline is otherwise self-contained and not dependent on self-citations. However, the paper's strongest claim—the 'top score of 0.7799' and rank 1 on the validation leaderboard—rests on a fusion step that explicitly chooses answers by their validation-set evaluation scores. Because the selection criterion is the metric being reported, the final number is partly constructed from the validation labels and does not represent the out-of-sample performance of a single system. This is a genuine, if localized, circularity: the 'predicted' fusion result is fitted to the target it purports to measure. The rest of the derivation (fine-tuning, depth enrichment, CoT) is not circular. I therefore assign a score of 6, reflecting one central prediction that reduces by construction to the evaluation target.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The central result is an empirical leaderboard score, so the ledger consists mainly of training hyperparameters and assumptions about external datasets and models. No new entities are introduced.

free parameters (7)
  • lora_rank = 8
    Rank of the low-rank adaptation; set by authors, affects model capacity and validation score.
  • lora_alpha = 16
    Scaling factor for LoRA updates; a hyperparameter chosen without reported tuning.
  • learning_rate = 2e-5
    Initial learning rate with cosine schedule; a standard value but still a choice that affects the final result.
  • training_epochs = 1
    One epoch chosen to prevent overfitting; affects model quality.
  • depth_percentile = 75
    75th percentile of per-pixel depth inside the bounding box is used as the representative object depth.
  • depth_window_size = 11x11
    Window around key object coordinates from which depth pixels are aggregated in the validation set.
  • close_far_thresholds = not specified
    The mapping from depth values to textual labels 'close' and 'far' is not given, so thresholds are effectively unspecified free choices.
axioms (4)
  • domain assumption DriveLM-nuScenes annotations and the official evaluation metrics are correct and used as provided.
    The entire score depends on the dataset labels and metric implementation; no verification is provided in the paper.
  • domain assumption Depth Anything monocular depth estimates provide sufficiently accurate relative depth for the objects of interest.
    The depth-text cues injected into prompts come from this external model, whose errors propagate into the VLM input.
  • domain assumption Public LLaVA-1.5/NeXT checkpoints are the intended pretrained models and behave as documented.
    Fine-tuning starts from these external checkpoints; the paper does not verify their provenance or contents.
  • domain assumption Parameter-efficient fine-tuning (LoRA/DoRA) preserves the base model's visual-linguistic capabilities while adapting to the driving domain.
    The validity of the method is taken from the cited papers without additional checks.

pith-pipeline@v1.3.0-alltime-deepseek · 5478 in / 13158 out tokens · 131399 ms · 2026-08-04T17:06:47.160665+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of The System Description of CPS Team for Track on Driving with Language of CVPR 2024 Autonomous Grand Challenge." pith.science (2026). https://pith.science/paper/VL5YX7QC

@misc{pith2026250911071,
  author       = {Pith},
  title        = {Pith review of: The System Description of CPS Team for Track on Driving with Language of CVPR 2024 Autonomous Grand Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VL5YX7QC}},
  note         = {Machine review of arXiv:2509.11071}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

This report outlines our approach using vision language model systems for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge. We have exclusively utilized the DriveLM-nuScenes dataset for training our models. Our systems are built on the LLaVA models, which we enhanced through fine-tuning with the LoRA and DoRA methods. Additionally, we have integrated depth information from open-source depth estimation models to enrich the training and inference processes. For inference, particularly with multiple-choice and yes/no questions, we adopted a Chain-of-Thought reasoning approach to improve the accuracy of the results. This comprehensive methodology enabled us to achieve a top score of 0.7799 on the validation set leaderboard, ranking 1st on the leaderboard.

Figures

Figures reproduced from arXiv: 2509.11071 by Dehui Du, Jinghan Peng, Jingwen Wang, Xing Yu.

Figure 1
Figure 1. Figure 1: Diagram of six images from a sample frame, captured by cameras mounted on the vehicle in six directions: front, front left, front [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The diagram displays three raw images (Top) and their corresponding depth estimation images (Bottom) from a sample frame. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The architecture of our proposed inference framework. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

14 extracted references · 6 linked inside Pith

  1. [1]

    Lan- guage models are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901, 2020

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Sub- biah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakan- tan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Lan- guage models are few-shot learners.Advances in neural in- formation processing systems, 33:1877–1901, 2020. 4

    1901

  2. [2]

    Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom

    Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A mul- timodal dataset for autonomous driving.arXiv preprint arXiv:1903.11027, 2019. 2

    Pith/arXiv arXiv 1903

  3. [3]

    A survey on multimodal large lan- guage models for autonomous driving

    Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang, Kuei-Da Liao, et al. A survey on multimodal large lan- guage models for autonomous driving. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958–979, 2024. 1

    2024

  4. [4]

    A sur- vey of vision-language pre-trained models

    Yifan Du, Zikang Liu, Junyi Li, and Wayne Xin Zhao. A sur- vey of vision-language pre-trained models. InProceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5436–5443. International Joint Conferences on Artificial Intelligence Organization, 2022. Survey Track. 1

    2022

  5. [5]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Represen- tations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net, 2022. 2

    2022

  6. [6]

    The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning.arXiv preprint arXiv:2305.14045, 2023

    Seungone Kim, Se June Joo, Doyoung Kim, Joel Jang, Seonghyeon Ye, Jamin Shin, and Minjoon Seo. The cot collection: Improving zero-shot and few-shot learning of language models via chain-of-thought fine-tuning.arXiv preprint arXiv:2305.14045, 2023. 4

    Pith/arXiv arXiv 2023

  7. [7]

    Large language models are zero-shot reasoners.Advances in neural information pro- cessing systems, 35:22199–22213, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners.Advances in neural information pro- cessing systems, 35:22199–22213, 2022. 4

    2022

  8. [8]

    Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning.arXiv preprint arXiv:2310.03744, 2023. 2

    Pith/arXiv arXiv 2023

  9. [9]

    Visual instruction tuning.Advances in neural information processing systems, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36, 2024. 2

    2024

  10. [10]

    Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353, 2024

    Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation.arXiv preprint arXiv:2402.09353, 2024. 2

    Pith/arXiv arXiv 2024

  11. [11]

    Drivelm: Driving with graph visual ques- tion answering.arXiv preprint arXiv:2312.14150, 2023

    Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual ques- tion answering.arXiv preprint arXiv:2312.14150, 2023. 1

    Pith/arXiv arXiv 2023

  12. [12]

    Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large lan- guage models.Advances in neural information processing systems, 35:24824–24837, 2022. 4

    2022

  13. [13]

    Depth anything: Unleash- ing the power of large-scale unlabeled data.arXiv preprint arXiv:2401.10891, 2024

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Ji- ashi Feng, and Hengshuang Zhao. Depth anything: Unleash- ing the power of large-scale unlabeled data.arXiv preprint arXiv:2401.10891, 2024. 2

    Pith/arXiv arXiv 2024

  14. [14]

    Vision-language models for vision tasks: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence,

    Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey.IEEE Transactions on Pattern Analysis and Machine Intelligence,