REVIEW 4 major objections 4 minor 4 references
Driving with InternVL: Oustanding Champion in the Track on Driving with Language of the Autonomous Grand Challenge at CVPR 2024
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper claims that a general, open-source vision-language model, InternVL-1.5, wins the DriveLM driving-with-language track after full-parameter fine-tuning on multi-view concatenated images and SAM-generated bounding boxes, achieving…
desk verdict A solid, honestly-written competition report whose score is plausible but unverifiable; the SAM-box conversion is the only load-bearing assumption and it is acknowledged, not validated. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a 2-by-3 grid of resized 896x448 camera frames, each prefixed with a text label of the camera direction and then concatenated into one 2688x896 image; the grid is split into twelve 448x448 tiles plus a thumbnail and fed through InternVL. The second mechanism is the annotation conversion: a DriveLM center point is used as a point prompt for Segment Anything, the largest sampled mask is taken as the object, and its bounding box is computed and re-normalized to InternVL's integer 0-1000 coordinate space. The argument that this works rests on the model's pretrained ability to ground language in boxes, so the input format lets the model inherit that capability.
What would settle it
Take a random sample of the training images, draw the SAM-derived bounding boxes for objects whose center points are on traffic lights and other small or partially occluded objects, and compare them with human-labeled boxes; if a substantial fraction have near-zero IoU, the reported score cannot be credited to the claimed clean perception labels, and replacing those boxes should change accuracy.
Extended reading notes
Core claim
The paper's central claim is that a general-purpose vision-language model can dominate an autonomous-driving reasoning benchmark without any task-specific perception head. The authors take InternVL-1.5 and fine-tune it end-to-end on DriveLM-nuScenes, feeding it a single concatenated image made from the six camera views, each labeled with its direction (FRONT LEFT, FRONT, FRONT RIGHT, BACK LEFT, BACK, BACK RIGHT). They replace the dataset's center-point object representation with bounding boxes extracted by prompting Segment Anything from the center point and taking the largest mask, then normalize coordinates to InternVL's 0-1000 token format. Their single best model, InternVL4Drive-v2, achieves a final leaderboard score of 0.6002; a version trained on only about 10% of the data already reaches 0.5862, and the authors state that ensembling the two would score higher.
Load-bearing premise
The automatic labeling step assumes each dataset center point lies on the object's main body, so the largest automatically generated mask is the whole object and its bounding box is a clean training label, an assumption the paper itself notes fails for traffic lights.
Editorial extensions
If this is right
- Under the paper's claim, a task-general open vision-language model with simple input reformatting is enough to outperform purpose-built driving reasoning systems on the competition's metric.
- Because the benchmark gates reasoning questions on correct perception, the SAM-derived bounding boxes directly determine how much reasoning score the model can reach, so better boxes should yield higher scores.
- The paper's ensemble observation implies the final score can be raised further without changing the architecture, since the two versions have complementary accuracy and ChatGPT behavior.
- The failed temporal variant shows that simply adding the previous keyframe does not exploit temporal context in this setup, leaving temporal fusion an open problem.
Reading between the lines
- A natural next experiment, not run in the paper, is to hand-validate the Segment Anything boxes against human annotations; if traffic-light boxes are bad, fixing those labels could push the ChatGPT and Match scores above the reported values.
- The same recipe of labeled multi-view concatenation plus prompt-based box grounding could transfer to other multi-camera visual question answering benchmarks, since it does not depend on task-specific architecture.
- The accuracy drop from v1 to v2 while the ChatGPT score rose suggests the full-data fine-tune trades some exact perception for more fluent or preferred answers; splitting the evaluation by question type would reveal where the trade happens.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports the authors' winning submission to the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge. The method is based on InternVL-1.5, which is fine-tuned with full parameters on the DriveLM-nuScenes dataset. To handle the six surround-view camera images, the authors concatenate them into a 2x3 grid with orientation labels and resize the composite image. They also replace the center-point object representation in DriveLM with bounding boxes obtained by prompting Segment Anything with the center point and taking the largest resulting mask. Their best single model, InternVL4Drive-v2, is reported to achieve a final score of 0.6002 on the competition leaderboard, with Accuracy 0.7339 and ChatGPT score 65.2512, outperforming their v1 model. A preliminary temporal variant scored lower and is attributed to data format issues.
Significance. If the reported leaderboard score is accurate, the paper demonstrates that a general open-source vision-language model, combined with simple preprocessing and automatic annotation, can achieve a leading result on a driving-language benchmark. This is an interesting empirical data point for the community. The paper also honestly acknowledges the limitation of the SAM-based annotation pipeline for cases where the center point is not on the object's main body. However, the scientific significance is limited by the absence of reproducibility artifacts, ablation studies, and validation of the automatic annotation quality. The central claim is an externally evaluated competition score, but the paper does not provide enough evidence to verify the method's contribution beyond that score.
major comments (4)
- [Section 2] The automatic annotation strategy is a load-bearing component of the method, but its quality is not validated. The paper states that the largest SAM mask is consistently selected and that this "works well in most cases," while acknowledging that traffic lights can produce incorrect boxes. Because the competition scoring gates reasoning questions on correct perception (as stated in the Introduction), corrupted bounding boxes could directly lower the reported accuracy and ChatGPT scores. The authors should provide quantitative validation of the annotation pipeline, for example by measuring IoU against a human-annotated subset, presenting a few qualitative examples, or performing an ablation that replaces the SAM boxes with the original center points or with manually corrected boxes.
- [Section 3] The paper attributes the good performance to the concatenation of multi-view images and the replacement of center points with bounding boxes, but no baseline or ablation is provided. Without comparing InternVL4Drive-v2 to a model trained on the original center-point representation, or to a model that processes the six views without concatenation, the reader cannot assess whether these design choices actually contribute to the reported score. The central claim of the paper is the leaderboard result, but the methodological claims are unsupported without these controls.
- [Table 2] The temporal model InternVL4Drive-T is reported to have a much lower final score (0.4600) and the failure is attributed to "data format issues," but no diagnosis is provided. Including this result without any explanation of what the data format issue was, or how it was detected, does not advance the reader's understanding and weakens the credibility of the experimental section. Either the issue should be analyzed and resolved in the paper, or the temporal experiment should be clearly labeled as an inconclusive preliminary exploration.
- [General reproducibility] The manuscript provides no code, checkpoints, annotation scripts, leaderboard link, or frame-level result dump. The reported score of 0.6002 therefore cannot be independently verified or decomposed from the manuscript alone. For a journal publication, the authors should make available the essential implementation details and, if possible, the exact evaluation results or a link to the public leaderboard entry.
minor comments (4)
- [Title and Abstract] The title contains a typo: "Oustanding" should be "Outstanding," and the abstract uses "leadboard" instead of "leaderboard."
- [Section 4] The sentence "Based in this sub-dataset, our model actually achieves all higher score except on the ChatGPT score" is grammatically unclear; it likely means "Based on this sub-dataset, our model achieves higher scores on all metrics except the ChatGPT score."
- [Section 3] The heading "T emporal F usion" contains stray spaces and the section would benefit from a clearer separation of the temporal experiment from the main single-frame method.
- [Figures] Figures 1 and 2 are not referenced in the text; the authors should explicitly refer to them in Sections 3 and 4 where the architecture and the concatenated image are described.
Circularity Check
No circular reasoning: the reported 0.6002 leaderboard score is an externally evaluated empirical result, and the SAM-based box annotation is a stated data preprocessing assumption, not a fitted input renamed as a prediction.
full rationale
The paper makes no analytic derivation that reduces to its inputs. Its central claim is the empirical final score of 0.6002 (Accuracy 0.7339, ChatGPT 65.2512) for InternVL4Drive-v2 on the DriveLM competition leaderboard after full-parameter fine-tuning of InternVL-1.5. That score is produced by the competition's external evaluation protocol on held-out question-answer pairs, not by any equation inside the report, so it cannot be circular in the sense of a prediction being equivalent to a fitted parameter. The v1/v2/T comparisons are ordinary empirical model selection on measured metrics; choosing the variant with the higher leaderboard score does not manufacture that score. The automatic annotation strategy in Section 2, which converts a center point to a bounding box by selecting the largest SAM mask, is a training-label generation heuristic that the authors explicitly note can fail for traffic lights; that is an unvalidated label-quality assumption, not a circular derivation. No load-bearing step is justified by self-citation: the cited works (InternVL, DriveLM, nuScenes, and SAM) are external prior results, and the authors do not invoke their own prior uniqueness theorems or fitted values to force the outcome. The lack of code, weights, annotation-quality validation, and a diagnosis of the temporal model's failure is a reproducibility and verification weakness, not a circularity defect. Therefore the appropriate finding is no significant circularity with score 0.
Assumptions & free parameters
free parameters (2)
- Training hyperparameters =
LR=2e-5, batch size=1024, one epoch
- Image grid and coordinate renormalization =
2x3 grid, 896x448 per view, 2688x896 composite, coordinates in 0-1000
assumptions (3)
- domain assumption The largest SAM mask generated from an object center point corresponds to the complete target object.
- domain assumption InternVL can reason correctly over a 2x3 concatenated composite image with added camera labels and renormalized coordinates.
- domain assumption The competition metrics, including Accuracy, ChatGPT, and text-similarity scores, are a valid proxy for driving perception and reasoning capability.
Cite this review
Pith. "Pith review of Driving with InternVL: Oustanding Champion in the Track on Driving with Language of the Autonomous Grand Challenge at CVPR 2024." pith.science (2026). https://pith.science/paper/2XZATT27
@misc{pith2026241207247,
author = {Pith},
title = {Pith review of: Driving with InternVL: Oustanding Champion in the Track on Driving with Language of the Autonomous Grand Challenge at CVPR 2024},
year = {2026},
howpublished = {\url{https://pith.science/paper/2XZATT27}},
note = {Machine review of arXiv:2412.07247}
}
read the original abstract
This technical report describes the methods we employed for the Driving with Language track of the CVPR 2024 Autonomous Grand Challenge. We utilized a powerful open-source multimodal model, InternVL-1.5, and conducted a full-parameter fine-tuning on the competition dataset, DriveLM-nuScenes. To effectively handle the multi-view images of nuScenes and seamlessly inherit InternVL's outstanding multimodal understanding capabilities, we formatted and concatenated the multi-view images in a specific manner. This ensured that the final model could meet the specific requirements of the competition task while leveraging InternVL's powerful image understanding capabilities. Meanwhile, we designed a simple automatic annotation strategy that converts the center points of objects in DriveLM-nuScenes into corresponding bounding boxes. As a result, our single model achieved a score of 0.6002 on the final leadboard.
Figures
Reference graph
Works this paper leans on
-
[1]
nuscenes: A multimodal dataset for autonomous driving
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. 2020
work page 2020
-
[2]
Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Zhong Muyan, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312.14238 , 2023
arXiv 2023
-
[3]
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015--4026, 2023
work page 2023
-
[4]
Drivelm: Driving with graph visual question answering
Chonghao Sima, Katrin Renz, Kashyap Chitta, Li Chen, Hanxue Zhang, Chengen Xie, Ping Luo, Andreas Geiger, and Hongyang Li. Drivelm: Driving with graph visual question answering. arXiv preprint arXiv:2312.14150 , 2023
arXiv 2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.