Pith. sign in

REVIEW 6 major objections 5 minor 45 references

The paper's central claim is that object-level counterfactual ablation, via photorealistic inpainting, isolates each object's causal effect on a driving model's predicted trajectory, and that the trajectory expert is a nonlinear amplifier o

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 19:28 UTC pith:5UBDGMCK

load-bearing objection A useful object-level counterfactual audit framework for VLA driving models, but the evaluation protocol may be severely out-of-distribution; the headline numbers need confirmation before they are trusted. the 6 major comments →

arxiv 2607.16938 v1 pith:5UBDGMCK submitted 2026-07-18 cs.CV cs.AIcs.LGcs.RO

What Do They See? Interpreting Complex Road Scenarios Through the Eyes of Vision-Language-Action Models for Safe and Trustworthy Autonomous Vehicle Learning

Kalpana Panda , Wesley Maia , Vinti Agarwal , Ross Greer This is my paper
classification cs.CV cs.AIcs.LGcs.RO
keywords counterfactual attributionvision-language-action modeltrajectory predictionphotorealistic inpaintingmechanistic interpretabilityautonomous driving safetyobject importancedistributional shift metrics
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

CVAA is a counterfactual ablation pipeline: for each detected object in a front-camera driving image, the object is removed with photorealistic inpainting, the Alpamayo 1 trajectory predictor is rerun, and the shift in the predicted trajectory distribution is measured. The paper's central claim is that this isolates the causal effect of each object's presence on the model's planning behavior, and that the resulting object rankings can audit a driving model's trustworthiness. Across 210 nuScenes scenes, class-level effects are intuitive—buses, trucks, persons, and red traffic lights cause the largest deviations, and mask area correlates with influence—but individual object ranks are unstable across seeds. White-box tracing shows the influence travels through at least four pathways, and the trajectory expert acts as a nonlinear amplifier, turning near-zero handoff perturbations into large output shifts, sometimes via a 'silent' route that bypasses the trajectory token. The paper argues this two-stage path, from behavioral attribution to mechanistic analysis, is what explainable and trustworthy driving systems need.

Core claim

The paper's core claim: object-level counterfactual attribution, built from photorealistic inpainting and mean-trajectory shift, makes each scene object's causal influence measurable in a vision-language-action trajectory predictor. It reports that ego-path vehicles and pedestrians dominate as expected; traffic lights, especially red ones, shift trajectories disproportionately to their image size; mask area predicts influence; and 49.3% of removals move the best trajectory closer to ground truth. White-box tracing then identifies four propagation regimes, and shows the trajectory expert acts as a nonlinear amplifier that turns near-zero handoff perturbations into large output shifts, includi

What carries the argument

The central mechanism is the counterfactual semantic set: for each scene, each detected object is removed independently via photorealistic inpainting, producing paired original/inpainted images. Two distributional metrics—Average Deviation (AD), the mean distance between original and counterfactual predicted trajectories over the horizon, and Final Deviation (FD), the terminal-point distance—turn each removal into an object-level causal score. The white-box stage uses layer-wise cosine deltas between original and inpainted hidden states, Logit Lens projections, and attention weights from the trajectory handoff token, classifying variants into four regimes. The named entity 'trajectory expert

Load-bearing premise

The load-bearing premise is that photorealistic inpainting removes only the intended object and leaves the rest of the image indistinguishable from real driving data, so any trajectory shift is caused by the object's absence rather than by inpainting artifacts or changes to surrounding context.

What would settle it

Take a scene where a small, peripheral object such as a distant tree is removed by photorealistic inpainting, and compare the trajectory shift to a black-box removal (solid color fill) of the same object at the same mask; if the two shifts differ widely, the 'causal isolation' claim is contaminated by inpainting artifacts. A stronger test: build a synthetic scene with a known ground-truth causal object, add a decoy object, run the CVAA pipeline, and check whether AD/FD ranks match the true cause; if the decoy consistently outranks the cause, the attribution metric is not measuring causal influ

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Class-level sensitivities (large vehicles, red lights, persons near the path) can be turned into automated safety checks: a driving model that does not register these removal-induced shifts should be flagged before deployment.
  • Audits that only examine the final trajectory token will miss a class of safety-relevant influences; monitoring the trajectory expert's amplification is necessary to catch silent pathway effects.
  • The instability of single-object ranks across seeds warns that object-level explanations should not be used as the sole basis for claims about why a model acted; scene-level or ensemble attributions are more robust.
  • Since 49.3% of object removals improve ground-truth accuracy, the model appears to over-rely on many objects; interventions that reduce this over-reliance could improve planning accuracy.
  • The failure of Logit Lens indicates that interpretability for VLA driving models must be developed in continuous feature space, not through language vocabulary projection.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the expert amplifier claim holds, then tiny nondeterminism in the VLM—different sampling seeds, slightly different prompts, or numerical noise—could be inflated into meter-level trajectory differences; a stress test that perturbs the handoff token by controlled epsilon values would test this directly.
  • The silent pathway suggests that an object's influence may be distributed across many hidden features rather than concentrated in one token; this predicts that single-token activation patching will under-estimate attributable objects, and multi-token or distributed interventions would recover them.
  • Since red-state traffic lights produced larger shifts than green-state ones while the object class is identical, the model is likely tracking the signal's semantic state, not just its presence; an inpainting run that changes only the light's color while keeping footprint identical would separate 'class presence' from 'state meaning'.
  • The z-score dominance in most scenes suggests a scene-level 'attribution concentration' score could be a cheap safety monitor: scenes where one object dominates the deviation landscape may deserve extra human review, even if that object is not the obvious hazard.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 5 minor

Summary. The paper proposes Counterfactual Vision Action Analysis (CVAA), a counterfactual ablation framework that removes individual detected objects from front-camera images using photorealistic inpainting, and measures the resulting shift in the trajectory distribution of the Alpamayo 1 VLA model via two new metrics, Average Deviation (AD) and Final Deviation (FD). A benchmark dataset, Counter-nuScenes, is constructed from 210 nuScenes scenes yielding 3,062 counterfactual image pairs. The black-box results are used to rank object classes and object instances by causal influence, and a white-box analysis of hidden-state deltas, Logit Lens, and attention weights identifies four propagation regimes and claims that the trajectory expert acts as a nonlinear amplifier of small perturbations. The paper's headline claims are that vehicles and pedestrians in the model's path dominate causal influence, that traffic lights exert disproportionate influence, and that a 'silent' pathway bypasses the trajectory handoff token.

Significance. If correct, this work would provide a valuable object-level counterfactual attribution method for VLA-based driving models, a reusable benchmark dataset, and one of the first white-box (mechanistic) analyses of a trajectory-predicting VLA. The combination of a photorealistic inpainting intervention with distributional-shift metrics is a promising direction, and the identification of 'silent' propagation regimes is an interesting, falsifiable finding. However, several load-bearing methodological issues and internal inconsistencies in the current manuscript prevent confidence in the quantitative results.

major comments (6)
  1. [§4 vs §3.1] The evaluation protocol is internally inconsistent. §3.1 states frames are retained only if preceded by 1.6 s of continuous pose data, 'matching the history window expected by the model,' whereas §4 states the model receives a single still image and inputs from all cameras except the front camera are zeroed. If Alpamayo 1 expects a temporal history or multi-camera context, every AD/FD value and every hidden-state delta in §5 is measured on out-of-distribution inputs. Please justify that the model can be validly run with a single still image and zeroed auxiliary cameras, or rerun the evaluation in the intended input regime. This concern affects all of the paper's quantitative claims.
  2. [§5.1 / Table 3] The main text contains a literal placeholder: 'Class bus seems to produce the highest deviation (AD=X, Ym).' This is not a finished result. Separately, §5.1 says 'Of 2,975 objects,' while the dataset description and the sum of per-class counts in Table 3 both give 3,062 objects. Reconcile the object count and clarify whether any counterfactual pairs were filtered and why; the percentages in the text depend on this number.
  3. [§3.2 / §5] The causal claim rests on the assumption that photorealistic inpainting removes only the target object and leaves the rest of the image in-distribution (§3.2). No validation is provided: no human evaluation, no artifact metric (e.g., FID/LPIPS on the masked region), and no analysis of failure cases. Without this, AD/FD differences could be induced by inpainting artifacts rather than by the object's absence. Please add such validation or explicitly discuss its absence as a limitation.
  4. [§1, §2, §5.2] The paper repeatedly advertises the use of sparse autoencoders and activation patching (abstract, §1, §2), but the white-box experiments reported in §4 and §5.2 only compute cosine deltas, Logit Lens, attention weights, and KL maps. No sparse-autoencoder or activation-patching experiment appears in the manuscript. Either add these analyses or revise the claims to match what was actually done.
  5. [§5.2] The 'Expert Amplification' ratios ('mean 115–180×, median 47–106×' and 'ratios exceed 1000×') are reported without any definition of the amplification metric. Please define precisely what is being ratioed (e.g., output AD divided by handoff delta), report per-class error bars or distributions, and clarify whether the '100% of variants' claim refers to all 3,062 counterfactual pairs or a subset. Given the OOD protocol concern, these ratios need scrutiny.
  6. [§5.1 / Abstract] Per-class AD/FD values in Table 3 are reported without confidence intervals or significance tests, yet the abstract claims traffic lights 'exert disproportionate effect relative to their image footprint.' No footprint-normalized analysis is presented; raw mean AD alone does not establish disproportionate per-pixel influence. Either add a per-area or per-mask-size normalized analysis or soften the claim to match the presented evidence.
minor comments (5)
  1. [§5.1] The rank-stability text says '5.6% of objects keep their rank with respect to AD,' but Table 1 reports 5.7%. Align the numbers.
  2. [§5.1] The reference to 'Section 1' near the discussion of rank stability should be 'Table 1'.
  3. [Abstract / throughout] The dataset name is inconsistently typeset as 'Counter -nuScenes' with an extra space in the abstract; use 'Counter-nuScenes' consistently.
  4. [§4] The model description says '180 tokens per camera image after spatial merging' and a 3006-token sequence. Clarify how the tokens from the (zeroed) non-front cameras are handled—are they zeroed embeddings or simply absent? This matters for the OOD concerns.
  5. [§3.2] It is stated that an 'open-source ablation pipeline based on LaMa combined with FLUX Fill 1.0' is maintained, but the manuscript does not state which inpainting tool generated the reported Counter-nuScenes images. Please specify the exact tool and version used for the main results.

Circularity Check

0 steps flagged

No significant circularity: AD/FD are model-relative counterfactual shift metrics, and the few self-citations are background, not load-bearing.

full rationale

The central CVAA claim is that removing an object and measuring AD/FD against the model's own original trajectory distribution quantifies counterfactual influence. This is a definition of the metric, not a fitted parameter later relabeled as a prediction; AD and FD are computed directly from the two forward passes and are not themselves used to construct the counterfactual inputs. The white-box four-regime analysis splits variants by median δ_traj and |ΔminADE| and then reports that silent variants (low handoff delta) can still show large output AD; that is an empirical observation about the resulting quadrant, not a tautology, and the 'expert amplification' ratio is a measured gain, not an output forced by definition. The paper explicitly limits its own claims (Section 7) by noting these probes 'cannot explain why, nor identify which internal computations are causally responsible,' so the causal language is hedged. Self-citations [18,19,20] and [25] are used only to motivate why pixel saliency can be artifact-prone and to distinguish AD/FD from minADE/minFDE; they do not supply any theorem or fitted value on which the Counter-nuScenes results depend. The single-image protocol in Section 4 is a validity risk relative to the stated 1.6 s history window, but it is an out-of-distribution concern, not a circularity. Overall, the derivation chain is self-contained: the empirical claims reduce to measurements of model output shifts, not to the definitions of those shifts.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

No new physical or mathematical entities are postulated; the framework introduces a dataset, metrics, and descriptive categories. The main free parameters are the data-dependent thresholds used in the white-box regime classification and the high-AD subset analysis.

free parameters (2)
  • median split thresholds for four propagation regimes = medians of δ_traj and |ΔminADE| (not numerically stated)
    The white-box classification into Coupled/Transparent/Decoupled/Silent variants is defined by whether δ_traj and |ΔminADE| are above or below their median values (§5.2); these thresholds are fit to the dataset and would change the regime proportions if moved.
  • high-AD threshold = 0.516 m
    Objects in the top 25% of AD are labeled high-AD (§5.1); the cutoff is data-dependent and used to compute the AD-vs-ΔminADE correlation on a subset.
axioms (5)
  • domain assumption YOLOv8 object detections are accurate and complete (no missed objects)
    Used to select frames and prompt SAM2 masks (§3.1); missed objects would be absent from counterfactual sets and rankings.
  • domain assumption SAM2 masks tightly cover the target object without bleeding
    Masks define what is inpainted; bleeding would remove context or leave object fragments (§3.1).
  • domain assumption Photorealistic inpainting yields images in-distribution and removes only the target object
    Central to interpreting output shifts as causal (§3.2); no human evaluation or artifact metric is provided.
  • domain assumption Alpamayo 1 accepts a single still front-camera image with other cameras zeroed as a valid input
    The model is described as expecting a history window; evaluation feeds one still image (§3.1, §4). This may be out-of-distribution.
  • domain assumption Projecting bounding boxes onto token grids accurately identifies object tokens
    White-box layer-wise deltas are computed at token positions corresponding to the object (§4); misalignment would mislocate signals.

pith-pipeline@v1.3.0-alltime-deepseek · 11902 in / 11597 out tokens · 112022 ms · 2026-08-01T19:28:52.678759+00:00 · methodology

0 comments
read the original abstract

End-to-end autonomous driving models are now able to navigate complex road scenarios, mapping raw sensor observations directly to observed paths for open-loop evaluation and often effective driving in closed-loop evaluation. Yet the internal logic of these safety-critical systems remains largely opaque, due to the complexity of traffic scenes. We propose a counterfactual ablation framework called Counterfactual Vision Action Analysis (CVAA) that systematically removes individual detected objects from front-camera images using photorealistic generative inpainting to prepare counterfactual sets to evaluate the difference in the model's response. This isolates the causal effect of each object's presence on the model's planning behaviour. Applied to the Alpamayo 1 trajectory predictor across 210 nuScenes driving scenes, we create a dataset Counter -nuScenes, using which we see that vehicles and pedestrians within the model's 'path' dominate causal influence as expected, while traffic lights, as expected, exert disproportionate effect relative to their image footprint. However, we also find cases where the model responds strongly to objects a human driver would consider irrelevant. This brings forth a deeper question: does the model itself view the scene as a sum of individual objects influencing the outcome, or does it encode an entirely different set of internal features that do not correspond to human-legible scene elements? To further understand this, we compare intermediate representations of original and inpainted image pairs using mechanistic interpretability techniques and examine the effect of the removal through the various model layers. Together, these two stages offer a path from behavioral auditing to representational understanding, creating explainable driving systems and solidifying human-AI trust.

Figures

Figures reproduced from arXiv: 2607.16938 by Kalpana Panda, Ross Greer, Vinti Agarwal, Wesley Maia.

Figure 1
Figure 1. Figure 1: Counter-nuScenes and CVAA construction pipeline. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Scene Explorer — the interactive black-box visualisation dashboard for [PITH_FULL_IMAGE:figures/full_fig_p013_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Mean hidden-state delta curves (1 − cossim, original vs. inpainted) by object category across all LM layers and vision encoder blocks. Top left: object-token delta, averaged over patch positions spatially covering the removed object. Top right: all-visual delta, averaged over all 180 visual tokens; when this substantially exceeds the object-token delta, the removal is rippling into surrounding context. Bot… view at source ↗
Figure 4
Figure 4. Figure 4: Distribution of the z-score of the maximum-AD object per scene, [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

45 extracted references · 1 canonical work pages

  1. [1]

    Geiger, P

    A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 3354–3361, 2012. doi:10.1109/CVPR.2012.6248074

    arXiv 2012

  2. [2]

    Caesar, V

    H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11618–11628, 2020. doi:10.1109/CVPR42600.2020.01164

    arXiv 2020

  3. [3]

    P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V . Patnaik, P. Tsui, J. Guo, Y . Zhou, Y . Chai, B. Caine, V . Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, S. Zhao, S. Cheng, Y . Zhang, J. Shlens, Z. Chen, and D. Anguelov. Scalability in perception for autonomous driving: Waymo open dataset.arXiv, 2019

    2019

  4. [4]

    F. Yu, H. Chen, X. Wang, W. Xian, Y . Chen, F. Liu, V . Madhavan, and T. Darrell. BDD100K: A diverse driving dataset for heterogeneous multitask learning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2636–2645, 2020

    2020

  5. [5]

    Y . Hu, J. Yang, L. Chen, K. Li, C. Sima, X. Zhu, S. Chai, S. Du, T. Lin, W. Wang, L. Lu, X. Jia, Q. Liu, J. Dai, Y . Qiao, and H. Li. Planning-oriented autonomous driving. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17853–17862, 2023

    2023

  6. [6]

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao. DriveGPT4: Interpretable end-to-end autonomous driving via large language model, 2023

    2023

  7. [7]

    H. Shao, Y . Hu, L. Wang, S. L. Waslander, Y . Liu, and H. Li. LMDrive: Closed-loop end- to-end driving with large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024

    2024

  8. [8]

    C. Sima, K. Renz, K. Chitta, L. Chen, H. Zhang, C. Xie, J. Beisswenger, P. Luo, A. Geiger, and H. Li. DriveLM: Driving with graph visual question answering, 2024

    2024

  9. [9]

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, W. Zhan, P. Wang, H. Wang, K. K. Zhao, et al. DriveVLM: The convergence of autonomous driving and large vision-language models, 2024

    2024

  10. [10]

    EMMA: End-to-end multimodal model for autonomous driving, 2024

    Waymo Research. EMMA: End-to-end multimodal model for autonomous driving, 2024

    2024

  11. [11]

    de Haan, D

    P. de Haan, D. Jayaraman, and S. Levine. Causal confusion in imitation learning. InAdvances in Neural Information Processing Systems (NeurIPS), 2019

    2019

  12. [12]

    Bansal, A

    M. Bansal, A. Krizhevsky, and A. Ogale. ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst, 2019

    2019

  13. [13]

    Roelofs, L

    R. Roelofs, L. Sun, B. Caine, K. S. Refaat, B. Sapp, S. Ettinger, and W. Chai. CausalAgents: A robustness benchmark for motion forecasting using causal relationships. InProceedings of the International Conference on Learning Representations (ICLR), 2023

    2023

  14. [14]

    Simonyan, A

    K. Simonyan, A. Vedaldi, and A. Zisserman. Deep inside convolutional networks: Visualis- ing image classification models and saliency maps, 2014. URLhttps://arxiv.org/abs/ 1312.6034

    Pith/arXiv arXiv 2014

  15. [15]

    R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626, 2017. doi:10.1109/ICCV .2017.74

    doi:10.1109/iccv 2017

  16. [16]

    Petsiuk, A

    V . Petsiuk, A. Das, and K. Saenko. RISE: Randomized input sampling for explanation of black-box models. InProceedings of the British Machine Vision Conference (BMVC), 2018. 10

    2018

  17. [17]

    Lundberg and S.-I

    S. Lundberg and S.-I. Lee. A unified approach to interpreting model predictions, 2017. URL https://arxiv.org/abs/1705.07874

    Pith/arXiv arXiv 2017

  18. [18]

    Greer, J

    R. Greer, J. Isa, N. Deo, A. Rangesh, and M. M. Trivedi. On salience-sensitive sign classifica- tion in autonomous vehicle path planning: Experimental explorations with a novel dataset. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 636–644, 2022

    2022

  19. [19]

    Greer, A

    R. Greer, A. Gopalkrishnan, J. Landgren, L. Rakla, A. Gopalan, and M. Trivedi. Robust traffic light detection using salience-sensitive loss: Computational framework and evaluations. In 2023 IEEE Intelligent Vehicles Symposium (IV), pages 1–7. IEEE, 2023

    2023

  20. [20]

    Greer, A

    R. Greer, A. Gopalkrishnan, N. Deo, A. Rangesh, and M. Trivedi. Salient sign detection in safe autonomous driving: Ai which reasons over full visual context. In27th International Technical Conference on the Enhanced Safety of Vehicles (ESV) National Highway Traffic Safety Administration, number 23-0333, 2023

    2023

  21. [21]

    Chang, E

    C.-H. Chang, E. Creager, A. Goldenberg, and D. Duvenaud. Explaining image classifiers by counterfactual generation. InProceedings of the International Conference on Learning Representations (ICLR), 2019

    2019

  22. [22]

    Zhang, W

    R. Zhang, W. Li, P. Wang, C. Guan, J. Fang, Y . Song, J. Yu, B. Chen, W. Xu, and R. Yang. AutoRemover: Automatic object removal for autonomous driving videos. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12853–12861, 2020. doi: 10.1609/aaai.v34i07.6982

    work page doi:10.1609/aaai.v34i07.6982 2020

  23. [23]

    Suvorov, E

    R. Suvorov, E. Logacheva, A. Mashikhin, A. Remizova, A. Ashukha, A. Silvestrov, N. Kong, H. Goka, K. Park, and V . Lempitsky. Resolution-robust large mask inpainting with fourier convolutions, 2021. URLhttps://arxiv.org/abs/2109.07161

    Pith/arXiv arXiv 2021

  24. [24]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R ¨adle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Doll ´ar, and C. Feichtenhofer. Sam 2: Segment anything in images and videos, 2024. URLhttps: //arxiv.org/abs/2408.00714

    Pith/arXiv arXiv 2024

  25. [25]

    Greer, N

    R. Greer, N. Deo, and M. Trivedi. Trajectory prediction in autonomous driving with a lane heading auxiliary loss.IEEE Robotics and Automation Letters, 6(3):4907–4914, 2021

    2021

  26. [26]

    Elhage, N

    N. Elhage, N. Nanda, C. Olsson, T. Henighan, N. Joseph, B. Mann, A. Askell, Y . Bai, A. Chen, T. Conerly, N. DasSarma, D. Drain, D. Ganguli, Z. Hatfield-Dodds, D. Hernan- dez, S. Johnston, A. Jones, J. Kernion, L. Lovitt, K. Ndousse, D. Amodei, T. Brown, J. Clark, J. Kaplan, S. McCandlish, and C. Olah. A mathematical framework for transformer circuits. ht...

    2021

  27. [27]

    K. Meng, D. Bau, A. Andonian, and Y . Belinkov. Locating and editing factual associations in GPT. InAdvances in Neural Information Processing Systems (NeurIPS), 2022

    2022

  28. [28]

    Bricken, A

    T. Bricken, A. Templeton, J. Batson, B. Chen, A. Jermyn, T. Conerly, N. L. Turner, C. Anil, C. Denison, A. Askell, R. Lasenby, Y . Wu, S. Kravec, N. Schiefer, T. Maxwell, N. Joseph, Z. Hatfield-Dodds, A. Tamkin, K. Nguyen, B. McLean, J. E. Burke, T. Hume, S. Carter, T. Henighan, and C. Olah. Towards monosemanticity: Decomposing lan- guage models with dict...

    2023

  29. [29]

    Cunningham, A

    H. Cunningham, A. Ewart, L. Riggs, R. Huben, and L. Sharkey. Sparse autoencoders find highly interpretable features in language models, 2023. 11

    2023

  30. [30]

    X. Zhou, X. Han, F. Yang, Y . Ma, V . Tresp, and A. Knoll. OpenDriveVLA: Towards end-to- end autonomous driving with large vision language action model. InProceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 13782–13790, 2026. doi:10.1609/aaai. v40i16.38386

    doi:10.1609/aaai 2026

  31. [31]

    Z. Zhou, T. Cai, S. Z. Zhao, Y . Zhang, Z. Huang, B. Zhou, and J. Ma. AutoVLA: A vision- language-action model for end-to-end autonomous driving with adaptive reasoning and re- inforcement fine-tuning. InAdvances in Neural Information Processing Systems (NeurIPS), 2025

    2025

  32. [32]

    Zhang, Z

    D. Zhang, Z. Yuan, Z. Chen, C.-T. Liao, Y . Chen, F. Shen, Q. Zhou, and T.-S. Chua. Reasoning- VLA: A fast and general vision-language-action reasoning model for autonomous driving, 2025

    2025

  33. [33]

    Alpamayo-R1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail, 2025

    NVIDIA Research. Alpamayo-R1: Bridging reasoning and action prediction for generalizable autonomous driving in the long tail, 2025

    2025

  34. [34]

    NVIDIA Alpamayo: Open reasoning vision-language-action models for au- tonomous vehicle development.https://developer.nvidia.com/drive/alpamayo, 2026

    NVIDIA. NVIDIA Alpamayo: Open reasoning vision-language-action models for au- tonomous vehicle development.https://developer.nvidia.com/drive/alpamayo, 2026

    2026

  35. [35]

    R. C. Fong and A. Vedaldi. Interpretable explanations of black boxes by meaningful perturba- tion. InProceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3429–3437, 2017

    2017

  36. [36]

    Goldshmidt

    R. Goldshmidt. Understanding what vision-language models see in traffic: Pixelshap for object-level attribution in autonomous driving. InProceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pages 1794–1802. IEEE Computer So- ciety, 2025. doi:10.1109/ICCVW69036.2025.00189

    arXiv 2025

  37. [37]

    Golovanevsky, W

    M. Golovanevsky, W. Rudman, V . Palit, R. Singh, and C. Eickhoff. What do vlms notice? a mechanistic interpretability pipeline for gaussian-noise-free text-image corruption and evalua- tion, 2025. URLhttps://arxiv.org/abs/2406.16320

    Pith/arXiv arXiv 2025

  38. [38]

    M. Pach, S. Karthik, Q. Bouniot, S. Belongie, and Z. Akata. Sparse autoencoders learn monosemantic features in vision-language models. InAdvances in Neural Information Pro- cessing Systems (NeurIPS), 2025

    2025

  39. [39]

    Jocher, A

    G. Jocher, A. Chaurasia, and J. Qiu. YOLOv8 by ultralytics.https://github.com/ ultralytics/ultralytics, 2023

    2023

  40. [40]

    Gemini image generation and editing.https://deepmind.google/ technologies/gemini/, 2024

    Google DeepMind. Gemini image generation and editing.https://deepmind.google/ technologies/gemini/, 2024

    2024

  41. [41]

    B. F. Labs. Flux.https://github.com/black-forest-labs/flux, 2024

    2024

  42. [42]

    Palit, R

    V . Palit, R. Pandey, A. Arora, and P. P. Liang. Towards vision-language mechanistic inter- pretability: A causal tracing tool for blip. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 2856–2861, October 2023

    2023

  43. [43]

    H ¨aon, K

    B. H ¨aon, K. Stocking, I. Chuang, and C. Tomlin. Mechanistic interpretability for steering vision-language-action models, 2025. URLhttps://arxiv.org/abs/2509.00328

    Pith/arXiv arXiv 2025

  44. [44]

    C. Neo, L. Ong, P. Torr, M. Geva, D. Krueger, and F. Barez. Towards interpreting visual information processing in vision-language models. In Y . Yue, A. Garg, N. Peng, F. Sha, and R. Yu, editors,International Conference on Learning Representations, volume 2025, pages 57172–57189, 2025. URLhttps://proceedings.iclr.cc/paper_files/paper/2025/ file/900fb3439e...

    2025

  45. [45]

    Interpreting gpt: The logit lens.https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens, 2020

    nostalgebraist. Interpreting gpt: The logit lens.https://www.lesswrong.com/posts/ AcKRB8wDpdaN6v6ru/interpreting-gpt-the-logit-lens, 2020. LessWrong. Ac- cessed: 2026-05-29. A Supplementary Tables and Figures This appendix contains supporting visualisations and tables for the white-box and black-box analy- ses described in Sections 5.1 and 5.2. A.1 Class-...

    2020