Pith. sign in

REVIEW 3 major objections 5 minor 49 references

Unsupervised Discovery of Failure Taxonomies from Deployment Logs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An unsupervised pipeline turns raw failure logs into expert-aligned failure taxonomies that improve targeted data collection and runtime monitoring.

desk verdict New problem and a plausible pipeline, but the expert-alignment evidence leans on LLM self-judgment until human validation is added. read the letter →

arxiv 2506.06570 v4 pith:5IGHIK5S submitted 2025-06-06 cs.RO

Aryaman Gupta , Yusuf Umut Ciftci , Somil Bansal This is my paper
classification cs.RO
keywords unsupervisedfailuretaxonomydiscoveryvision-languagemodelsexplanationsemanticclusteringdeploymentlogsruntimemonitoringtargeteddatacollectionroboticfailures
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that the recurring causes of a robot's failures can be discovered automatically from raw deployment logs, without any human-labelled failure categories. The proposed pipeline has vision-language models write structured explanations of why each recorded trajectory failed, then clusters those explanations in a semantic reasoning space to form an interpretable taxonomy of failure modes. Across robotic manipulation, indoor navigation, and dashcam crash footage, the discovered taxonomies align with expert-defined categories that were built independently, such as an eight-mode manipulation taxonomy and a government pre-crash typology. Because the taxonomy is computer-readable, it can be fed back into the system: the paper shows it improves runtime failure detection, giving earlier warnings and better out-of-distribution performance than supervised classifiers, and that collecting training data in taxonomy-identified failure regions cuts collision rates more than uniform data collection. If correct, the work automates a manual expert-review bottleneck in safety engineering for deployed robots.

What carries the argument

The load-bearing object is the semantic reasoning space: the space of structured, natural-language failure explanations produced by a vision-language model from each trajectory, in which clustering is performed. Clustering in this space rather than in raw pixel or visual-embedding space is what lets the pipeline find categories like 'Insufficient Following Distance' instead of visually similar but causally unrelated groups. Two mechanisms carry the argument: bidirectional embedding-space change-point downsampling, which uses cosine similarity in a CLIP vision-language embedding space to keep only semantically distinct frames around the failure event so that failure-relevant transitions fit inside the model's context window; and ensemble-and-refine taxonomy aggregation, in which a large language model generates several different clusterings of the explanations from rephrased prompts and is then prompted to reconcile them into one consolidated taxonomy. The same class of model performs the final mapping of each explanation to a taxonomy node, with unmatched explanations flagged as outliers that can seed future taxonomy refinement.

What would settle it

Take a sample of the failure trajectories and have independent human experts label the failure cause and group the trajectories into categories; if the discovered clusters agree with the human panel no better than chance, the taxonomy quality claims fail. A cheaper check is to re-run the entire pipeline with a different model family as the similarity judge: if the reported alignment scores (semantic alignment 0.958, taxonomy coverage 1.0, assignment F1 85.53%) collapse toward chance, then the scores measure the judge's self-consistency rather than recoverable failure structure.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that failure taxonomies are an emergent by-product of semantic aggregation: once individual failure episodes are converted into natural-language explanations that state the inferred cause (for instance, that a robot dropped a pot while carrying it because the grasp was unstable), the cluster structure of those explanations in language space is the failure taxonomy. The method makes this concrete by compressing each trajectory around the failure frame with a CLIP-based change-point selector that keeps only semantically distinct frames, prompting a vision-language model with chain-of-thought to write the explanation, having a large language model cluster all explanations into named, described, keyword-tagged failure modes, and reconciling several independent clusterings into a consensus taxonomy. The paper reports that the resulting taxonomy covers all eight expert-defined RoboFail failure modes (taxonomy coverage 1.0 and semantic alignment 0.958), that trajectory-to-taxonomy assignment reaches 85.53% weighted F1, and that the same unsupervised procedure rediscovers known driving and navigation failure categories from raw video. It presents the result as unsupervised in the strong sense: no predefined labels and no task-specific engineering, only the reasoning capacities of pre-trained models applied to deployment logs.

Load-bearing premise

The whole pipeline rests on trusting the language models themselves: the vision-language model's explanations of why a trajectory failed, and the language model's judgments of which explanations are similar, are taken as reliable enough to define the taxonomy and to measure how good it is, with no human check on either step.

Editorial extensions

If this is right

  • Failure-mode discovery no longer requires a human to watch logs: the pipeline produces named, described, and keyword-tagged failure categories straight from raw trajectories.
  • The discovered taxonomy is a usable artifact for safety workflows, improving runtime monitoring F1 and lead time over no-taxonomy and supervised baselines, especially on out-of-distribution data.
  • Failure-guided data collection is more sample-efficient than uniform collection: fine-tuning a navigation policy on taxonomy-targeted regions reduced failure rate from 46% to 18%, versus 34% with an equal amount of uniform data.
  • The consensus-aggregation step is what pushes the taxonomy to full expert coverage; a single clustering run scores lower, so ensemble reasoning over clusterings is integral to the method's alignment with expert categories.
  • The same unsupervised pipeline transfers across manipulation, driving, and navigation with no task-specific engineering, suggesting it generalizes to new deployment settings as-is.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the pipeline scales to fleet-scale logs as the paper projects, deployed autonomous systems could continuously mine their own near-misses into an evolving, auditable safety-case vocabulary rather than relying on pre-deployment testing alone.
  • A test the paper does not run is model-confounding: because the model family that generates the explanations also judges cluster similarity, swapping the judging model for a different family, or adding a human-annotated cluster-agreement study, would reveal whether the reported alignment scores (semantic alignment 0.958, taxonomy coverage 1.0) reflect true semantic structure or self-consistency.
  • The outlier-flagging step implies a live version of the system could detect novel failure modes as they emerge in deployment and propose taxonomy updates, turning the taxonomy from a one-time artifact into a dynamically maintained model of the robot's failure landscape.
  • Grounding explanations in causal validation or formal safety-analysis methods of the kind the authors name (STPA, FRAM) could make the taxonomy predictive rather than descriptive, linking a cluster such as 'Narrow Gap Misjudgement' to geometric conditions testable by reachability-style analysis.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces an unsupervised pipeline for discovering failure taxonomies from multimodal robotic deployment logs. The method downsamples failure-centered trajectories using CLIP similarity, generates structured failure explanations with a vision-language model, clusters these explanations via an ensemble-and-refine LLM procedure, and assigns trajectories to the resulting failure modes. The approach is evaluated on RoboFail manipulation data, Nexar dashcam crash videos, and indoor robot navigation, with additional experiments on runtime failure monitoring and targeted data collection for policy refinement. The central claim is that the pipeline produces interpretable, expert-aligned failure taxonomies without predefined labels and that these taxonomies provide measurable downstream safety benefits.

Significance. If the validation were sound, the contribution would be practically valuable: it automates a labor-intensive failure-analysis step and demonstrates closed-loop safety improvements, including a reactive safeguard intervention and a reduction in failure rate from 46% to 18% after taxonomy-guided data collection. The paper's strengths include multi-domain evaluation, comparisons to BERTopic and fine-tuned VLMs, and concrete downstream experiments. However, the central expert-alignment evidence is weakened by the use of the same LLM family as both generator and judge, and by the absence of variance and significance reporting. The significance therefore currently depends on an unverified assumption that LLM-based similarity judgments reflect true semantic alignment rather than self-consistency.

major comments (3)
  1. [Section V.A.b, Table IV] The CP/TC/SAS metrics are computed by prompting o4-mini to score semantic similarity between predicted clusters and RoboFail's expert-defined failure modes, while Section V states that o4-mini is also used for taxonomy discovery, trajectory-to-cluster assignment, runtime monitoring, and LLM-based evaluation. This self-referential setup is load-bearing because the central claim of expert-aligned taxonomies rests on Table IV; a high SAS of 0.958 may reflect the judge's preference for its own generation style rather than genuine correspondence to expert categories. The paper's Section VII acknowledges that VLMs 'may produce plausible but incorrect explanations,' but no human validation or independent judge is provided. Please add a human-annotated cluster-expert similarity study or use an external fixed embedding-based measure that is independent of the clustering model.
  2. [Tables I, II, IV, V] All quantitative comparisons are point estimates with no variance, confidence intervals, or significance tests. For instance, Table IV reports 'Single Run' as an average over four independent runs but gives no standard deviation, and Table V reports F1 scores for single test sets without repeated evaluation. The 46%-to-18% versus 34% failure-rate comparison in Section VI.B is also presented without error bars or multiple seeds. Because the headline claims of improvement and alignment depend on these magnitudes, please report run-to-run variability and statistical tests for the main comparisons.
  3. [Section V.A.c] The trajectory-to-cluster assignment result (weighted F1 of 85.53%) is obtained with an LLM assigner drawn from the same model family that produced the clusters, and the 32.41% baseline is described only as 'embedding-similarity' with no details of the embedding, threshold, or assignment rule. This makes it difficult to interpret the claimed discriminative power of the taxonomy. Please specify the baseline exactly and, ideally, evaluate assignment against human-labeled trajectory-to-cluster ground truth or at least a fixed, non-LLM embedding baseline.
minor comments (5)
  1. [Section III] In the definition of the cluster tuple C_l, the listed items are out of order: the text gives (i), (ii), (iii), (v), then (iv). Please renumber the tuple components consistently.
  2. [Section IV.3] The number of clusters L is said to be 'implicitly optimized' to satisfy the stated criteria, but no procedure or stopping criterion is described. Please clarify how L is determined in practice, since it directly affects all downstream metrics.
  3. [Table I caption] The caption phrase 'proposes Gemini 2.5 Pro as the best choice overall' is grammatically unclear; please rephrase to describe the comparison and the basis for model selection.
  4. [Section V.B] The claim that the Nexar taxonomy 'closely aligns' with the U.S. DoT Volpe pre-crash typology is qualitative; please provide a concrete mapping or quantitative overlap measure if this correspondence is intended as evidence.
  5. [Section VI.B] The fine-tuning experiments for targeted data collection do not state the number of repeated trials or random seeds; please report these details so the 46%-to-18% result can be assessed for stability.

Circularity Check

1 steps flagged · score 6.0 of 10

Table IV's expert-alignment scores are computed by the same OpenAI o4-mini model that generated the clusters; without a human or independent judge, the claimed taxonomy alignment is a self-assessment by construction.

  1. self definitional [Section V (model-selection paragraph) and Section V.A.b 'Validation of Taxonomy Recovery', Table IV]
    "We use Gemini 2.5 Pro for failure explanation and OpenAI o4-mini for taxonomy discovery, trajectory-to-cluster assignment, runtime monitoring, and LLM-based evaluation. [...] For quantitative comparison, we compute a similarity matrix between the discovered clusters and RoboFail's expert-defined taxonomy by prompting an LLM to score semantic similarity for each pair of predicted and expert-defined failure mode."

    The clusters whose quality is claimed to be 'expert-consistent' (Table IV: TC=1.0, SAS=0.958) are produced by the taxonomy-discovery stage, and the similarity scores underlying CP/TC/SAS are produced by 'prompting an LLM' — per the model-selection sentence, the same OpenAI o4-mini model used for 'taxonomy discovery.' The judge and the cluster generator therefore coincide: the expert-alignment metric is o4-mini's rating of its own cluster labels, a self-referential measurement. LLM judges are known to rate outputs sharing their own vocabulary and style as similar, so the high scores can reflect judge self-consistency rather than genuine correspondence to RoboFail's eight expert modes.

full rationale

Most of the pipeline is non-circular and externally anchored: failure explanations are produced by Gemini 2.5 Pro from perceptual trajectories; Table I's cosine-similarity and ROUGE-L scores of the final system are independent of LLM judgment; the trajectory-to-cluster assignment F1 (85.53% vs. a 32.41% embedding baseline) and the Section VI failure-rate reduction (46% to 18% with taxonomy-guided data collection vs. 34% with uniform collection) are objective external measurements of utility; and the self-citations ([30], [41], [43], [46], [47]) point to externally grounded results (e.g., reachability-analysis-derived failure types in [43]) rather than to an unverified uniqueness premise. The load-bearing circularity is confined to the expert-alignment validation: Table IV's CP/TC/SAS are computed from an LLM-scored similarity matrix, and the model-selection sentence assigns OpenAI o4-mini to both 'taxonomy discovery' and 'LLM-based evaluation.' The cluster-producer therefore also serves as the judge of its own clusters' similarity to the RoboFail expert taxonomy, so the headline result (TC=1.0, SAS=0.958) is a self-assessment subject to LLM self-preference; the paper supplies no human rating or independent judge for these scores, and Section VII concedes that the underlying VLMs 'may produce plausible but incorrect explanations.' The judge does show some discrimination (BERTopic drops to SAS=0.696), so the metric is not vacuous, but it cannot by itself certify external expert alignment. This is partial circularity in the central validation claim rather than a fully forced derivation, hence a score of 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central method rests on several domain assumptions about what VLM/LLM outputs mean, plus several unreported hyperparameters. The paper's own conclusion flags the most fragile link: VLMs may produce plausible but incorrect explanations. The evaluation also assumes that LLM semantic similarity scoring is a valid proxy for human judgment of taxonomy alignment.

free parameters (5)
  • CLIP similarity threshold tau
    Frame downsampling retains frames only when cosine similarity to the previous selected frame is below tau; the value is not reported, and it directly controls which evidence the VLM sees.
  • Temporal window lengths Tp and Ts
    The context window before and after the failure frame is a free design choice; exact values are not given, and they affect the failure explanation quality.
  • Number of clusters L
    The LLM implicitly optimizes the number of clusters, but no criterion or range is specified, so cluster granularity is chosen by the model rather than by a principled rule.
  • Ensemble clustering repetition count
    The ensemble-and-refine strategy generates multiple candidate taxonomies, but the number of rephrasings and runs is not reported, making the stability and cost unclear.
  • Additional 40K training samples for targeted data collection = 40,000
    The fine-tuning experiment uses a hand-selected set of 40K additional samples in high-risk regions; this is a single experiment with no ablation over data size or region selection strategy.
assumptions (4)
  • domain assumption VLM-generated failure explanations faithfully represent true failure causes
    The entire clustering pipeline operates on the set of explanations r_n; if the VLM hallucinates plausible but incorrect failures, the taxonomy inherits those errors. The paper concedes this risk in Section VII.
  • domain assumption LLM semantic similarity to an expert taxonomy is a valid measure of taxonomy quality
    The CP/TC/SAS metrics in Table IV depend on an LLM scoring the similarity between predicted clusters and expert-defined failure modes; no human raters or inter-rater agreement are provided to validate this measurement.
  • domain assumption CLIP cosine similarity selects semantically meaningful change points for failure reasoning
    The downsampling step assumes that embedding-space novelty corresponds to causally relevant transitions; no analysis is given for cases where a critical frame is dropped because it lies below the similarity threshold.
  • domain assumption The RoboFail expert taxonomy is complete and correctly labeled
    The RoboFail expert-defined taxonomy is used as ground truth for explanation, clustering, and assignment evaluations; the authors do not audit the quality or completeness of those labels.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Unsupervised Discovery of Failure Taxonomies from Deployment Logs." pith.science (2026). https://pith.science/paper/5IGHIK5S

@misc{pith2026250606570,
  author       = {Pith},
  title        = {Pith review of: Unsupervised Discovery of Failure Taxonomies from Deployment Logs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5IGHIK5S}},
  note         = {Machine review of arXiv:2506.06570}
}
read the original abstract

As robotic systems become increasingly integrated into real-world environments, ranging from autonomous vehicles to household assistants, they inevitably encounter diverse and unstructured scenarios that lead to failures. While such failures pose safety and reliability challenges, they also provide rich perceptual data for improving system robustness. However, manually analyzing large-scale failure datasets is impractical and does not scale. In this work, we introduce the problem of unsupervised discovery of failure taxonomies from large volumes of raw failure logs, aiming to obtain semantically coherent and actionable failure modes directly from perceptual trajectories. Our approach first infers structured failure explanations from multimodal inputs using vision language reasoning, then clusters them in the resulting semantic reasoning space, discovering recurring failure modes rather than isolated episode-level descriptions. We evaluate our method across robotic manipulation, indoor navigation, and autonomous driving domains, demonstrating that the discovered taxonomies are consistent, interpretable, and useful in practice. In particular, we show that structured failure taxonomies guide targeted data collection for offline policy refinement and enhance runtime failure monitoring systems. Website: https://mllm-failure-clustering.github.io/

Figures

Figures reproduced from arXiv: 2506.06570 by the authors.

Figure 1
Figure 1. A framework for unsupervised failure taxonomy discovery from deployment logs. Given failure-centered multimodal [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. A failure explanation example where the robot [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Manipulation failure clusters with keywords and [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Heatmaps showing semantic similarity between the RoboFail expert-defined failure modes (columns) and the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 6
Figure 6. Figure 6: Indoor navigation clusters with examples. [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: Robot failing under nominal policy (red) due to [PITH_FULL_IMAGE:figures/full_fig_p007_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

49 extracted references · 38 canonical work pages

  1. [1]

    Wod-e2e: Waymo open dataset for end-to-end driving in challenging long-tail scenarios,

    R. Xu et al., “Wod-e2e: Waymo open dataset for end-to-end driving in challenging long-tail scenarios,”arXiv preprint arXiv:2510.26125, 2025

    arXiv 2025

  2. [2]

    Reflect: Summarizing robot experiences for failure explanation and correction,

    Z. Liu, A. Bahety, and S. Song, “Reflect: Summarizing robot experiences for failure explanation and correction,”ArXiv, vol. abs/2306.15724, 2023

    arXiv 2023

  3. [3]

    Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation,

    J. Duan et al., “Aha: A vision-language-model for detecting and reasoning over failures in robotic manipulation,”ArXiv, vol. abs/2410.00371, 2024

    arXiv 2024

  4. [4]

    Robofac: A comprehensive framework for robotic failure analysis and correc- tion,

    W. Lu, M. Ye, Z. Ye, R. Tao, S. Yang, and B. Zhao, “Robofac: A comprehensive framework for robotic failure analysis and correc- tion,”ArXiv, 2025

    work page 2025

  5. [5]

    Self-refining vision language model for robotic failure detection and reasoning,

    C. Qi et al., “Self-refining vision language model for robotic failure detection and reasoning,” inThe Fourteenth International Conference on Learning Representations, 2026

    work page 2026

  6. [6]

    Guardian: Detecting robotic planning and execution errors with vision-language models,

    P. Pacaud, R. Garcia, S. Chen, and C. Schmid, “Guardian: Detecting robotic planning and execution errors with vision-language models,” arXiv preprint arXiv:2512.01946, 2025

    arXiv 2025

  7. [7]

    Image clustering conditioned on text criteria,

    S. Kwon, J. Park, M. Kim, J. Cho, E. K. Ryu, and K. Lee, “Image clustering conditioned on text criteria,”ArXiv, vol. abs/2310.18297, 2023

    arXiv 2023

  8. [8]

    Organizing unstructured image collections using natural language,

    M. Liu, Z. Zhong, J. Li, G. Franchi, S. Roy, and E. Ricci, “Organizing unstructured image collections using natural language,” ArXiv, vol. abs/2410.05217, 2024

    arXiv 2024

Show all 49 references
  1. [9]

    Describing differences in image sets with natural language,

    L. Dunlap et al., “Describing differences in image sets with natural language,”CVPR, pp. 24 199–24 208, 2023

    2023

  2. [10]

    Llm as dataset analyst: Subpopulation structure discovery with large language model,

    Y . Luo, R. An, B. Zou, Y . Tang, J. Liu, and S. Zhang, “Llm as dataset analyst: Subpopulation structure discovery with large language model,” inECCV, 2024

    2024

  3. [11]

    Sse: Multimodal semantic data selection and enrichment for industrial-scale data assimilation,

    M. Shen, N. Chang, S. Liu, and J. M. Álvarez, “Sse: Multimodal semantic data selection and enrichment for industrial-scale data assimilation,”ArXiv, vol. abs/2409.13860, 2024

    2024 arXiv

  4. [12]

    Octis: Comparing and optimizing topic models is simple!

    S. Terragni, E. Fersini, B. G. Galuzzi, P. Tropeano, and A. Cande- lieri, “Octis: Comparing and optimizing topic models is simple!” In EACL SysDemo, 2021, pp. 263–270

    2021

  5. [13]

    Topic modelling meets deep neural networks: A survey,

    H. Zhao, D. Phung, V . Huynh, Y . Jin, L. Du, and W. Buntine, “Topic modelling meets deep neural networks: A survey,”ArXiv, 2021

    2021

  6. [14]

    A neural autoregressive topic model,

    H. Larochelle and S. Lauly, “A neural autoregressive topic model,” NeurIPS, vol. 25, 2012

    2012

  7. [15]

    A novel neural topic model and its supervised extension,

    Z. Cao, S. Li, Y . Liu, W. Li, and H. Ji, “A novel neural topic model and its supervised extension,” inAAAI, vol. 29, 2015

    2015

  8. [16]

    Topic modeling over short texts by incorporating word embeddings,

    J. Qiang, P. Chen, T. Wang, and X. Wu, “Topic modeling over short texts by incorporating word embeddings,” inPAKDD, Springer, 2017, pp. 363–374

    2017

  9. [17]

    We-lda: A word embeddings augmented lda model for web services clustering,

    M. Shi, J. Liu, D. Zhou, M. Tang, and B. Cao, “We-lda: A word embeddings augmented lda model for web services clustering,” in ICWS, IEEE, 2017, pp. 9–16

    2017

  10. [18]

    Improving topic models with latent feature word representations,

    D. Q. Nguyen, R. Billingsley, L. Du, and M. Johnson, “Improving topic models with latent feature word representations,”TACL, 2015

    2015

  11. [19]

    Topical word embeddings,

    Y . Liu, Z. Liu, T.-S. Chua, and M. Sun, “Topical word embeddings,” inAAAI, 2015

    2015

  12. [20]

    Cross- lingual contextualized topic models with zero-shot learning,

    F. Bianchi, S. Terragni, D. Hovy, D. Nozza, and E. Fersini, “Cross- lingual contextualized topic models with zero-shot learning,”ArXiv, 2020

    2020

  13. [21]

    Topic modeling in embedding spaces,

    A. B. Dieng, F. J. Ruiz, and D. M. Blei, “Topic modeling in embedding spaces,”TACL, vol. 8, pp. 439–453, 2020

    2020

  14. [22]

    Topic modeling with contextualized word representation clusters,

    L. Thompson and D. Mimno, “Topic modeling with contextualized word representation clusters,”ArXiv, 2020

    2020

  15. [23]

    Pre-training is a hot topic: Contextualized document embeddings improve topic coherence,

    F. Bianchi, S. Terragni, and D. Hovy, “Pre-training is a hot topic: Contextualized document embeddings improve topic coherence,” ArXiv, 2020

    2020

  16. [24]

    Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too!

    S. Sia, A. Dalmia, and S. J. Mielke, “Tired of topic models? clusters of pretrained word embeddings make for fast and good topics too!” ArXiv, 2020

    2020

  17. [25]

    Top2vec: Distributed representations of topics,

    D. Angelov, “Top2vec: Distributed representations of topics,”ArXiv, 2020

    2020

  18. [26]

    Bertopic: Neural topic modeling with a class- based tf-idf procedure,

    M. Grootendorst, “Bertopic: Neural topic modeling with a class- based tf-idf procedure,”ArXiv, 2022

    2022

  19. [27]

    A scenario-based platform for testing au- tonomous vehicle behavior prediction models in simulation,

    F. Indaheng et al., “A scenario-based platform for testing au- tonomous vehicle behavior prediction models in simulation,”ArXiv, vol. abs/2110.14870, 2021

    2021 arXiv

  20. [28]

    Formal scenario-based testing of autonomous vehicles: From simulation to the real world,

    D. J. Fremont et al., “Formal scenario-based testing of autonomous vehicles: From simulation to the real world,”IEEE ITSC, pp. 1–8, 2020

    2020

  21. [29]

    Systematic testing of convolutional neural networks for autonomous driving,

    T. Dreossi, S. Ghosh, A. L. Sangiovanni-Vincentelli, and S. A. Seshia, “Systematic testing of convolutional neural networks for autonomous driving,”ArXiv, vol. abs/1708.03309, 2017

    2017 arXiv

  22. [30]

    Safety-aware imitation learning via mpc-guided disturbance injection,

    L. Qiu, Y . U. Ciftci, and S. Bansal, “Safety-aware imitation learning via mpc-guided disturbance injection,”ArXiv, vol. abs/2508.03129, 2025

    2025 arXiv

  23. [31]

    Verifai: A toolkit for the formal design and analysis of artificial intelligence-based systems,

    T. Dreossi et al., “Verifai: A toolkit for the formal design and analysis of artificial intelligence-based systems,” inCAV, 2019

    2019

  24. [32]

    Formal certification methods for automated vehicle safety assessment,

    T. Zhao, E. Yurtsever, J. A. Paulson, and G. Rizzoni, “Formal certification methods for automated vehicle safety assessment,” IEEE T-IV, vol. 8, pp. 232–249, 2022

    2022

  25. [33]

    Counterexample- guided synthesis of perception models and control,

    S. Ghosh, H. Ravanbakhsh, and S. A. Seshia, “Counterexample- guided synthesis of perception models and control,”ACC, pp. 3447– 3454, 2019

    2019

  26. [34]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,”NeurIPS, vol. 35, pp. 24 824–24 837, 2022

    2022

  27. [35]

    Large language models as optimizers,

    C. Yang et al., “Large language models as optimizers,”ArXiv, vol. abs/2309.03409, 2023

    2023 arXiv

  28. [36]

    Calibrate before use: Improving few-shot performance of language models,

    Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate before use: Improving few-shot performance of language models,” inICML, PMLR, 2021

    2021

  29. [37]

    Self-refine: Iterative refinement with self- feedback,

    A. Madaan et al., “Self-refine: Iterative refinement with self- feedback,” inAdvances in Neural Information Processing Systems, A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, Eds., vol. 36, Curran Associates, Inc., 2023, pp. 46 534– 46 594

    2023

  30. [38]

    Test-time recursive thinking: Self-improvement without external feedback,

    Y . Zhuang et al., “Test-time recursive thinking: Self-improvement without external feedback,”arXiv preprint arXiv:2602.03094, 2026

    2026

  31. [39]

    Nexar dashcam collision predic- tion dataset and challenge,

    D. Moura, S. Zhu, and O. Zvitia, “Nexar dashcam collision predic- tion dataset and challenge,” inCVPR, 2025, pp. 2583–2591

    2025

  32. [40]

    Pre-crash scenario typology for crash avoidance research,

    W. G. Najm, J. D. Smith, and M. Yanagisawa, “Pre-crash scenario typology for crash avoidance research,” 2007

    2007

  33. [41]

    Com- bining optimal control and learning for visual navigation in novel environments,

    S. Bansal, V . Tolani, S. Gupta, J. Malik, and C. Tomlin, “Com- bining optimal control and learning for visual navigation in novel environments,” inCoRL, 2020

    2020

  34. [42]

    3d semantic parsing of large-scale indoor spaces,

    I. Armeni et al., “3d semantic parsing of large-scale indoor spaces,” CVPR, pp. 1534–1543, 2016

    2016

  35. [43]

    Discovering closed-loop failures of vision-based controllers via reachability analysis,

    K. Chakraborty and S. Bansal, “Discovering closed-loop failures of vision-based controllers via reachability analysis,”IEEE RA-L, vol. 8, no. 5, pp. 2692–2699, 2023

    2023

  36. [44]

    Semantic anomaly detection with large language models,

    A. Elhafsi, R. Sinha, C. Agia, E. Schmerling, I. A. D. Nesnas, and M. Pavone, “Semantic anomaly detection with large language models,” Autonomous Robots, vol. 47, pp. 1035–1055, 2023

    2023

  37. [45]

    Videomae v2: Scaling video masked autoencoders with dual masking,

    L. Wang et al., “Videomae v2: Scaling video masked autoencoders with dual masking,” inCVPR, 2023, pp. 14 549–14 560

    2023

  38. [46]

    Enhancing safety and robustness of vision-based controllers via reachability analysis,

    K. Chakraborty, A. Gupta, and S. Bansal, “Enhancing safety and robustness of vision-based controllers via reachability analysis,” arXiv preprint arXiv:2410.21736, 2024

    2024 arXiv

  39. [47]

    Detecting and mitigat- ing system-level anomalies of vision-based controllers,

    A. Gupta, K. Chakraborty, and S. Bansal, “Detecting and mitigat- ing system-level anomalies of vision-based controllers,” in2024 IEEE International Conference on Robotics and Automation (ICRA), IEEE, 2024, pp. 9953–9959

    2024

  40. [48]

    Counterexample-guided data aug- mentation,

    T. Dreossi, S. Ghosh, X. Yue, K. Keutzer, A. Sangiovanni- Vincentelli, and S. A. Seshia, “Counterexample-guided data aug- mentation,”ArXiv, 2018

    2018

  41. [49]

    Specification-guided data aggregation for semantically aware imitation learning,

    A. Shah, J. DeCastro, J. Gideon, B. Yalcinkaya, G. Rosman, and S. A. Seshia, “Specification-guided data aggregation for semantically aware imitation learning,”ArXiv, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.