REVIEW 14 cited by
Real-Time Anomaly Detection and Reactive Planning with Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Signed reviews
read the original abstract
Foundation models, e.g., large language models (LLMs), trained on internet-scale data possess zero-shot generalization capabilities that make them a promising technology towards detecting and mitigating out-of-distribution failure modes of robotic systems. Fully realizing this promise, however, poses two challenges: (i) mitigating the considerable computational expense of these models such that they may be applied online, and (ii) incorporating their judgement regarding potential anomalies into a safe control framework. In this work, we present a two-stage reasoning framework: First is a fast binary anomaly classifier that analyzes observations in an LLM embedding space, which may then trigger a slower fallback selection stage that utilizes the reasoning capabilities of generative LLMs. These stages correspond to branch points in a model predictive control strategy that maintains the joint feasibility of continuing along various fallback plans to account for the slow reasoner's latency as soon as an anomaly is detected, thus ensuring safety. We show that our fast anomaly classifier outperforms autoregressive reasoning with state-of-the-art GPT models, even when instantiated with relatively small language models. This enables our runtime monitor to improve the trustworthiness of dynamic robotic systems, such as quadrotors or autonomous vehicles, under resource and time constraints. Videos illustrating our approach in both simulation and real-world experiments are available on this project page: https://sites.google.com/view/aesop-llm.
Forward citations
Cited by 14 Pith papers
-
Towards General Language-Conditioned Latent Safety Filters
A single Hamilton-Jacobi safety filter conditioned on language constraints reduces violations in simulated pick-and-place, wiping, and stacking, with partial transfer to unseen constraint instances.
-
RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models
RoboMonkey shows that test-time sampling with Gaussian perturbation and a VLM-based action verifier improves the success rate of vision-language-action models on manipulation tasks.
-
Parallel Branch Model Predictive Control on GPUs
A GPU iterative LQR solver for branch MPC that uses parallel scan to parallelize over time and scenarios, and reports 2.5 to 4.5 times speedups on large tree problems.
-
Adversarial Attacks on Robotic Vision Language Action Models
Text-based adversarial suffixes can make OpenVLA robot policies elicit chosen target actions with over 90% success on one-hot targets and persist across rollout steps.
-
Deferred-Decision Trajectory Optimization
A deterministic optimization framework that maximizes how long multiple candidate targets remain reachable, shown equivalent to a cardinality minimization problem and solved via QCVX, MICP, and SCP methods.
-
Predictive Red Teaming: Breaking Policies Without Breaking Robots
A generative image editing plus anomaly detection pipeline predicts a visuomotor policy's success-rate degradation across off-nominal environmental factors, with an average prediction error below 0.19 in hardware trials.
-
Fine-Tuned Language Models as Space Systems Controllers
Fine-tuned LLMs can act as numerical controllers for simplified space systems after training on optimal trajectories, and one model can serve multiple guidance tasks.
-
Don't Fool Me Twice: Adapting to Adversity in the Wild with Experience-Driven Reasoning
By detecting trajectory disturbances, attributing them to visual causes with a VLM, and fitting a few-shot spatial disturbance model, robots build personalized danger libraries that improve later navigation.
-
Seeing, Saying, Solving: An LLM-to-TL Framework for Cooperative Robots
A BNF-constrained LLM converts natural-language help offers into STL formulas that are solved as MILPs, and simulation finds that global-time-impact helper selection outperforms proximity heuristics.
-
LLM-Land: Large Language Models for Context-Aware Drone Landing
A drone landing system that uses BLIP captions, a RAG-grounded lightweight LLM, and MPC raises simulated landing success against dynamic obstacles from 34% to 96% in open-field trials.
-
Generating Out-Of-Distribution Scenarios Using Language Models
The paper generates rare driving scenarios via an LLM-built tree, simulates them in CARLA, but its OOD-ness metric is interpreted in a way that contradicts its equation.
-
Bounding Distributional Shifts in World Modeling through Novelty Detection
Attaching a VAE novelty detector to the DINO-WM world model and penalizing out-of-distribution predicted states in CEM planning lowers Chamfer distance on small-data robot manipulation benchmarks.
-
SafeDrive: Knowledge- and Data-Driven Risk-Sensitive Decision-Making for Autonomous Vehicles with Large Language Models
SafeDrive couples a 360-degree driver risk field with GPT-4 reasoning, memory, and reflection, reporting 100% safety and over 85% human alignment on three real-world trajectory datasets.
-
Domain Specific Benchmarks for Evaluating Multimodal Large Language Models
A review paper that organizes domain-specific MLLM benchmarks into an eight-discipline taxonomy, with summary tables and performance highlights.
Discussion (0). Continue with ORCID to comment.