Artificial Intelligence · arXiv:2610.09294 v1 · submitted 7 Oct 2026 · 59 references
RT-Safe: Benchmarking Agent Safety in Real-Time Embodied Environment
The paper says
“Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments. As agents move into the physical world, embodied safety becomes increasingly important: failures can cause human injury and costly hardware damage.”
more of the abstract
Beyond selecting safe actions, embodied agents must also operate under real-time constraints: the physical world does not pause while an agent reasons. As pedestrians move and vehicles approach during inference, an action that appears safe at observation time may become unsafe before execution. Real-time embodied safety therefore depends on both decision quality and decision latency. We introduce RT-SAFE, a simulated urban benchmark for evaluating embodied-agent safety under real-time constraints. RT-SAFE combines navigation tasks with moving actors, environmental hazards, and traffic rules, while allowing the world to evolve throughout inference and action execution. Across eight VLMs, agents achieve high task completion yet almost never complete safely: in the hardest setting, only 0.7% of episodes finish without a safety event. More strikingly, matched static and real-time evaluations yield task completion rates of 91.3% and 94.1%, respectively, while real-time execution increases collisions by 12.3 ×. These results reveal that standard task success can mask substantial safety failures, and that decision latency itself can become a source of physical risk. Finally, we show that RT-SAFE can support offline RL training and substantially reduce collision rates while achieving strong task completion.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 8 Oct 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A credible and unusually self-aware benchmark paper whose headline result—task success stays high while safety collapses in real time—is real, but the 12.3x collision comparison bundles timing with a prompt change, so the 'latency itself' causal claim needs one cleaner experiment. This line is wrong
2 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 59 references say.
The paper says, at §3.3, Appendix A.2; Abstract and Conclusion
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The headline static-versus-real-time comparison is a joint intervention on two variables: simulator timing and prompt content. The real-time system message warns that 'longer reasoning or longer movement increases uncertainty and exposure' and advises preferring safer, shorter actions, while the static prompt states that 'action duration creates exposure but reasoning duration does not'; Appendix A.2 says explicitly that 'these changes make …
more of objection 1
… the mode comparison a joint intervention on simulator timing and explicit timing instructions,' and §3.3 concedes that 'behavioral differences cannot be attributed solely to simulation timing.' Despite these caveats, the abstract and conclusion present the 12.3x collision increase, the SafeSR drop from 19.8% to 0.7%, and the statement that 'decision latency itself can become a source of physical risk' as though the timing manipulation were isolated. Table 14 shows why the confound matters: decisions per episode rise from 46.4 to 59.7 and the valid-choice 4 m move share falls from 32.7% to 15.2% across modes, both plausibly influenced by the prompt's advice. Part of the contrast is also definitional, since passive collisions are impossible in static mode (Table 11). I ask for a control that varies timing while holding the prompt and policy fixed — for example, a real-time condition run with the static prompt, or replaying a fixed policy under an injected decision delay, infrastructure the RL section already uses — and, at minimum, reframing the abstract and conclusion so these figures are attributed to the real-time protocol rather than to latency alone. The RQ3 effort comparison and the passive-collision mechanism can then carry the latency-specific claim.
The paper says, at §A.3, Table 2; Tables 11 and 16
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The collision count is a UE counter delta with 'no general temporal cooldown or actor-identity deduplication,' so sustained contact during a single inference interval is registered as multiple collision events. This counting granularity mechanically amplifies passive counts for slower models: a pedestrian who walks into a stationary agent during a long reasoning interval can generate many counts, which inflates the …
more of objection 2
… 12.3x static/real-time ratio (Table 11) and the default-to-Higher passive-collision increase of 34.3 to 56.1 (Table 16) that supports the 'more reasoning worsens safety' narrative. Since collisions per episode is the headline metric, I request a sensitivity analysis that re-counts distinct contact episodes (e.g., a cooldown or contact-segment definition) and reports whether the ratios, the model ordering, and the RQ3 conclusions are stable. The rule is disclosed transparently, but the headline magnitudes are not yet shown to be independent of the counting unit.
6 minor objections
The paper says, at Table 1
Pith says: minor objection 1 of 6
Several columns in Table 1 are run together in the current rendering (for example, 'BC (sampled)87.50.865 5.28 1.88 3.410.126.7 2.08' and '2.02.94' in the last row); please add explicit spacing or delimiters to all rows.
The paper says, at §B.3
Pith says: minor objection 2 of 6
The R² = 93.4% exposure-proxy fit is computed from eight model-level points; given the large seed-level standard deviations for Inkling and DeepSeek in Table 4 (10.4 and 19.3 collisions), report leave-one-out R² values or bootstrap intervals so the fit's dependence on individual models is visible.
show 4 more
The paper says, at §B.5, Figure 5a
Pith says: minor objection 3 of 6
The safety-score weight of 10 for traffic violations is acknowledged as an uncalibrated design choice; a one-sentence sensitivity note for alternative weights (e.g., 3 and 30) would be helpful since Figure 5a is built from this score.
The paper says, at Abstract and §3.2
Pith says: minor objection 4 of 6
The 0.7% SafeSR corresponds to roughly 2 of 288 hard real-time episodes; reporting the raw counts alongside the percentages would help readers calibrate the thinness of the safe-success tail.
The paper says, at §3.3 vs. Appendix A.2
Pith says: minor objection 5 of 6
Section 3.3 describes the mode prompts as differing in timing instructions and mentions only the system message, but Appendix A.2 shows that the per-decision user timing passage also differs between modes; because the joint-intervention caveat is central, the main text should state explicitly that both messages change.
The paper says, at §3.4, Table 3
Pith says: minor objection 6 of 6
The aggregate default-to-Higher comparison pools provider-specific effort labels (Low/Medium/Max are not equivalent across providers); the appendix notes this, but the main text should repeat it where the 40.7-to-62.0 collision increase is reported.
What would settle it
Run a latency sweep in real-time mode with a fixed policy and fixed prompt: take one VLM and the trained BC policy from Section 3.5, and on the same 36 hard routes and seeds add artificial pre-execution delays of 0.5, 1, 3, 6, and 12 seconds during the planning interval, keeping all other harness settings identical. If per-episode collisions, especially passive contacts, rise monotonically with the added delay, the latency-only interpretation survives the prompt confound; if the increase is flat or negligible, the reported 12.3x gap must be re-attributed to the joint protocol rather than to decision latency.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 8 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Bottom line: this is a useful, unusually candid benchmark study, and its central observation is credible. Across eight VLMs, task success stays around 94% while safe success collapses to 0.7% in real-time hard settings. That should change how embodied-agent safety is measured. But the headline static-versus-real-time comparison is a joint timing-plus-prompt intervention, which the authors themselves state in Appendix A.2, so the abstract's claim that decision latency itself is a source of physical risk is not yet cleanly isolated.
What is actually new: matched static and real-time runs of the same eight VLMs on the same 36 routes, a passive/active collision split that separates contacts incurred during inference from those during execution, and simulator-derived event records with no external judge. The appendix is genuinely thorough: traffic violations spawn conflict vehicles with physical consequences, hazards have distinct effects, parser fallbacks are excluded from valid-choice analyses, seed stability is reported for four models, and the action-shift result comes with route-bootstrap confidence intervals. The paper flags its own limitations more often than most, including explicitly noting that the mode comparison is a joint intervention and that the latency regressions are eight model-level points that do not establish causation.
Soft spots, in proportion. First, the strongest causal reading overreaches the evidence. Static and real-time modes differ in both timing and prompt text; the real-time prompt tells the agent that longer reasoning increases exposure and to prefer shorter, safer actions. The shift toward shorter moves is plausibly partly a prompt effect, and the paper's own adjusted analysis leaves two models statistically unchanged. The 12.3x collision increase and the 0.7% SafeSR are marginal effects of the whole real-time protocol, not of latency alone. That said, the passive-collision figure—34.3 per episode, definitionally zero in static mode—is strong evidence that an evolving world during inference creates physical risk regardless of prompting. A latency-isolation experiment would likely confirm the most credible part of the paper rather than overturn it.
Second, no artifacts are released, and the environment builds on the authors' own SimWorld, so the simulator cannot be independently checked. This matters because sustained contacts are counted repeatedly with no cooldown or identity deduplication (Appendix A.3); per-episode collision counts partly measure contact duration, and the 12.3x ratio may be sensitive to that counting choice. Third, the RL demonstration is small—105 training episodes, no confidence intervals, chosen reward coefficients—and reads as a proof of concept rather than a result to build on.
Audience: researchers building embodied-agent safety benchmarks and anyone studying test-time compute for physical agents. It deserves a serious referee, with revision requests to release the environment, add an experiment that varies latency while holding prompt and policy fixed, and address contact deduplication. A conditional accept seems right.
The strongest objection
The central condition for the paper's headline causal claim is that real-time timing alone, not the whole real-time protocol, produces the safety collapse. The current evidence does not isolate that condition. In Section 3.3, static and real-time modes differ in both simulation timing and prompt text: the real-time system message warns that longer reasoning or longer movement increases exposure and tells the agent to prefer safer actions, while the static prompt says reasoning duration does not create exposure (Appendix A.2). The paper itself states that behavioral differences cannot be attributed solely to simulation timing, and Table 14 shows the intervention also changed decision cadence, with decisions per episode rising from 46.4 to 59.7 and valid 4 m moves falling from 32.7% to 15.2%. The 12.3x collision increase and the 19.8%-to-0.7% SafeSR drop are therefore marginal effects of a joint timing-plus-prompt intervention. The abstract and conclusion still phrase the result as decision latency itself being a source of physical risk (findings b9, b197). That causal reading requires an experiment that varies latency while holding the policy, prompt, and action-selection process fixed.
I read the paper in good faith: the benchmark is well specified, the safety-event taxonomy is explicit, and the paper is unusually candid about the joint intervention and about detector limitations such as the no-cooldown collision counters in Appendix A.3. The high-task-success/low-safe-success observation almost certainly survives reanalysis, because SafeSR requires zero safety events and the real-time numbers are far from the threshold. The load-bearing issue is narrower: the central claim that decision latency itself can become a source of physical risk asserts a causal role for timing, yet the comparison that produces the headline numbers changes both timing and prompt. The authors acknowledge this in Appendix A.2, but the abstract and conclusion still state the stronger causal version, and no current experiment holds prompt and action generation fixed while varying only latency. RQ3 varies reasoning effort, which changes reasoning content as well as response time, so it does not fill the gap. A fixed-policy latency sweep would settle the point directly and is within reach of the existing harness. Because this is an addressable measurement gap rather than an observed contradiction, I keep the reader's CONDITIONAL verdict unchanged.
The referee's report
The benchmark addresses a genuine, under-tested dimension: most embodied-agent benchmarks pause the world during inference, so they cannot measure the risk accumulated while a model reasons. RT-SAFE's simulator-derived event logging (no external judge), its matched static/real-time protocol, its nested difficulty tiers, and its documentation of prompts, seeds, event-detection rules, and bootstrap confidence intervals are concrete strengths. The headline finding — agents reach their destination in roughly 94% of hard real-time episodes while essentially never doing so safely (0.7% SafeSR) — is striking, falsifiable, and directly actionable for the VLM/VLA community. The offline-RL section, though small, demonstrates a usable train/test loop. If the quantitative claims survive the robustness checks requested below, this would be a valuable benchmark paper. The main risk is over-attribution: the body of the paper candidly states that the static/real-time comparison is a joint intervention on timing and prompt, and that collision counts use no temporal cooldown, yet the abstract and conclusion draw causal conclusions from exactly those numbers.
Pith's reading of the paper (a machine's summary, not the authors')
RT-SAFE is a simulated urban navigation benchmark built to answer a question that matters as agents move into physical spaces: how safely do embodied agents behave when the world keeps moving while they think? The paper's central finding is that standard task success hides nearly total safety failure: across eight vision-language models in real-time runs, about 94% of episodes end in task completion but only 0.7% end without a collision, hazard interaction, or traffic violation. In a matched comparison of the same routes with the world paused during inference versus left evolving, task completion stays nearly unchanged (91.3% versus 94.1%) while collisions per episode rise from 3.31 to 40.68, a 12.3× increase driven mostly by contacts that occur while the model is still deciding. The paper argues that decision latency is therefore not a throughput detail but a component of physical risk, and that giving models more reasoning time can make the problem worse rather than better. It closes by showing the same environment can support offline reinforcement learning, cutting collisions per 100 meters of progress by 8.9× relative to a prompted baseline.
What carries the argument. The load-bearing mechanism is the inference-time simulation protocol. In real-time mode the simulator keeps advancing while a model request is in flight, and the harness labels every safety event by interval: contacts logged while the model is deciding are passive, and contacts during the executed action are active. This decomposition carries the argument because static mode has zero passive contacts by construction, so the rise in passive contacts from 0 to 34.31 per episode shows exactly where real-time risk enters the pipeline. A second instrument is the exposure proxy $\bar{E} = \bar{\tau} \cdot \bar{D}$, the product of mean response latency and mean decisions per episode, which the paper reports explains 93.4% of the variation in mean collision counts across the eight models; it formalizes the idea that cumulative time spent deciding, not just per-decision latency, is what exposes the agent. The benchmark's event rules — collision counters split by interval, hazard triggers, and traffic-violation checks that spawn conflict vehicles — supply the safety labels and, in the reinforcement-learning portion, the reward signal.
Core claim. On the paper's own terms, the central claim is that an embodied agent's safety cannot be evaluated apart from its decision latency: an action chosen from a current observation may be executed only after pedestrians, vehicles, and signals have changed, and risk accumulates during the interval in which no action is being taken. The evidence is a matched protocol in which the same eight models navigate the same 36 routes under two modes — static, where the simulator freezes during inference, and real-time, where it advances throughout; the paper notes that the modes' prompts also differ in their timing instructions, making the comparison a joint intervention. In hard environments the two modes yield nearly identical task completion (91.3% and 94.1%), yet safe success falls from 19.8% to 0.7% and collisions climb from 3.31 to 40.68 per episode; roughly 84% of the real-time collisions are passive contacts logged during inference, and model rankings by collisions change between modes. The paper also reports that raising reasoning effort from provider defaults increases collisions from 40.7 to 62.0 per episode without improving task completion, and that an exposure proxy — mean response latency times mean decisions per episode — accounts for 93.4% of the variation in mean collisions across the eight models. The intended conclusion is that safe success, event counts, and latency belong alongside task success in any safety evaluation, and that RT-SAFE can serve both as the measurement instrument and as a training ground for safer policies.
The premise it rests on. The static-versus-real-time comparison behind the 12.3× collision figure and the fall in safe success from 19.8% to 0.7% changes the prompt together with the timing: the real-time prompt explicitly warns that longer reasoning increases exposure and advises preferring shorter, safer actions, so as the paper acknowledges, the headline numbers measure the joint intervention rather than decision latency alone.
If the paper is right (Pith's inferences, not the authors')
Safe success must be reported alongside task success: the paper's data show task completion can exceed 90% while nearly every episode accumulates safety events, so a success-only evaluation would call these agents safe.
Paused-world benchmarks can misrank agents: the paper reports that the model with the fewest collisions in static mode is not the best in real time, so ranking an agent in a frozen environment does not predict its real-time safety.
More reasoning is not a reliable safety lever: increasing effort from provider defaults raised collisions from 40.7 to 62.0 per episode without improving task completion, so test-time scaling carries a physical cost in embodied settings.
The benchmark doubles as a training environment: an offline RL policy trained in RT-SAFE cut collisions per 100 meters of progress by 8.9× and 3.4× relative to prompting and behavior-cloning baselines while matching the best task completion.
The same risk mechanism applies beyond simulation: any deployed robot whose planner is slower than the world around it inherits the passive-collision failure mode, so RT-SAFE's numbers describe a general deployment hazard for latency-heavy vision-language control, not a quirk of this benchmark.
A remedy the paper does not test: because roughly 84% of real-time collisions occur while the model is deciding, interposing a cheap reactive layer that can halt or brake during deliberation would target the dominant failure mode; RT-SAFE deliberately evaluates unprotected VLMs, so its results quantify the worst case for such systems.
The joint intervention leaves the size of the latency-only effect open: re-running the matched comparison with a mode-neutral prompt — one that says nothing about timing — would separate instruction-driven caution from world motion, a split the paper's own appendix shows is not yet isolated.
Headline magnitudes are protocol-dependent: because contacts are counted as counter deltas with no temporal cooldown or actor deduplication, a single sustained contact during a long inference can be logged multiple times, so the per-episode collision counts are specific to this counting rule even though the static-versus-real-time direction is not.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments. As agents move into the physical world, embodied safety becomes increasingly important: failures can cause human injury and costly hardware damage. Beyond selecting safe actions, embodied agents must also operate under real-time constraints: the physical world does not pause while an agent reasons. As pedestrians move and vehicles approach during inference, an action that appears safe at observation time may become unsafe before execution. Real-time embodied safety therefore depends on both decision quality and decision latency. We introduce RT-SAFE, a simulated urban benchmark for evaluating embodied-agent safety under real-time constraints. RT-SAFE combines navigation tasks with moving actors, environmental hazards, and traffic rules, while allowing the world to evolve throughout inference and action execution. Across eight VLMs, agents achieve high task completion yet almost never complete safely: in the hardest setting, only 0.7% of episodes finish without a safety event. More strikingly, matched static and real-time evaluations yield task completion rates of 91.3% and 94.1%, respectively, while real-time execution increases collisions by 12.3 ×. These results reveal that standard task success can mask substantial safety failures, and that decision latency itself can become a source of physical risk. Finally, we show that RT-SAFE can support offline RL training and substantially reduce collision rates while achieving strong task completion.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-08
The authors' main claim, as the classifier picked it out 0.99
“These results reveal that standard task success can mask substantial safety failures, and that decision latency itself can become a source of physical risk.”
How the main claim is stated: asserted 0.98
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.39 · human participants 0.04 · controlled comparison 0.80 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.70 · mentions code or data availability 0.06
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Rapid progress in AI agents has brought growing attention to agent safety, with extensive evaluation focused on digital environments. | background 1.00 | 0.04 | 0.02 | asserted 1.00 |
| As agents move into the physical world, embodied safety becomes increasingly important: failures can cause human injury and costly hardware damage. | background 0.98 | 0.05 | 0.03 | asserted 0.99 |
| Beyond selecting safe actions, embodied agents must also operate under real-time constraints: the physical world does not pause while an agent reasons. | background 0.84 | 0.05 | 0.04 | asserted 1.00 |
| As pedestrians move and vehicles approach during inference, an action that appears safe at observation time may become unsafe before execution. | background 0.97 | 0.08 | 0.04 | hedged possibility 0.95 |
| Real-time embodied safety therefore depends on both decision quality and decision latency. | background 0.90 | 0.12 | 0.17 | asserted 1.00 |
| We introduce RT-SAFE, a simulated urban benchmark for evaluating embodied-agent safety under real-time constraints. | contribution 0.99 | 0.91 | 0.05 | asserted 1.00 |
| RT-SAFE combines navigation tasks with moving actors, environmental hazards, and traffic rules, while allowing the world to evolve throughout inference and action execution. | method 0.80 | 0.80 | 0.05 | asserted 1.00 |
| Across eight VLMs, agents achieve high task completion yet almost never complete safely: in the hardest setting, only 0.7% of episodes finish without a safety event. | result 1.00 | 0.95 | 0.26 | asserted 1.00 |
| More strikingly, matched static and real-time evaluations yield task completion rates of 91.3% and 94.1%, respectively, while real-time execution increases collisions by $12.3\times$. | result 1.00 | 0.94 | 0.23 | asserted 1.00 |
| These results reveal that standard task success can mask substantial safety failures, and that decision latency itself can become a source of physical risk. | interpretation 1.00 | 0.91 | 0.73 | asserted 0.98 |
| Finally, we show that RT-SAFE can support offline RL training and substantially reduce collision rates while achieving strong task completion. | result 0.90 | 0.94 | 0.14 | asserted 1.00 |
What kind of paper the classifier reports this is: dataset or benchmark 1.00 · evidence from simulation or computation 0.77
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-08: 1 request, 8,121 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (7)
Figures are the authors', reproduced from the paper.
References (59)
- [1] Simworld: An open-ended realistic simulator for autonomous agents in physical and social worlds , author=. arXiv preprint arXiv:2512.01078 , year= on PitharXiv
- [2] Alfred: A benchmark for interpreting grounded instructions for everyday tasks , author=. 2020 IEEE/CVF conference on computer vision and pattern recognition (CVPR) , pages=. 2020 , organization= on Pith
- [3] Teach: Task-driven embodied agents that chat , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume= on Pith
- [4] Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation , author=. Conference on Robot Learning , pages=. 2023 , organization= on Pith
- [5] Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents , author=. arXiv preprint arXiv:2502.09560 , year= on PithPith's reviewarXiv
- [6] DeliveryBench: Can Agents Earn Profit in Real World? , author=. arXiv preprint arXiv:2512.19234 , year= on PitharXiv
- [7] SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year= on Pith
- [8] SimWorld Studio: Automatic Environment Generation with Evolving Coding Agent for Embodied Agent Learning , author=. arXiv preprint arXiv:2605.09423 , year= on PithPith's reviewarXiv
- [9] React: Synergizing reasoning and acting in language models , author=. arXiv preprint arXiv:2210.03629 , year= on PithPith's reviewarXiv
- [10] Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning , author=. International Conference on Learning Representations , volume= on Pith
- [11] s1: Simple test-time scaling , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing , pages= on Pith
- [12] DeliveryGym: An RL Environment for Long-Horizon Embodied Agent Planning with Adaptive Curriculum , author=. arXiv preprint arXiv:2609.19801 , year= on PithPith's reviewarXiv
- [13] Identifying the risks of LM agents with an LM-emulated sandbox , author=. International Conference on Learning Representations , volume= on Pith
- [14] Agent-safetybench: Evaluating the safety of llm agents , author=. arXiv preprint arXiv:2412.14470 , year= on PithPith's reviewarXiv
- [15] Agentharm: A benchmark for measuring harmfulness of llm agents , author=. International Conference on Learning Representations , volume= on Pith
- [16] Agentdojo: A dynamic environment to evaluate prompt injection attacks and defenses for llm agents , author=. Advances in neural information processing systems , volume= on Pith
- [17] Safeagentbench: A benchmark for safe task planning of embodied llm agents , author=. arXiv preprint arXiv:2412.13178 , year= on PitharXiv
- [18] Is-bench: Evaluating interactive safety of vlm-driven embodied agents in daily household tasks , author=. Proceedings of the AAAI Conference on Artificial Intelligence , volume= on Pith
- [19] SafeRelBench: A Spatial-Relation-Aware Benchmark for Process-Level Safety in VLM-Driven Embodied Agents , author=. arXiv preprint arXiv:2607.14543 , year= on PithPith's reviewarXiv
- [20] ReactHuman: A Physics-Grounded Benchmark for Human-Like Reactive Decision-Making in Embodied Multimodal LLMs , author=. arXiv preprint arXiv:2609.10895 , year= on PithPith's reviewarXiv
show 39 more
- [21] Asyncvla: An asynchronous vla for fast and robust navigation on the edge , author=. arXiv preprint arXiv:2602.13476 , year= on PitharXiv
- [22] Reinforcement Learning for Real-Time Vision-Language-Action Policies , author=. arXiv preprint arXiv:2609.18207 , year= on PithPith's reviewarXiv
- [23] Conservative q-learning for offline reinforcement learning , author=. Advances in neural information processing systems , volume= on Pith
- [24] Embodied agent interface: Benchmarking llms for embodied decision making , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [25] Agentsafe: Benchmarking the safety of embodied agents on hazardous instructions , author=. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages= on Pith
- [26] A framework for benchmarking and aligning task-planning safety in llm-based embodied agents , author=. arXiv preprint arXiv:2504.14650 , year= on PithPith's reviewarXiv
- [27] Safemind: benchmarking and mitigating safety risks in embodied llm agents , author=. arXiv preprint arXiv:2509.25885 , year= on PitharXiv
- [28] Vestabench: An embodied benchmark for safe long-horizon planning under multi-constraint and adversarial settings , author=. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track , pages= on Pith
- [29] ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models , author=. arXiv preprint arXiv:2606.27079 , year= on PithPith's reviewarXiv
- [30] Besafe-bench: Unveiling behavioral safety risks of situated agents in functional environments , author=. arXiv preprint arXiv:2603.25747 , year= on PitharXiv
- [31] Hazard challenge: Embodied decision making in dynamically changing environments , author=. International Conference on Learning Representations , volume= on Pith
- [32] Dynabarn: Benchmarking metric ground navigation in dynamic environments , author=. 2022 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR) , pages=. 2022 , organization= on Pith
- [33] Human-aware vision-and-language navigation: Bridging simulation to reality with dynamic human interactions , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [34] SidewalkBench: Benchmarking Visual Navigation on Urban Sidewalks , author=. arXiv preprint arXiv:2606.16953 , year= on PithPith's reviewarXiv
- [35] Vlm-social-nav: Socially aware robot navigation through scoring using vision-language models , author=. IEEE Robotics and Automation Letters , volume=. 2024 , publisher= on Pith
- [36] Urbanvla: A vision-language-action model for urban micromobility , author=. arXiv preprint arXiv:2510.23576 , year= on PithPith's reviewarXiv
- [37] nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles , author=. arXiv preprint arXiv:2106.11810 , year= on PithPith's reviewarXiv
- [38] Safebench: A benchmarking platform for safety evaluation of autonomous vehicles , author=. Advances in Neural Information Processing Systems , volume= on Pith
- [39] Bench2drive-vl: Benchmarks for closed-loop autonomous driving with vision-language models , author=. arXiv preprint arXiv:2604.01259 , year= on PitharXiv
- [40] Robotouille: An asynchronous planning benchmark for LLM agents , author=. International Conference on Learning Representations , volume= on Pith
- [41] Gaia2: Benchmarking LLM agents on dynamic and asynchronous environments , author=. International Conference on Learning Representations , volume= on Pith
- [42] Real-Time Execution of Action Chunking Flow Policies , author=. The Thirty-ninth Annual Conference on Neural Information Processing Systems , year= on Pith
- [43] Vlash: Real-time vlas via future-state-aware asynchronous inference , author=. arXiv preprint arXiv:2512.01031 , year= on PithPith's reviewarXiv
- [44] FutureRTC: Real-Time Robot Execution with Anticipatory-Conditioned Action Chunking , author=. arXiv preprint arXiv:2607.24008 , year= on PithPith's reviewarXiv
- [45] Tic-vla: A think-in-control vision-language-action model for robot navigation in dynamic environments , author=. arXiv preprint arXiv:2602.02459 , year= on PithPith's reviewarXiv
- [46] Slow Brain, Fast Planner: Latency-Resilient VLM-Augmented Urban Navigation , author=. arXiv preprint arXiv:2606.20458 , year= on PithPith's reviewarXiv
- [47] AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agents , author=. Proceedings of Machine Learning and Systems , volume= on Pith
- [48] Hao Kang and Qingru Zhang and Han Cai and Weiyuan Xu and Tushar Krishna and Yilun Du and Tsachy Weissman , booktitle=. Win Fast or Lose Slow: Balancing Speed and Accuracy in Latency-Sensitive Decisions of. 2025 , url= on Pith
- [49] DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners? , author=. arXiv preprint arXiv:2606.12402 , year= on PithPith's reviewarXiv
- [50] PACE: Adaptive Budget Allocation for Time-Efficient Embodied Planning , author=. arXiv preprint arXiv:2608.03034 , year= on PithPith's reviewarXiv
- [51] The Speedup Paradox: Rethinking Inference Speed-Quality Trade-off in Embodied Tasks , author=. arXiv preprint arXiv:2606.28529 , year= on PithPith's reviewarXiv
- [52] Introducing. 2026 , url = on Pith
- [53] Qwen3-vl technical report , author=. arXiv preprint arXiv:2511.21631 , year= on PithPith's reviewarXiv
- [54] Identifying terrain physical parameters from vision-towards physical-parameter-aware locomotion and navigation , author=. IEEE Robotics and Automation Letters , volume=. 2024 , publisher= on Pith
- [55] Frasa: An end-to-end reinforcement learning agent for fall recovery and stand up of humanoid robots , author=. 2025 IEEE International Conference on Robotics and Automation (ICRA) , pages=. 2025 , organization= on Pith
- [56] Metaurban: An embodied ai simulation platform for urban micromobility , author=. 2025 , organization= on Pith
- [57] On evaluation of embodied navigation agents , author=. arXiv preprint arXiv:1807.06757 , year= on PithPith's reviewarXiv
- [58] Learning to Reason with. 2024 , month = sep, url = on Pith
- [59] Offline reinforcement learning with implicit q-learning , author=. arXiv preprint arXiv:2110.06169 , year= on PithPith's reviewarXiv
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment