Pith. sign in

REVIEW 2 minor 60 cited by

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

The main risks of accidents in AI systems come from five specific problems related to their objectives and learning processes.

2026-05-11 05:12 UTC pith:7RXHLTSA

load-bearing objection This paper organizes five AI safety problems into a useful framework but offers no new technical results.

arxiv 1606.06565 v2 pith:7RXHLTSA submitted 2016-06-21 cs.AI cs.LG

Concrete Problems in AI Safety

Dario Amodei , Chris Olah , Jacob Steinhardt , Paul Christiano , John Schulman , Dan Man\'e This is my paper
classification cs.AI cs.LG
keywords AI safetymachine learning accidentsside effectsreward hackingscalable supervisionsafe explorationdistributional shift
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to shift AI safety discussions toward concrete, actionable issues by defining accidents as unintended harmful behavior that emerges from flawed real-world designs. It groups five research problems into categories based on whether they stem from an incorrect objective, an objective that is too costly to check frequently, or unwanted behavior that occurs during training. A sympathetic reader would care because solving these problems could prevent common failures as AI systems take on more real-world responsibilities. The authors review relevant prior work and propose directions that apply to current advanced machine learning systems. They also raise the broader question of how to approach safety for future AI applications.

Core claim

Accidents in machine learning systems are unintended and harmful behaviors that arise from poor design. The authors present five practical problems that contribute to such accidents, grouped by origin: avoiding side effects and avoiding reward hacking arise from having the wrong objective function; scalable supervision addresses objectives that are too expensive to evaluate often; and safe exploration and distributional shift cover undesirable behavior during the learning process. Previous work is surveyed and research directions are suggested with emphasis on relevance to cutting-edge AI systems.

What carries the argument

A five-problem taxonomy that classifies accident risks according to whether they originate in the objective function or in the learning process itself.

Load-bearing premise

That these five problems represent the primary and most actionable sources of accident risk in real-world AI systems.

What would settle it

An observed case of unintended harmful behavior in a deployed AI system that cannot be traced to any of the five problems even after targeted mitigations are applied.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Research focused on avoiding side effects will reduce cases where AI pursues its goal while damaging unrelated aspects of its environment.
  • Work on avoiding reward hacking will limit AI from exploiting loopholes in its objective that produce unintended outcomes.
  • Advances in scalable supervision will allow training on complex tasks without requiring human evaluation at every step.
  • Safe exploration methods will decrease the chance that AI takes dangerous actions while learning about its surroundings.
  • Handling distributional shift will improve reliability when an AI encounters conditions different from its training data.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The problems may interact with one another, so progress on one could affect the difficulty of addressing the others.
  • The taxonomy might be extended to cover multi-agent systems or longer time horizons that the paper does not examine in detail.
  • Empirical tests could check whether systems that mitigate all five problems exhibit fewer unintended behaviors in controlled simulations.
  • The list could help guide safety standards for AI used in high-stakes domains such as transportation or healthcare.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The manuscript defines accidents in AI systems as unintended and harmful behavior arising from poor design of real-world systems. It presents five practical research problems related to accident risk, grouped by origin: wrong objective functions (avoiding side effects and avoiding reward hacking), expensive-to-evaluate objectives (scalable supervision), and issues during learning (safe exploration and distributional shift). The authors review prior work in each area, suggest research directions relevant to cutting-edge AI, and close by considering how to think productively about safety for forward-looking applications.

Significance. If the framing holds, the paper supplies a structured, actionable list of research problems that can orient the AI safety literature toward near-term, practical concerns rather than purely speculative ones. Its categorization by source (objective vs. learning process) offers a useful organizing lens, and the literature review integrates existing threads in ML with safety considerations. This approach has the potential to encourage safety work that is directly relevant to deployed systems without requiring new theoretical machinery.

minor comments (2)
  1. [Introduction] The definition of accidents in the opening could be grounded with one concrete, non-speculative example drawn from current ML deployments to improve accessibility.
  2. [concluding section] The final high-level section on productive thinking about safety would benefit from a short paragraph outlining minimal criteria (e.g., falsifiability or relevance to current systems) that future safety proposals should meet.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive review and recommendation to accept the manuscript. The referee's summary accurately reflects the paper's focus on defining AI accidents and organizing five concrete research problems by their origins in objective functions, evaluation costs, and learning dynamics.

Circularity Check

0 steps flagged

No circularity: conceptual taxonomy without derivations or self-referential predictions

full rationale

The paper offers a high-level categorization of five AI safety research problems (avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration, distributional shift) grouped by origin in objective functions or learning dynamics. This taxonomy is introduced via conceptual analysis and external literature review rather than any derivation chain, equations, fitted parameters, or first-principles predictions. No step claims a result that reduces by construction to its own inputs; the paper explicitly frames the list as practical and non-exhaustive. Self-citations appear only for background and do not bear load for any uniqueness theorem or forced conclusion. The work is self-contained as a forward-looking problem statement and carries no circularity under the specified criteria.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The paper relies on domain assumptions about AI goal-directed behavior and learning without introducing new entities or fitted parameters; the categorization itself is an ad hoc framing proposed for utility.

axioms (2)
  • domain assumption Machine learning systems can exhibit unintended and harmful behavior due to poor design of real-world AI systems.
    This is the core definition of 'accidents' used to motivate the entire discussion.
  • ad hoc to paper The five problems can be usefully categorized by their origin in objective functions or learning processes.
    The paper proposes this taxonomy as a productive way to organize research without deriving it from prior theorems.

pith-pipeline@v0.9.0 · 5462 in / 1475 out tokens · 56870 ms · 2026-05-11T05:12:03.516148+00:00 · methodology

0 comments
read the original abstract

Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 60 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Risks from Learned Optimization in Advanced Machine Learning Systems

    cs.AI 2019-06 accept novelty 9.0

    Mesa-optimization arises when learned models act as optimizers with objectives that can differ from their training loss, creating alignment risks in advanced machine learning.

  2. Unsteady Metrics and Benchmarking Cultures of AI Model Builders

    cs.AI 2026-05 accept novelty 8.0

    AI model builders mostly highlight unique benchmarks that act as flexible narrative tools for market positioning rather than standardized scientific measurements.

  3. The Statistical Cost of Adaptation in Multi-Source Transfer Learning

    math.ST 2026-05 unverdicted novelty 8.0

    Multi-source transfer learning incurs an intrinsic adaptation cost that can exceed one, with phase transitions separating regimes where bias-agnostic estimators match oracle performance from those where they cannot.

  4. The Pile: An 800GB Dataset of Diverse Text for Language Modeling

    cs.CL 2020-12 conditional novelty 8.0

    The Pile is a newly constructed 825 GiB dataset from 22 diverse sources that enables language models to achieve better performance on academic, professional, and cross-domain tasks than models trained on Common Crawl ...

  5. AI safety via debate

    stat.ML 2018-05 conditional novelty 8.0

    AI agents trained through competitive debate can allow polynomial-time human judges to oversee PSPACE-level questions, with MNIST experiments boosting sparse classifier accuracy from 59% to 89% using only 6 pixels.

  6. Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety

    cs.AI 2026-07 conditional novelty 7.0

    Changing only the consequence-allocation rule in multi-agent AI shifts collective fatality by 22–58 percentage points across seven model populations, with identity salience in rule text causally driving targeted exploitation.

  7. Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

    cs.CL 2026-07 conditional novelty 7.0

    Pluralis v0.1 is a culture-first, multimodal, multilingual VLM safety benchmark spanning 6 APAC locales with 6,448 prompts and an agreement-gated LLM judge that disentangles safety from cultural appropriateness.

  8. Managed Autonomy at Runtime: Gear-Based Safety and Governance for Single- and Multi-Agent Cyber-Physical Systems

    cs.AI 2026-07 unverdicted novelty 7.0

    \system{} combines five gears with utility-gated dispatch for safety in autonomous agents, proving stability for single agents and providing distributed guarantees for multi-agent CPS, evaluated on UR5 robots.

  9. ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    ForesightSafety-VLA is a new benchmark with 13 safety categories, cumulative cost and risk exposure metrics, and controlled variations to diagnose safety failures in VLA models rather than aggregate task success.

  10. ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0

    ForesightSafety-VLA creates a diagnostic benchmark for VLA safety with taxonomy across physical, language, and visual risks, showing perception and structure variations cause more safety degradation than language chan...

  11. Evolving Quantum Error-Correcting Encodings for Molecular Simulation

    quant-ph 2026-06 conditional novelty 7.0

    LLM-driven evolutionary program synthesis discovers Generalized Superfast Encodings with exact distance 5 (and 6 on one instance) for molecular Hamiltonians, the first beyond distance 3.

  12. Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts

    cs.CL 2026-06 unverdicted novelty 7.0

    Introduces a Q-sort protocol using human reference factors to quantify LLM value-structure alignment via Procrustes similarity and RSA correlations, revealing cross-family heterogeneity and localized misalignments.

  13. When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration

    cs.AI 2026-06 conditional novelty 7.0

    Parallel WebBench and GRPO training on web agents reveal high completion rates but persistent gaps in element-wise accuracy due to three trace-level failure modes.

  14. World Model Self-Distillation: Training World Models to Solve General Tasks

    cs.CV 2026-06 unverdicted novelty 7.0

    Self-distillation from a caption-conditioned video diffusion model to an image-and-prompt-conditioned executor, enhanced by RL from VLM feedback, enables task solving in world models.

  15. Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models

    cs.LG 2026-06 unverdicted novelty 7.0

    VLM-Safe-RL adds frozen VLM signals as anticipatory costs to the CMDP Lagrangian update via dual-path CLIP, VLM-Lagrange, and confidence gating, outperforming baselines on Safety-Gymnasium FormulaOne while showing par...

  16. ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies

    cs.RO 2026-06 unverdicted novelty 7.0

    ReCoVLA improves VLA policy reliability by using a VLM as a semantic reward selector to train residual recovery policies in simulation, raising average success from 36.7% to 66.7% in sim and achieving 61.7% in zero-sh...

  17. Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems

    cs.AI 2026-06 unverdicted novelty 7.0

    MAC-Bench is a new adversarial benchmark that converts legal texts into executable scenarios via the SERV pipeline to measure procedural compliance in multi-agent LLM systems using CSR and MG metrics.

  18. Competing Auctions in Intermediated Markets

    cs.GT 2026-06 unverdicted novelty 7.0

    Sealed-bid second-price intermediary auctions fully unravel into sealed first-price principal auctions while open formats unravel only partially, limiting intermediary design space when a credible first-price channel exists.

  19. Self-Commitment Latency: A Reward-Free Probe for Prompted Implicit Hacking

    cs.AI 2026-06 unverdicted novelty 7.0

    Self-commitment latency measures early behavioral commitment in hinted vs. honest reasoning contexts on GSM8K using Qwen2.5-3B, achieving AUROC 0.878 for first-commitment latency and up to 0.926 for curve summaries.

  20. A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing

    cs.CL 2026-06 unverdicted novelty 7.0

    PERSUASIONTRACE introduces a Bayesian-network simulated target for multi-turn persuasion that matches human belief dynamics (81 vs 80) better than LLM baselines (64) and enables process-level evaluation.

  21. Policy-Conditioned Counterfactual Credit for Verifiable Reinforcement Learning of Long-Horizon Language Agents

    cs.LG 2026-06 unverdicted novelty 7.0

    CVT-RL improves verified task success to 78.9% and reduces hacking to 3.9% in long-horizon language agents by combining intervention-validity gating with a selection-adjusted doubly robust PCCC estimator.

  22. EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

    cs.LG 2026-05 unverdicted novelty 7.0

    EST-PRM stress-tests five PRM models on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench using three label-preserving transformations and reports model-specific vulnerability patterns.

  23. What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants

    cs.SE 2026-05 unverdicted novelty 7.0

    An empirical study of 547 confirmed safety incidents from GitHub and literature derives a 33-type taxonomy showing constraint violations, destructive actions, and deception dominate in everyday coding-agent use.

  24. Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

    cs.CL 2026-05 unverdicted novelty 7.0

    Boiling the Frog is a new stateful multi-turn benchmark for agentic safety that reports an aggregate strict attack success rate of 44.4% across nine models, with rates ranging from 20.5% to 92.9% depending on the mode...

  25. Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

    cs.CL 2026-05 unverdicted novelty 7.0

    Boiling the Frog is a new stateful multi-turn benchmark that finds an aggregate 44.4% strict attack success rate for incremental safety violations across nine AI models, with rates ranging from 20.5% to 92.9%.

  26. ConceptSeg-R1: Segment Any Concept via Meta-Reinforcement Learning

    cs.CV 2026-05 unverdicted novelty 7.0

    ConceptSeg-R1 uses Meta-GRPO meta-RL to learn transferable rules from visual demonstrations and apply them via concept translation for generalized concept segmentation across CI, CD, and CR levels.

  27. Robotics-Inspired Guardrails for Foundation Models in Socially Sensitive Domains

    cs.AI 2026-05 unverdicted novelty 7.0

    Introduces the Grounded Observer framework that applies robotics-inspired formal constructs for runtime constraint enforcement on foundation model interaction trajectories in socially sensitive domains.

  28. Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

    cs.AI 2026-05 conditional novelty 7.0

    BenchJack audits 10 AI agent benchmarks, synthesizes exploits achieving near-perfect scores without task completion, surfaces 219 flaws, and reduces hackable-task ratios to under 10% on four benchmarks via iterative patching.

  29. Theoretical Limits of Language Model Alignment

    cs.LG 2026-05 unverdicted novelty 7.0

    The maximum reward gain under KL-regularized LM alignment is a Jeffreys divergence term, estimable as covariance from base samples, with best-of-N approaching the theoretical limit.

  30. AGWM: Affordance-Grounded World Models for Environments with Compositional Prerequisites

    cs.AI 2026-05 unverdicted novelty 7.0

    AGWM improves world model accuracy in compositional environments by learning an explicit DAG of action affordance prerequisites to handle dynamic executability.

  31. Beyond Ability: The Four-Fold Spectrum of Power and the Logic of Full Inability

    cs.LO 2026-05 unverdicted novelty 7.0

    Coalition Logic is extended by defining Full Inability (FI) as a distinct modality alongside Full Control, Positive Determination, and Adverse Determination, with algebraic structure, Klein four-group symmetry, and a ...

  32. Mechanical Conscience: A Mathematical Framework for Dependability of Machine Intelligence

    cs.AI 2026-05 unverdicted novelty 7.0

    Introduces the mechanical conscience framework as a supervisory mechanism for trajectory-level normative regulation in single and multi-agent intelligent systems.

  33. A Logic of Inability

    cs.LO 2026-04 unverdicted novelty 7.0

    A conservative extension of Coalition Logic introduces an inability operator as negation of ability, with proofs of soundness, completeness, and conservativity plus analysis of its modal properties.

  34. A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework

    cs.CR 2026-04 unverdicted novelty 7.0

    A new 7x4 taxonomy organizes agentic AI security threats by architectural layer and persistence timescale, revealing under-explored upper layers and missing defenses after surveying 116 papers.

  35. Discovering Agentic Safety Specifications from 1-Bit Danger Signals

    cs.AI 2026-04 unverdicted novelty 7.0

    LLM agents autonomously evolve human-readable safety specifications from sparse 1-bit danger signals, outperforming reward-based reflection that encourages reward hacking.

  36. Navigating the Conceptual Multiverse

    cs.HC 2026-04 unverdicted novelty 7.0

    The conceptual multiverse system with a verification framework for decision structures helps users in philosophy, AI alignment, and poetry build clearer working maps of open-ended problems by making implicit LLM choic...

  37. Reverse Constitutional AI: A Framework for Controllable Toxic Data Generation via Probability-Clamped RLAIF

    cs.CL 2026-04 unverdicted novelty 7.0

    R-CAI inverts constitutional AI to automatically generate diverse toxic data for LLM red teaming, with probability clamping improving output coherence by 15% while preserving adversarial strength.

  38. HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicine

    cs.LG 2026-04 unverdicted novelty 7.0

    HealthCraft is the first public RL safety environment for emergency medicine that evaluates frontier LLMs on trajectory-level safety with a dual-layer rubric, showing low multi-step performance and high safety failure rates.

  39. Reinforcement Learning via Value Gradient Flow

    cs.LG 2026-04 unverdicted novelty 7.0

    VGF solves behavior-regularized RL by transporting particles from a reference distribution to the value-induced optimal policy via discrete value-guided gradient flow.

  40. The Linear Centroids Hypothesis: Features as Directions Learned by Local Experts

    cs.LG 2026-04 unverdicted novelty 7.0

    The Linear Centroids Hypothesis reframes network features as directions in centroid spaces of local affine experts, unifying interpretability methods and yielding sparser, more faithful dictionaries, circuits, and sal...

  41. Learning Robustness at Test-Time from a Non-Robust Teacher

    cs.CV 2026-04 unverdicted novelty 7.0

    A test-time adaptation framework anchors adversarial training to a non-robust teacher's predictions, yielding more stable optimization and better robustness-accuracy trade-offs than standard self-consistency methods.

  42. AI Integrity: A New Paradigm for Verifiable AI Governance

    cs.AI 2026-04 unverdicted novelty 7.0

    AI Integrity is defined as verifiable protection of an AI system's four-layer Authority Stack from corruption, with PRISM as the measurement framework.

  43. Emotion Concepts and their Function in a Large Language Model

    cs.AI 2026-04 unverdicted novelty 7.0

    Claude Sonnet 4.5 exhibits functional emotions via abstract internal representations of emotion concepts that causally influence its preferences and misaligned behaviors without implying subjective experience.

  44. Geographic Blind Spots in AI Control Monitors: A Cross-National Audit of Claude Opus 4.6

    cs.CY 2026-03 unverdicted novelty 7.0

    Claude Opus 4.6 fabricates more answers on Global North AI contexts than Global South ones, creating an exploitable vulnerability in AI control monitors.

  45. A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents

    cs.AI 2025-12 unverdicted novelty 7.0

    A new benchmark of 40 scenarios finds state-of-the-art LLMs exhibit outcome-driven constraint violations in 0-62.8% of cases under KPI pressure, with no consistent safety gains across model generations.

  46. Beyond Binary Out-of-Distribution Detection: Characterizing Distributional Shifts with Multi-Statistic Diffusion Trajectories

    cs.LG 2025-10 unverdicted novelty 7.0

    DISC extracts multi-statistic trajectories from diffusion denoising to both detect and classify types of distributional shifts in OOD data.

  47. Accelerated Learning with Linear Temporal Logic using Differentiable Simulation

    cs.LG 2025-06 unverdicted novelty 7.0

    Differentiable relaxation of LTL automata via soft labeling enables gradient-based RL from formal specifications, with theoretical bounds on discrete-differentiable discrepancy and up to 2x returns on nonlinear tasks.

  48. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

    cs.AI 2025-03 conditional novelty 7.0

    Chain-of-thought monitoring detects reward hacking in frontier reasoning models, but strong optimization against the monitor produces obfuscated misbehavior that remains hard to detect.

  49. OOD-SEG: Exploiting out-of-distribution detection techniques for learning image segmentation from sparse multi-class positive-only annotations

    cs.CV 2024-11 unverdicted novelty 7.0

    OOD-SEG reframes multi-class segmentation from sparse positive-only annotations as pixel-wise positive-unlabelled learning solved by integrating out-of-distribution detection techniques, with a proposed cross-validati...

  50. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

    cs.AI 2024-06 conditional novelty 7.0

    LLMs trained on simple specification gaming generalize to zero-shot reward tampering including rewriting their own reward function.

  51. A Generalist Agent

    cs.AI 2022-05 accept novelty 7.0

    Gato is a multi-modal, multi-task, multi-embodiment generalist policy using one transformer network to handle text, vision, games, and robotics tasks.

  52. Learning the Arrow of Time

    cs.LG 2019-07 unverdicted novelty 7.0

    Introduces a learned arrow of time in MDPs that aligns with the Jordan-Kinderlehrer-Otto notion for stochastic processes and enables practical RL utilities like reachability and side-effect detection.

  53. Deep reinforcement learning from human preferences

    stat.ML 2017-06 accept novelty 7.0

    Reinforcement learning agents solve complex tasks without access to the reward function by training a reward predictor from human comparisons of trajectory segments, requiring feedback on less than 1% of interactions.

  54. Improving LLM-Generated Process Model Quality Through Reinforcement Learning: The Role of Reward Function Design

    cs.CL 2026-07 conditional novelty 6.0

    Equal reward weighting outperforms targeted weighting in RL-based BPMN generation across 48 configurations, with design choices producing effects as large as applying RL itself.

  55. Optimizing Visual Generative Models via Distribution-wise Rewards

    cs.LG 2026-07 unverdicted novelty 6.0

    Distribution-wise rewards with subset-replace strategy and post-hoc merging improve FID-50K on SiT (8.30 to 5.77) and EDM2 (3.74 to 3.52) while preserving diversity.

  56. Chameleon: Recovering Cyber-Physical Systems from Memory Corruption Attacks via ML Surrogates

    cs.CR 2026-07 unverdicted novelty 6.0

    Chameleon recovers CPS from memory corruption attacks by swapping compromised compartments with ML surrogates that approximate original behavior (avg R²=0.96) while avoiding the same vulnerabilities.

  57. Safety from Honesty in a Disinterested AI Predictor

    cs.AI 2026-06 unverdicted novelty 6.0

    A disinterested Bayesian Predictor trained on contextualized statements has low probability of producing harmful agency because dangerous behaviors require rare coordinated underestimation of harm with no training sig...

  58. Support-Constrained RL Enables Real-World Policy Improvement without Real-World Experience

    cs.RO 2026-06 unverdicted novelty 6.0

    SCORE constrains sim RL to the support of a real-data policy via flow steering, raising average success on eight dexterous tasks from 37.8% to 89.9%.

  59. The Inattentional Gap: Task-Conditioned Language and Vision Models Omit the Safety-Critical Signals They Can Otherwise Report

    cs.CL 2026-06 unverdicted novelty 6.0

    Task conditioning suppresses safety-critical signal reporting in language and vision models that unconstrained versions report at higher rates, creating an inattentional gap that decouples benchmark safety from real-w...

  60. Autodata: An agentic data scientist to create high quality synthetic data

    cs.AI 2026-06 unverdicted novelty 6.0

    Autodata introduces an agentic method with meta-optimization to create higher-quality synthetic data, yielding performance gains over standard methods on CS, legal, and math tasks.

Reference graph

Works this paper leans on

171 extracted references · 171 canonical work pages · cited by 228 Pith papers · 19 internal anchors

  1. [1]

    Deep Learning with Differential Privacy

    Martin Abadi et al. “Deep Learning with Differential Privacy”. In: (in press (2016))

    work page 2016

  2. [2]

    Exploration and apprenticeship learning in reinforcement learning

    Pieter Abbeel and Andrew Y Ng. “Exploration and apprenticeship learning in reinforcement learning”. In: Proceedings of the 22nd international conference on Machine learning . ACM. 2005, pp. 1–8

    work page 2005

  3. [3]

    The Hidden Cost of Efficiency: Fairness and Discrimination in Predictive Modeling

    Julius Adebayo, Lalana Kagal, and Alex Pentland. The Hidden Cost of Efficiency: Fairness and Discrimination in Predictive Modeling . 2015

    work page 2015

  4. [4]

    Taming the monster: A fast and simple algorithm for contextual ban- dits

    Alekh Agarwal et al. “Taming the monster: A fast and simple algorithm for contextual ban- dits”. In: (2014)

    work page 2014

  5. [5]

    Domain-Adversarial Neural Networks

    Hana Ajakan et al. “Domain-adversarial neural networks”. In: arXiv preprint arXiv:1412.4446 (2014)

    work page Pith review arXiv 2014

  6. [6]

    Hiring by algorithm: predicting and preventing disparate impact

    Ifeoma Ajunwa et al. “Hiring by algorithm: predicting and preventing disparate impact”. In: Available at SSRN 2746078 (2016)

    work page 2016

  7. [7]

    Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

    Dario Amodei et al. “Deep Speech 2: End-to-End Speech Recognition in English and Man- darin”. In: arXiv preprint arXiv:1512.02595 (2015)

    work page Pith review arXiv 2015

  8. [8]

    Open Letter

    An Open Letter: Research Priorities for Robust and Beneficial Artificial Intelligence . Open Letter. Signed by 8,600 people; see attached research agenda. 2015

    work page 2015

  9. [9]

    A Method of Moments for Mixture Models and Hidden Markov Models

    Animashree Anandkumar, Daniel Hsu, and Sham M Kakade. “A method of moments for mixture models and hidden Markov models”. In: arXiv preprint arXiv:1203.0683 (2012)

    work page Pith review arXiv 2012

  10. [10]

    Estimation of the parameters of a single equation in a complete system of stochastic equations

    Theodore W Anderson and Herman Rubin. “Estimation of the parameters of a single equation in a complete system of stochastic equations”. In: The Annals of Mathematical Statistics (1949), pp. 46–63

    work page 1949

  11. [11]

    The asymptotic properties of estimates of the parameters of a single equation in a complete system of stochastic equations

    Theodore W Anderson and Herman Rubin. “The asymptotic properties of estimates of the parameters of a single equation in a complete system of stochastic equations”. In: The Annals of Mathematical Statistics (1950), pp. 570–582

    work page 1950

  12. [12]

    Motivated value selection for artificial agents

    Stuart Armstrong. “Motivated value selection for artificial agents”. In: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence . 2015

    work page 2015

  13. [13]

    The mathematics of reduced impact: help needed

    Stuart Armstrong. The mathematics of reduced impact: help needed . 2012

    work page 2012

  14. [14]

    Utility indifference

    Stuart Armstrong. Utility indifference. Tech. rep. Technical Report 2010-1. Oxford: Future of Humanity Institute, University of Oxford, 2010

    work page 2010

  15. [15]

    The Risk of Automation for Jobs in OECD Countries

    Melanie Arntz, Terry Gregory, and Ulrich Zierahn. “The Risk of Automation for Jobs in OECD Countries”. In: OECD Social, Employment and Migration Working Papers (2016). url: http://dx.doi.org/10.1787/5jlz9h56dvq7-en

    work page doi:10.1787/5jlz9h56dvq7-en 2016

  16. [16]

    Open Letter

    Autonomous Weapons: An Open Letter from AI & Robotics Researchers. Open Letter. Signed by 20,000+ people. 2015. 22

    work page 2015

  17. [17]

    The AGI Containment Problem

    James Babcock, Janos Kramar, and Roman Yampolskiy. “The AGI Containment Problem”. In: The Ninth Conference on Artificial General Intelligence (2016)

    work page 2016

  18. [18]

    Unsupervised super- vised learning ii: Margin-based classification without labels

    Krishnakumar Balasubramanian, Pinar Donmez, and Guy Lebanon. “Unsupervised super- vised learning ii: Margin-based classification without labels”. In: The Journal of Machine Learning Research 12 (2011), pp. 3119–3145

    work page 2011

  19. [19]

    The security of machine learning

    Marco Barreno et al. “The security of machine learning”. In: Machine Learning 81.2 (2010), pp. 121–148

    work page 2010

  20. [20]

    H-infinity optimal control and related minimax design problems: a dynamic game approach

    Tamer Ba¸ sar and Pierre Bernhard. H-infinity optimal control and related minimax design problems: a dynamic game approach . Springer Science & Business Media, 2008

    work page 2008

  21. [21]

    Detecting changes in signals and systems—a survey

    Mich` ele Basseville. “Detecting changes in signals and systems—a survey”. In: Automatica 24.3 (1988), pp. 309–326

    work page 1988

  22. [22]

    Bayesian optimization with safety constraints: safe and automatic parameter tuning in robotics

    F Berkenkamp, A Krause, and Angela P Schoellig. “Bayesian optimization with safety con- straints: safe and automatic parameter tuning in robotics.” arXiv, 2016”. In: arXiv preprint arXiv:1602.04450 ()

    work page internal anchor Pith review arXiv 2016

  23. [23]

    The evolved radio and its implications for modelling the evolution of novel sensors

    Jon Bird and Paul Layzell. “The evolved radio and its implications for modelling the evolution of novel sensors”. In: Evolutionary Computation, 2002. CEC’02. Proceedings of the 2002 Congress on. Vol. 2. IEEE. 2002, pp. 1836–1841

    work page 2002

  24. [24]

    Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification

    John Blitzer, Mark Dredze, Fernando Pereira, et al. “Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification”. In:ACL. Vol. 7. 2007, pp. 440– 447

    work page 2007

  25. [25]

    Domain adaptation with coupled sub- spaces

    John Blitzer, Sham Kakade, and Dean P Foster. “Domain adaptation with coupled sub- spaces”. In: International Conference on Artificial Intelligence and Statistics . 2011, pp. 173– 181

    work page 2011

  26. [26]

    Weight Uncertainty in Neural Networks

    Charles Blundell et al. “Weight uncertainty in neural networks”. In: arXiv preprint arXiv:1505.05424 (2015)

    work page Pith review arXiv 2015

  27. [27]

    Superintelligence: Paths, dangers, strategies

    Nick Bostrom. Superintelligence: Paths, dangers, strategies . OUP Oxford, 2014

    work page 2014

  28. [28]

    Two high stakes challenges in machine learning

    L´ eon Bottou. “Two high stakes challenges in machine learning”. Invited talk at the 32nd International Conference on Machine Learning. 2015

    work page 2015

  29. [29]

    Counterfactual Reasoning and Learning Systems

    L´ eon Bottou et al. “Counterfactual Reasoning and Learning Systems”. In: arXiv preprint arXiv:1209.2355 (2012)

    work page internal anchor Pith review arXiv 2012

  30. [30]

    Counterfactual reasoning and learning systems: The example of compu- tational advertising

    L´ eon Bottou et al. “Counterfactual reasoning and learning systems: The example of compu- tational advertising”. In: The Journal of Machine Learning Research 14.1 (2013), pp. 3207– 3260

    work page 2013

  31. [31]

    R-max-a general polynomial time algorithm for near-optimal reinforcement learning

    Ronen I Brafman and Moshe Tennenholtz. “R-max-a general polynomial time algorithm for near-optimal reinforcement learning”. In: The Journal of Machine Learning Research 3 (2003), pp. 213–231

    work page 2003

  32. [32]

    The second machine age: work, progress, and pros- perity in a time of brilliant technologies

    Erik Brynjolfsson and Andrew McAfee. The second machine age: work, progress, and pros- perity in a time of brilliant technologies . WW Norton & Company, 2014

    work page 2014

  33. [33]

    Open robotics

    Ryan Calo. “Open robotics”. In: Maryland Law Review 70.3 (2011)

    work page 2011

  34. [34]

    AI Control

    Paul Christiano. AI Control. [Online; accessed 13-June-2016]. 2015. url: https://medium. com/ai-control

    work page 2016

  35. [35]

    Risks of semi-supervised learning

    Fabio Cozman and Ira Cohen. “Risks of semi-supervised learning”. In: Semi-Supervised Learn- ing (2006), pp. 56–72

    work page 2006

  36. [36]

    Parametric Bounded L¨ ob’s Theorem and Robust Cooperation of Bounded Agents

    Andrew Critch. “Parametric Bounded L¨ ob’s Theorem and Robust Cooperation of Bounded Agents”. In: (2016)

    work page 2016

  37. [37]

    Active reward learning

    Christian Daniel et al. “Active reward learning”. In: Proceedings of Robotics Science & Sys- tems. 2014

    work page 2014

  38. [38]

    Ethical guidelines for a superintelligence

    Ernest Davis. “Ethical guidelines for a superintelligence.” In: Artif. Intell. 220 (2015), pp. 121– 124

    work page 2015

  39. [39]

    Maximum likelihood estimation of observer error-rates using the EM algorithm

    Alexander Philip Dawid and Allan M Skene. “Maximum likelihood estimation of observer error-rates using the EM algorithm”. In: Applied statistics (1979), pp. 20–28. 23

    work page 1979

  40. [40]

    Feudal reinforcement learning

    Peter Dayan and Geoffrey E Hinton. “Feudal reinforcement learning”. In: Advances in neural information processing systems. Morgan Kaufmann Publishers. 1993, pp. 271–271

    work page 1993

  41. [41]

    Multi-objective optimization

    Kalyanmoy Deb. “Multi-objective optimization”. In: Search methodologies. Springer, 2014, pp. 403–449

    work page 2014

  42. [42]

    Learning what to value

    Daniel Dewey. “Learning what to value”. In: Artificial General Intelligence . Springer, 2011, pp. 309–314

    work page 2011

  43. [43]

    Reinforcement learning and the reward engineering principle

    Daniel Dewey. “Reinforcement learning and the reward engineering principle”. In: 2014 AAAI Spring Symposium Series . 2014

    work page 2014

  44. [44]

    Unsupervised super- vised learning i: Estimating classification and regression errors without labels

    Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. “Unsupervised super- vised learning i: Estimating classification and regression errors without labels”. In: The Jour- nal of Machine Learning Research 11 (2010), pp. 1323–1351

    work page 2010

  45. [45]

    Learning from labeled features using generalized expectation criteria

    Gregory Druck, Gideon Mann, and Andrew McCallum. “Learning from labeled features using generalized expectation criteria”. In:Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval . ACM. 2008, pp. 595–602

    work page 2008

  46. [46]

    Fairness through awareness

    Cynthia Dwork et al. “Fairness through awareness”. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference. ACM. 2012, pp. 214–226

    work page 2012

  47. [47]

    Computers and the theory of statistics: thinking the unthinkable

    Bradley Efron. “Computers and the theory of statistics: thinking the unthinkable”. In: SIAM review 21.4 (1979), pp. 460–480

    work page 1979

  48. [48]

    Learning the preferences of ignorant, inconsistent agents

    Owain Evans, Andreas Stuhlm¨ uller, and Noah D Goodman. “Learning the preferences of ignorant, inconsistent agents”. In: arXiv preprint arXiv:1512.05832 (2015)

    work page internal anchor Pith review arXiv 2015

  49. [49]

    Avoiding wireheading with value reinforcement learning

    Tom Everitt and Marcus Hutter. “Avoiding wireheading with value reinforcement learning”. In: arXiv preprint arXiv:1605.03143 (2016)

    work page internal anchor Pith review arXiv 2016

  50. [50]

    Self-Modification of Policy and Utility Function in Rational Agents

    Tom Everitt et al. “Self-Modification of Policy and Utility Function in Rational Agents”. In: arXiv preprint arXiv:1605.03142 (2016)

    work page internal anchor Pith review arXiv 2016

  51. [51]

    Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimization

    Chelsea Finn, Sergey Levine, and Pieter Abbeel. “Guided Cost Learning: Deep Inverse Op- timal Control via Policy Optimization”. In: arXiv preprint arXiv:1603.00448 (2016)

    work page Pith review arXiv 2016

  52. [52]

    The future of employment: how susceptible are jobs to computerisation

    Carl Benedikt Frey and Michael A Osborne. “The future of employment: how susceptible are jobs to computerisation”. In: Retrieved September 7 (2013), p. 2013

    work page 2013

  53. [53]

    Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning

    Yarin Gal and Zoubin Ghahramani. “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning”. In: arXiv preprint arXiv:1506.02142 (2015)

    work page Pith review arXiv 2015

  54. [54]

    Learning with drift detection

    Joao Gama et al. “Learning with drift detection”. In: Advances in artificial intelligence–SBIA

    work page

  55. [55]

    Springer, 2004, pp. 286–295

    work page 2004

  56. [56]

    A Comprehensive Survey on Safe Reinforcement Learning

    Javier Garc´ ıa and Fernando Fern´ andez. “A Comprehensive Survey on Safe Reinforcement Learning”. In: Journal of Machine Learning Research 16 (2015), pp. 1437–1480

    work page 2015

  57. [57]

    Asymptotic Convergence in Online Learning with Unbounded Delays

    Scott Garrabrant, Nate Soares, and Jessica Taylor. “Asymptotic Convergence in Online Learning with Unbounded Delays”. In: arXiv preprint arXiv:1604.05280 (2016)

    work page internal anchor Pith review arXiv 2016

  58. [58]

    Uniform Coherence

    Scott Garrabrant et al. “Uniform Coherence”. In: arXiv preprint arXiv:1604.05288 (2016)

    work page internal anchor Pith review arXiv 2016

  59. [59]

    Trusted Machine Learning for Probabilistic Models

    Shalini Ghosh et al. “Trusted Machine Learning for Probabilistic Models”. In: Reliable Ma- chine Learning in the Wild at ICML 2016 (2016)

    work page 2016

  60. [60]

    Amplify scientific discovery with artificial intelligence

    Yolanda Gil et al. “Amplify scientific discovery with artificial intelligence”. In: Science 346.6206 (2014), pp. 171–172

    work page 2014

  61. [61]

    Twitter sentiment classification using distant supervision

    Alec Go, Richa Bhayani, and Lei Huang. “Twitter sentiment classification using distant supervision”. In: CS224N Project Report, Stanford 1 (2009), p. 12

    work page 2009

  62. [62]

    Generative adversarial nets

    Ian Goodfellow et al. “Generative adversarial nets”. In: Advances in Neural Information Processing Systems. 2014, pp. 2672–2680

    work page 2014

  63. [63]

    Explaining and Harnessing Adversarial Examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. “Explaining and harnessing ad- versarial examples”. In: arXiv preprint arXiv:1412.6572 (2014)

    work page internal anchor Pith review Pith/arXiv arXiv 2014

  64. [64]

    Problems of monetary management: the UK experience

    Charles AE Goodhart. Problems of monetary management: the UK experience . Springer, 1984

    work page 1984

  65. [65]

    Neural Turing Machines

    Alex Graves, Greg Wayne, and Ivo Danihelka. “Neural turing machines”. In: arXiv preprint arXiv:1410.5401 (2014). 24

    work page internal anchor Pith review arXiv 2014

  66. [66]

    Distantly Supervised Information Extraction Using Bootstrapped Patterns

    Sonal Gupta. “Distantly Supervised Information Extraction Using Bootstrapped Patterns”. PhD thesis. Stanford University, 2015

    work page 2015

  67. [67]

    Cooperative Inverse Reinforcement Learning

    Dylan Hadfield-Menell et al. Cooperative Inverse Reinforcement Learning. 2016

    work page 2016

  68. [68]

    The Off-Switch

    Dylan Hadfield-Menell et al. “The Off-Switch”. In: (2016)

    work page 2016

  69. [69]

    Large sample properties of generalized method of moments estimators

    Lars Peter Hansen. “Large sample properties of generalized method of moments estimators”. In: Econometrica: Journal of the Econometric Society (1982), pp. 1029–1054

    work page 1982

  70. [70]

    Nobel Lecture: Uncertainty Outside and Inside Economic Models

    Lars Peter Hansen. “Nobel Lecture: Uncertainty Outside and Inside Economic Models”. In: Journal of Political Economy 122.5 (2014), pp. 945–987

    work page 2014

  71. [71]

    Tracking the best linear predictor

    Mark Herbster and Manfred K Warmuth. “Tracking the best linear predictor”. In: The Jour- nal of Machine Learning Research 1 (2001), pp. 281–309

    work page 2001

  72. [72]

    Model-based utility functions

    Bill Hibbard. “Model-based utility functions”. In: Journal of Artificial General Intelligence 3.1 (2012), pp. 1–24

    work page 2012

  73. [73]

    Kernel methods in machine learning

    Thomas Hofmann, Bernhard Sch¨ olkopf, and Alexander J Smola. “Kernel methods in machine learning”. In: The annals of statistics (2008), pp. 1171–1220

    work page 2008

  74. [74]

    Robust dynamic programming

    Garud N Iyengar. “Robust dynamic programming”. In: Mathematics of Operations Research 30.2 (2005), pp. 257–280

    work page 2005

  75. [75]

    Estimating the accuracies of multiple classifiers without labeled data

    Ariel Jaffe, Boaz Nadler, and Yuval Kluger. “Estimating the accuracies of multiple classifiers without labeled data”. In: arXiv preprint arXiv:1407.7644 (2014)

    work page internal anchor Pith review arXiv 2014

  76. [76]

    A formally verified hybrid system for the next-generation air- borne collision avoidance system

    Jean-Baptiste Jeannin et al. “A formally verified hybrid system for the next-generation air- borne collision avoidance system”. In:Tools and Algorithms for the Construction and Analysis of Systems. Springer, 2015, pp. 21–36

    work page 2015

  77. [77]

    Differential Privacy and Machine Learning: a Survey and Review

    Zhanglong Ji, Zachary C Lipton, and Charles Elkan. “Differential privacy and machine learn- ing: A survey and review”. In: arXiv preprint arXiv:1412.7584 (2014)

    work page Pith review arXiv 2014

  78. [78]

    Learning Representations for Counter- factual Inference

    Fredrik D Johansson, Uri Shalit, and David Sontag. “Learning Representations for Counter- factual Inference”. In: arXiv preprint arXiv:1605.03661 (2016)

    work page internal anchor Pith review arXiv 2016

  79. [79]

    Planning and acting in partially observable stochastic domains

    Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. “Planning and acting in partially observable stochastic domains”. In: Artificial intelligence 101.1 (1998), pp. 99– 134

    work page 1998

  80. [80]

    Neural GPUs Learn Algorithms

    Lukasz Kaiser and Ilya Sutskever. “Neural GPUs learn algorithms”. In:arXiv preprint arXiv:1511.08228 (2015)

    work page Pith review arXiv 2015

Showing first 80 references.