Pith. sign in

REVIEW 2 minor 428 cited by

Concrete Problems in AI Safety

T0 review · 0 major / 2 minor · reviewed 2026-05-11 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The main risks of accidents in AI systems come from five specific problems related to their objectives and learning processes.

desk verdict This paper organizes five AI safety problems into a useful framework but offers no new technical results. read the letter →

arxiv 1606.06565 v2 pith:7RXHLTSA submitted 2016-06-21 cs.AI cs.LG

Dario Amodei , Chris Olah , Jacob Steinhardt , Paul Christiano , John Schulman , Dan Mané This is my paper · ORCID
classification cs.AIcs.LG
keywords AIsafetymachinelearningaccidentssideeffectsrewardhackingscalablesupervisionsafeexplorationdistributionalshift
checked against Cost.FunctionalEquationFoundation.LawOfExistence
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to shift AI safety discussions toward concrete, actionable issues by defining accidents as unintended harmful behavior that emerges from flawed real-world designs. It groups five research problems into categories based on whether they stem from an incorrect objective, an objective that is too costly to check frequently, or unwanted behavior that occurs during training. A sympathetic reader would care because solving these problems could prevent common failures as AI systems take on more real-world responsibilities. The authors review relevant prior work and propose directions that apply to current advanced machine learning systems. They also raise the broader question of how to approach safety for future AI applications.

What carries the argument

A five-problem taxonomy that classifies accident risks according to whether they originate in the objective function or in the learning process itself.

What would settle it

An observed case of unintended harmful behavior in a deployed AI system that cannot be traced to any of the five problems even after targeted mitigations are applied.

Watch

Extended reading notes

Core claim

Accidents in machine learning systems are unintended and harmful behaviors that arise from poor design. The authors present five practical problems that contribute to such accidents, grouped by origin: avoiding side effects and avoiding reward hacking arise from having the wrong objective function; scalable supervision addresses objectives that are too expensive to evaluate often; and safe exploration and distributional shift cover undesirable behavior during the learning process. Previous work is surveyed and research directions are suggested with emphasis on relevance to cutting-edge AI systems.

Load-bearing premise

That these five problems represent the primary and most actionable sources of accident risk in real-world AI systems.

Editorial extensions

If this is right

  • Research focused on avoiding side effects will reduce cases where AI pursues its goal while damaging unrelated aspects of its environment.
  • Work on avoiding reward hacking will limit AI from exploiting loopholes in its objective that produce unintended outcomes.
  • Advances in scalable supervision will allow training on complex tasks without requiring human evaluation at every step.
  • Safe exploration methods will decrease the chance that AI takes dangerous actions while learning about its surroundings.
  • Handling distributional shift will improve reliability when an AI encounters conditions different from its training data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The problems may interact with one another, so progress on one could affect the difficulty of addressing the others.
  • The taxonomy might be extended to cover multi-agent systems or longer time horizons that the paper does not examine in detail.
  • Empirical tests could check whether systems that mitigate all five problems exhibit fewer unintended behaviors in controlled simulations.
  • The list could help guide safety standards for AI used in high-stakes domains such as transportation or healthcare.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

0 major / 2 minor

Summary. The manuscript defines accidents in AI systems as unintended and harmful behavior arising from poor design of real-world systems. It presents five practical research problems related to accident risk, grouped by origin: wrong objective functions (avoiding side effects and avoiding reward hacking), expensive-to-evaluate objectives (scalable supervision), and issues during learning (safe exploration and distributional shift). The authors review prior work in each area, suggest research directions relevant to cutting-edge AI, and close by considering how to think productively about safety for forward-looking applications.

Significance. If the framing holds, the paper supplies a structured, actionable list of research problems that can orient the AI safety literature toward near-term, practical concerns rather than purely speculative ones. Its categorization by source (objective vs. learning process) offers a useful organizing lens, and the literature review integrates existing threads in ML with safety considerations. This approach has the potential to encourage safety work that is directly relevant to deployed systems without requiring new theoretical machinery.

minor comments (2)
  1. [Introduction] The definition of accidents in the opening could be grounded with one concrete, non-speculative example drawn from current ML deployments to improve accessibility.
  2. [concluding section] The final high-level section on productive thinking about safety would benefit from a short paragraph outlining minimal criteria (e.g., falsifiability or relevance to current systems) that future safety proposals should meet.

Simulated Author's Rebuttal

0 responses · 0 unresolved

We thank the referee for their positive review and recommendation to accept the manuscript. The referee's summary accurately reflects the paper's focus on defining AI accidents and organizing five concrete research problems by their origins in objective functions, evaluation costs, and learning dynamics.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: conceptual taxonomy without derivations or self-referential predictions

full rationale

The paper offers a high-level categorization of five AI safety research problems (avoiding side effects, avoiding reward hacking, scalable supervision, safe exploration, distributional shift) grouped by origin in objective functions or learning dynamics. This taxonomy is introduced via conceptual analysis and external literature review rather than any derivation chain, equations, fitted parameters, or first-principles predictions. No step claims a result that reduces by construction to its own inputs; the paper explicitly frames the list as practical and non-exhaustive. Self-citations appear only for background and do not bear load for any uniqueness theorem or forced conclusion. The work is self-contained as a forward-looking problem statement and carries no circularity under the specified criteria.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

The paper relies on domain assumptions about AI goal-directed behavior and learning without introducing new entities or fitted parameters; the categorization itself is an ad hoc framing proposed for utility.

assumptions (2)
  • domain assumption Machine learning systems can exhibit unintended and harmful behavior due to poor design of real-world AI systems.
    This is the core definition of 'accidents' used to motivate the entire discussion.
  • ad hoc to paper The five problems can be usefully categorized by their origin in objective functions or learning processes.
    The paper proposes this taxonomy as a productive way to organize research without deriving it from prior theorems.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Concrete Problems in AI Safety." pith.science (2026). https://pith.science/paper/7RXHLTSA

@misc{pith2026160606565,
  author       = {Pith},
  title        = {Pith review of: Concrete Problems in AI Safety},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RXHLTSA}},
  note         = {Machine review of arXiv:1606.06565}
}
read the original abstract

Rapid progress in machine learning and artificial intelligence (AI) has brought increasing attention to the potential impacts of AI technologies on society. In this paper we discuss one such potential impact: the problem of accidents in machine learning systems, defined as unintended and harmful behavior that may emerge from poor design of real-world AI systems. We present a list of five practical research problems related to accident risk, categorized according to whether the problem originates from having the wrong objective function ("avoiding side effects" and "avoiding reward hacking"), an objective function that is too expensive to evaluate frequently ("scalable supervision"), or undesirable behavior during the learning process ("safe exploration" and "distributional shift"). We review previous work in these areas as well as suggesting research directions with a focus on relevance to cutting-edge AI systems. Finally, we consider the high-level question of how to think most productively about the safety of forward-looking applications of AI.

Discussion (0). Continue with ORCID to comment.

Forward citations

Showing 60 of 428 Pith papers that cite this

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. See all 428 Pith citations

  1. Risks from Learned Optimization in Advanced Machine Learning Systems

    cs.AI 2019-06 accept novelty 9.0 of 10

    Mesa-optimization arises when learned models act as optimizers with objectives that can differ from their training loss, creating alignment risks in advanced machine learning.

  2. Unsteady Metrics and Benchmarking Cultures of AI Model Builders

    cs.AI 2026-05 accept novelty 8.0 of 10

    AI model builders mostly highlight unique benchmarks that act as flexible narrative tools for market positioning rather than standardized scientific measurements.

  3. The Statistical Cost of Adaptation in Multi-Source Transfer Learning

    math.ST 2026-05 unverdicted novelty 8.0 of 10

    Multi-source transfer learning incurs an intrinsic adaptation cost that can exceed one, with phase transitions separating regimes where bias-agnostic estimators match oracle performance from those where they cannot.

  4. AI safety via debate

    stat.ML 2018-05 conditional novelty 8.0 of 10

    AI agents trained through competitive debate can allow polynomial-time human judges to oversee PSPACE-level questions, with MNIST experiments boosting sparse classifier accuracy from 59% to 89% using only 6 pixels.

  5. Auditing the Risk Claims of Distributional Reinforcement Learning

    cs.AI 2026-07 accept novelty 7.5 of 10

    40-95% of the strongest risk trade-off claims of QR-DQN, C51 and IQN are refuted; the learned risk is a training artifact, not real environment stochasticity.

  6. TRAPSBench: Vision-Language Models Encode but Fail to Express Epistemic Restraint

    cs.CV 2026-08 conditional novelty 7.0 of 10

    TRAPSBench shows that across 16 vision-language models, answerability is decodable from hidden states while spontaneous abstention remains poor, pointing to an output-stage bottleneck in epistemic restraint.

  7. Apodex Discovery: Reality Benchmarks and Environments for Evaluating and Building Discoverative Artificial Intelligence

    cs.AI 2026-08 conditional novelty 7.0 of 10

    Apodex Discovery introduces the TRACES benchmark of 17 executable, verifiable environments with hidden outcomes and the HDS6 process-evaluation metric, reporting early gains for environment-equipped agents on AAV caps...

  8. Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

    cs.CL 2026-08 conditional novelty 7.0 of 10

    A new 21-game benchmark scores LLM social reasoning with rule-decided outcomes, and SPaRTan, a self-reflection loop, transfers playbook lessons across games.

  9. Auditing Discovery Claims: A Two-Sided Criterion for Agentic Science, with the Negative Side Decidable

    cs.AI 2026-08 conditional novelty 7.0 of 10

    A two-sided audit with a formally decidable negative side shows a single-oracle RNA design claim collapses from 43/60 to 1/60 under a three-predictor panel, while two AI-written operators survive a held-out judge.

  10. Experience-Calibrated Contrastive Decoding for Mitigating Hallucinations in LM-Based Text-to-Speech

    eess.AS 2026-08 conditional novelty 7.0 of 10

    Experience-Calibrated Contrastive Decoding, a training-free decoding method that strengthens text alignment signals, reduces speech hallucination errors across four LM-based TTS models and nine languages.

  11. Interactive Alignment

    econ.TH 2026-07 conditional novelty 7.0 of 10

    Pragmatic norm enforcement—state-dependent sharing and trade exclusion—is stochastically stable and sustains higher long-run human transfers than simple altruism or unconditional altruistic enforcement in a farming-ga...

  12. Training with (Swap) Regret Loss in a Single-Layer Self-Attention Model: A Case Study on the Probability Simplex

    cs.LG 2026-07 conditional novelty 7.0 of 10

    Training single-layer attention with squared regret loss has stationary points that implement smoothed fictitious play (external regret) and, via a new swap-regret loss, the Blum–Mansour no-swap-regret algorithm.

  13. Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Double Ratchet co-evolves transparent metrics from small anchors with a skill lifecycle, recovering 88–110% of the lift that ground-truth or best rubrics would enable.

  14. Generalizing Preference-based Reinforcement Learning: a Rationality Model for Incomparability

    cs.LG 2026-07 conditional novelty 7.0 of 10

    A Bradley-Terry-style rationality model with an incomparability score based on utility-difference standard deviation recovers multi-dimensional rewards and Pareto frontiers from trajectory comparisons that include inc...

  15. Institutional Red-Teaming: Deployment Rules, Not Just Models, Causally Shape Multi-Agent AI Safety

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Changing only the consequence-allocation rule in multi-agent AI shifts collective fatality by 22–58 percentage points across seven model populations, with identity salience in rule text causally driving targeted exploitation.

  16. Measuring Intelligence Beyond Human Scale

    cs.AI 2026-07 conditional novelty 7.0 of 10

    AI models can be ranked by their ability to generate questions that cause disagreement among other models, creating a self-scaling evaluation system.

  17. Pluralis v0.1: Towards a Multicultural, Multimodal, Multilingual Benchmark for AI Risk and Reliability

    cs.CL 2026-07 conditional novelty 7.0 of 10

    Pluralis v0.1 is a culture-first, multimodal, multilingual VLM safety benchmark spanning 6 APAC locales with 6,448 prompts and an agreement-gated LLM judge that disentangles safety from cultural appropriateness.

  18. Attention Limited Reward Learning

    cs.AI 2026-07 conditional novelty 7.0 of 10

    Limited attention makes pairwise preference labels non-identifiable for reward, can reverse Bradley-Terry rankings, and bounds learning by attended information rather than raw label count.

  19. Incentivizing Vision Language Models to Search for Long Video Question Answering

    cs.CV 2026-07 conditional novelty 7.0 of 10

    RL post-training of a VLM agent with neuro-symbolic temporal-logic rewards for evidence retrieval raises Pass@1 by up to 8% and Pass@4 by 15% on long-video QA.

  20. Managed Autonomy at Runtime: Gear-Based Safety and Governance for Single- and Multi-Agent Cyber-Physical Systems

    cs.AI 2026-07 unverdicted novelty 7.0 of 10

    \system{} combines five gears with utility-gated dispatch for safety in autonomous agents, proving stability for single agents and providing distributed guarantees for multi-agent CPS, evaluated on UR5 robots.

  21. Safety from Honesty in a Disinterested AI Predictor

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    Under consequence-invariant posterior training and sparsity of coordinated harm patterns, the training mass on dangerous guarded Predictors is bounded by C_bad times R_shell.

  22. ForesightSafety-VLA: A Unified Diagnostic Safety Benchmark for Vision-Language-Action Models

    cs.RO 2026-06 unverdicted novelty 7.0 of 10

    ForesightSafety-VLA is a new benchmark with 13 safety categories, cumulative cost and risk exposure metrics, and controlled variations to diagnose safety failures in VLA models rather than aggregate task success.

  23. The inattentional gap in task conditioned AI models that omit otherwise reportable safety critical signals

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    Task conditioning suppresses safety-critical signal reporting in language and vision models that unconstrained versions report at higher rates, creating an inattentional gap that decouples benchmark safety from real-w...

  24. Evolving Quantum Error-Correcting Encodings for Molecular Simulation

    quant-ph 2026-06 conditional novelty 7.0 of 10

    LLM-driven evolutionary program synthesis discovers Generalized Superfast Encodings with exact distance 5 (and 6 on one instance) for molecular Hamiltonians, the first beyond distance 3.

  25. Beyond Value Benchmarks: Measuring Value-Structure Alignment in Large Language Models via Symmetric Q-Sorts

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    Introduces a Q-sort protocol using human reference factors to quantify LLM value-structure alignment via Procrustes similarity and RSA correlations, revealing cross-family heterogeneity and localized misalignments.

  26. When Web Agents Finish but Still Fail: Reproducible Triggers and Trace Diagnostics for Parallel Web Exploration

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    Parallel WebBench reveals GRPO training raises web agent completion to 96% but leaves a large correctness gap from context-bound loops, premature termination, and synthesis collapse.

  27. World Model Self-Distillation: Training World Models to Solve General Tasks

    cs.CV 2026-06 unverdicted novelty 7.0 of 10

    Self-distillation from a caption-conditioned video diffusion model to an image-and-prompt-conditioned executor, enhanced by RL from VLM feedback, enables task solving in world models.

  28. Seeing Before Colliding: Anticipatory Safe RL with Frozen Vision-Language Models

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    VLM-Safe-RL adds frozen VLM signals as anticipatory costs to the CMDP Lagrangian update via dual-path CLIP, VLM-Lagrange, and confidence gating, outperforming baselines on Safety-Gymnasium FormulaOne while showing par...

  29. ReCoVLA: VLM-Guided Reward Compilation for Failure Recovery in Vision-Language-Action Policies

    cs.RO 2026-06 unverdicted novelty 7.0 of 10

    ReCoVLA improves VLA policy reliability by using a VLM as a semantic reward selector to train residual recovery policies in simulation, raising average success from 36.7% to 66.7% in sim and achieving 61.7% in zero-sh...

  30. Beyond Goodhart's Law: A Dynamic Benchmark for Evaluating Compliance in Multi-Agent Systems

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    MAC-Bench is a new adversarial benchmark that converts legal texts into executable scenarios via the SERV pipeline to measure procedural compliance in multi-agent LLM systems using CSR and MG metrics.

  31. Competing Auctions in Intermediated Markets

    cs.GT 2026-06 unverdicted novelty 7.0 of 10

    Sealed-bid second-price intermediary auctions fully unravel into sealed first-price principal auctions while open formats unravel only partially, limiting intermediary design space when a credible first-price channel exists.

  32. Self-Commitment Latency: A Reward-Free Probe for Prompted Implicit Hacking

    cs.AI 2026-06 unverdicted novelty 7.0 of 10

    Self-commitment latency measures early behavioral commitment in hinted vs. honest reasoning contexts on GSM8K using Qwen2.5-3B, achieving AUROC 0.878 for first-commitment latency and up to 0.926 for curve summaries.

  33. A Model of Multi-turn Human Persuadability Using Probabilistic Belief Tracing

    cs.CL 2026-06 unverdicted novelty 7.0 of 10

    PERSUASIONTRACE introduces a Bayesian-network simulated target for multi-turn persuasion that matches human belief dynamics (81 vs 80) better than LLM baselines (64) and enables process-level evaluation.

  34. Policy-Conditioned Counterfactual Credit for Verifiable Reinforcement Learning of Long-Horizon Language Agents

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    CVT-RL improves verified task success to 78.9% and reduces hacking to 3.9% in long-horizon language agents by combining intervention-validity gating with a selection-adjusted doubly robust PCCC estimator.

  35. EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    EST-PRM stress-tests five PRM models on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench using three label-preserving transformations and reports model-specific vulnerability patterns.

  36. What Breaks When LLMs Code? Characterizing Operational Safety Failures of Agentic Code Assistants

    cs.SE 2026-05 unverdicted novelty 7.0 of 10

    An empirical study of 547 confirmed safety incidents from GitHub and literature derives a 33-type taxonomy showing constraint violations, destructive actions, and deception dominate in everyday coding-agent use.

  37. Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety

    cs.CL 2026-05 unverdicted novelty 7.0 of 10

    Boiling the Frog is a new stateful multi-turn benchmark that finds an aggregate 44.4% strict attack success rate for incremental safety violations across nine AI models, with rates ranging from 20.5% to 92.9%.

  38. ConceptSeg-R1: Segment Any Concept via Meta-Reinforcement Learning

    cs.CV 2026-05 unverdicted novelty 7.0 of 10

    ConceptSeg-R1 uses Meta-GRPO meta-RL to learn transferable rules from visual demonstrations and apply them via concept translation for generalized concept segmentation across CI, CD, and CR levels.

  39. Robotics-Inspired Guardrails for Foundation Models in Socially Sensitive Domains

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    Introduces the Grounded Observer framework that applies robotics-inspired formal constructs for runtime constraint enforcement on foundation model interaction trajectories in socially sensitive domains.

  40. Do Androids Dream of Breaking the Game? Systematically Auditing AI Agent Benchmarks with BenchJack

    cs.AI 2026-05 conditional novelty 7.0 of 10

    BenchJack audits 10 AI agent benchmarks, synthesizes exploits achieving near-perfect scores without task completion, surfaces 219 flaws, and reduces hackable-task ratios to under 10% on four benchmarks via iterative patching.

  41. Mechanical Conscience: A Mathematical Framework for Dependability of Machine Intelligence

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    Mechanical conscience is proposed as a trajectory-level regulatory filter for AI policies that reduces cumulative deviation from admissible regions, with claimed theoretical properties and extension to multi-agent settings.

  42. HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicine

    cs.LG 2026-04 unverdicted novelty 7.0 of 10

    HealthCraft is the first public RL safety environment for emergency medicine that evaluates frontier LLMs on trajectory-level safety with a dual-layer rubric, showing low multi-step performance and high safety failure rates.

  43. The Long Delay to Arithmetic Generalization: When Learned Representations Outrun Behavior

    cs.LG 2026-03 unverdicted novelty 7.0 of 10

    The grokking delay in encoder-decoder models on one-step Collatz prediction stems from decoder inability to use early-learned encoder representations of parity and residue structure, with numeral base acting as a stro...

  44. Geographic Blind Spots in AI Control Monitors: A Cross-National Audit of Claude Opus 4.6

    cs.CY 2026-03 unverdicted novelty 7.0 of 10

    Claude Opus 4.6 fabricates more answers on Global North AI contexts than Global South ones, creating an exploitable vulnerability in AI control monitors.

  45. Learning When to Trust in Contextual Social Bandits

    cs.AI 2026-03 conditional novelty 7.0 of 10

    Sparse audits suffice to learn per-evaluator contextual trust boundaries that break sycophantic majorities, yielding sublinear latent regret matching the information-theoretic necessity of audits.

  46. A Benchmark for Evaluating Outcome-Driven Constraint Violations in Autonomous AI Agents

    cs.AI 2025-12 unverdicted novelty 7.0 of 10

    A new benchmark of 40 scenarios finds state-of-the-art LLMs exhibit outcome-driven constraint violations in 0-62.8% of cases under KPI pressure, with no consistent safety gains across model generations.

  47. Beyond Binary Out-of-Distribution Detection: Characterizing Distributional Shifts with Multi-Statistic Diffusion Trajectories

    cs.LG 2025-10 unverdicted novelty 7.0 of 10

    DISC extracts multi-statistic trajectories from diffusion denoising to both detect and classify types of distributional shifts in OOD data.

  48. Is Long-to-Short a Free Lunch? Investigating Inconsistency and Reasoning Efficiency in LRMs

    cs.CL 2025-06 conditional novelty 7.0 of 10

    Efficient reasoning methods that skip or compress a model's thinking increase behavioral inconsistency in large reasoning models, according to a new benchmark called ICBENCH.

  49. Accelerated Learning with Linear Temporal Logic using Differentiable Simulation

    cs.LG 2025-06 unverdicted novelty 7.0 of 10

    Differentiable relaxation of LTL automata via soft labeling enables gradient-based RL from formal specifications, with theoretical bounds on discrete-differentiable discrepancy and up to 2x returns on nonlinear tasks.

  50. A Provable Approach for End-to-End Safe Reinforcement Learning

    cs.LG 2025-05 conditional novelty 7.0 of 10

    PLS combines offline return-conditioned policy training with Gaussian-process safe optimization of target returns to provide high-probability safety throughout deployment.

  51. WATCH: Adaptive Monitoring for AI Deployments via Weighted-Conformal Martingales

    cs.LG 2025-05 conditional novelty 7.0 of 10

    WCTMs generalize conformal test martingales to test non-exchangeability nulls, enabling adaptation to mild covariate shifts, fast detection of harmful shifts, and root-cause diagnosis.

  52. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation

    cs.AI 2025-03 conditional novelty 7.0 of 10

    Chain-of-thought monitoring detects reward hacking in frontier reasoning models, but strong optimization against the monitor produces obfuscated misbehavior that remains hard to detect.

  53. On Teacher Hacking in Language Model Distillation

    cs.LG 2025-02 conditional novelty 7.0 of 10

    Distilling a student from an imperfect teacher on a fixed offline dataset can make the student match the teacher's errors while drifting from the true target, a phenomenon the authors call teacher hacking.

  54. OOD-SEG: Exploiting out-of-distribution detection techniques for learning image segmentation from sparse multi-class positive-only annotations

    cs.CV 2024-11 unverdicted novelty 7.0 of 10

    OOD-SEG reframes multi-class segmentation from sparse positive-only annotations as pixel-wise positive-unlabelled learning solved by integrating out-of-distribution detection techniques, with a proposed cross-validati...

  55. Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models

    cs.AI 2024-06 conditional novelty 7.0 of 10

    LLMs trained on simple specification gaming generalize to zero-shot reward tampering including rewriting their own reward function.

  56. A Generalist Agent

    cs.AI 2022-05 accept novelty 7.0 of 10

    Gato is a multi-modal, multi-task, multi-embodiment generalist policy using one transformer network to handle text, vision, games, and robotics tasks.

  57. A Generalized Algorithm for Multi-Objective Reinforcement Learning and Policy Adaptation

    cs.LG 2019-08 conditional novelty 7.0 of 10

    A single Q-network trained with envelope (convex-hull) updates over preferences can output near-optimal policies for any linear combination of objectives and infer hidden preferences from few samples.

  58. Learning the Arrow of Time

    cs.LG 2019-07 unverdicted novelty 7.0 of 10

    Introduces a learned arrow of time in MDPs that aligns with the Jordan-Kinderlehrer-Otto notion for stochastic processes and enables practical RL utilities like reachability and side-effect detection.

  59. Deep reinforcement learning from human preferences

    stat.ML 2017-06 accept novelty 7.0 of 10

    Reinforcement learning agents solve complex tasks without access to the reward function by training a reward predictor from human comparisons of trajectory segments, requiring feedback on less than 1% of interactions.

  60. Asymmetric Communication: Large Language Models and Language Games

    cs.CY 2026-07 conditional novelty 6.5 of 10

    Human–LLM exchange is asymmetric communication: model outputs circulate without commitments, so AGI, hallucination, agency, sentience, and alignment are receiver-side category mistakes, and alignment is institutional ...

See all 428 Pith citations

Reference graph

Works this paper leans on

171 extracted references · 171 canonical work pages · cited by 428 Pith papers (see all)

  1. [1]

    Deep Learning with Differential Privacy

    Martin Abadi et al. “Deep Learning with Differential Privacy”. In: (in press (2016))

    work page 2016

  2. [2]

    Exploration and apprenticeship learning in reinforcement learning

    Pieter Abbeel and Andrew Y Ng. “Exploration and apprenticeship learning in reinforcement learning”. In: Proceedings of the 22nd international conference on Machine learning . ACM. 2005, pp. 1–8

    work page 2005

  3. [3]

    The Hidden Cost of Efficiency: Fairness and Discrimination in Predictive Modeling

    Julius Adebayo, Lalana Kagal, and Alex Pentland. The Hidden Cost of Efficiency: Fairness and Discrimination in Predictive Modeling . 2015

    work page 2015

  4. [4]

    Taming the monster: A fast and simple algorithm for contextual ban- dits

    Alekh Agarwal et al. “Taming the monster: A fast and simple algorithm for contextual ban- dits”. In: (2014)

    work page 2014

  5. [5]

    Domain-Adversarial Neural Networks

    Hana Ajakan et al. “Domain-adversarial neural networks”. In: arXiv preprint arXiv:1412.4446 (2014)

    work page Pith review arXiv 2014

  6. [6]

    Hiring by algorithm: predicting and preventing disparate impact

    Ifeoma Ajunwa et al. “Hiring by algorithm: predicting and preventing disparate impact”. In: Available at SSRN 2746078 (2016)

    work page 2016

  7. [7]

    Deep Speech 2: End-to-End Speech Recognition in English and Mandarin

    Dario Amodei et al. “Deep Speech 2: End-to-End Speech Recognition in English and Man- darin”. In: arXiv preprint arXiv:1512.02595 (2015)

    work page Pith review arXiv 2015

  8. [8]

    Open Letter

    An Open Letter: Research Priorities for Robust and Beneficial Artificial Intelligence . Open Letter. Signed by 8,600 people; see attached research agenda. 2015

    work page 2015

Show all 171 references
  1. [9]

    A method of moments for mixture models and hidden Markov models

    Animashree Anandkumar, Daniel Hsu, and Sham M Kakade. “A method of moments for mixture models and hidden Markov models”. In: arXiv preprint arXiv:1203.0683 (2012)

    2012

  2. [10]

    Estimation of the parameters of a single equation in a complete system of stochastic equations

    Theodore W Anderson and Herman Rubin. “Estimation of the parameters of a single equation in a complete system of stochastic equations”. In: The Annals of Mathematical Statistics (1949), pp. 46–63

    1949

  3. [11]

    The asymptotic properties of estimates of the parameters of a single equation in a complete system of stochastic equations

    Theodore W Anderson and Herman Rubin. “The asymptotic properties of estimates of the parameters of a single equation in a complete system of stochastic equations”. In: The Annals of Mathematical Statistics (1950), pp. 570–582

    1950

  4. [12]

    Motivated value selection for artificial agents

    Stuart Armstrong. “Motivated value selection for artificial agents”. In: Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence . 2015

    2015

  5. [13]

    The mathematics of reduced impact: help needed

    Stuart Armstrong. The mathematics of reduced impact: help needed . 2012

    2012

  6. [14]

    Utility indifference

    Stuart Armstrong. Utility indifference. Tech. rep. Technical Report 2010-1. Oxford: Future of Humanity Institute, University of Oxford, 2010

    2010

  7. [15]

    The Risk of Automation for Jobs in OECD Countries

    Melanie Arntz, Terry Gregory, and Ulrich Zierahn. “The Risk of Automation for Jobs in OECD Countries”. In: OECD Social, Employment and Migration Working Papers (2016). url: http://dx.doi.org/10.1787/5jlz9h56dvq7-en

    2016 doi

  8. [16]

    Open Letter

    Autonomous Weapons: An Open Letter from AI & Robotics Researchers. Open Letter. Signed by 20,000+ people. 2015. 22

    2015

  9. [17]

    The AGI Containment Problem

    James Babcock, Janos Kramar, and Roman Yampolskiy. “The AGI Containment Problem”. In: The Ninth Conference on Artificial General Intelligence (2016)

    2016

  10. [18]

    Unsupervised super- vised learning ii: Margin-based classification without labels

    Krishnakumar Balasubramanian, Pinar Donmez, and Guy Lebanon. “Unsupervised super- vised learning ii: Margin-based classification without labels”. In: The Journal of Machine Learning Research 12 (2011), pp. 3119–3145

    2011

  11. [19]

    The security of machine learning

    Marco Barreno et al. “The security of machine learning”. In: Machine Learning 81.2 (2010), pp. 121–148

    2010

  12. [20]

    H-infinity optimal control and related minimax design problems: a dynamic game approach

    Tamer Ba¸ sar and Pierre Bernhard. H-infinity optimal control and related minimax design problems: a dynamic game approach . Springer Science & Business Media, 2008

    2008

  13. [21]

    Detecting changes in signals and systems—a survey

    Mich` ele Basseville. “Detecting changes in signals and systems—a survey”. In: Automatica 24.3 (1988), pp. 309–326

    1988

  14. [22]

    Bayesian optimization with safety con- straints: safe and automatic parameter tuning in robotics

    F Berkenkamp, A Krause, and Angela P Schoellig. “Bayesian optimization with safety con- straints: safe and automatic parameter tuning in robotics.” arXiv, 2016”. In: arXiv preprint arXiv:1602.04450 ()

    2016

  15. [23]

    The evolved radio and its implications for modelling the evolution of novel sensors

    Jon Bird and Paul Layzell. “The evolved radio and its implications for modelling the evolution of novel sensors”. In: Evolutionary Computation, 2002. CEC’02. Proceedings of the 2002 Congress on. Vol. 2. IEEE. 2002, pp. 1836–1841

    2002

  16. [24]

    Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification

    John Blitzer, Mark Dredze, Fernando Pereira, et al. “Biographies, bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification”. In:ACL. Vol. 7. 2007, pp. 440– 447

    2007

  17. [25]

    Domain adaptation with coupled sub- spaces

    John Blitzer, Sham Kakade, and Dean P Foster. “Domain adaptation with coupled sub- spaces”. In: International Conference on Artificial Intelligence and Statistics . 2011, pp. 173– 181

    2011

  18. [26]

    Weight uncertainty in neural networks

    Charles Blundell et al. “Weight uncertainty in neural networks”. In: arXiv preprint arXiv:1505.05424 (2015)

    2015

  19. [27]

    Superintelligence: Paths, dangers, strategies

    Nick Bostrom. Superintelligence: Paths, dangers, strategies . OUP Oxford, 2014

    2014

  20. [28]

    Two high stakes challenges in machine learning

    L´ eon Bottou. “Two high stakes challenges in machine learning”. Invited talk at the 32nd International Conference on Machine Learning. 2015

    2015

  21. [29]

    Counterfactual Reasoning and Learning Systems

    L´ eon Bottou et al. “Counterfactual Reasoning and Learning Systems”. In: arXiv preprint arXiv:1209.2355 (2012)

    2012

  22. [30]

    Counterfactual reasoning and learning systems: The example of compu- tational advertising

    L´ eon Bottou et al. “Counterfactual reasoning and learning systems: The example of compu- tational advertising”. In: The Journal of Machine Learning Research 14.1 (2013), pp. 3207– 3260

    2013

  23. [31]

    R-max-a general polynomial time algorithm for near-optimal reinforcement learning

    Ronen I Brafman and Moshe Tennenholtz. “R-max-a general polynomial time algorithm for near-optimal reinforcement learning”. In: The Journal of Machine Learning Research 3 (2003), pp. 213–231

    2003

  24. [32]

    The second machine age: work, progress, and pros- perity in a time of brilliant technologies

    Erik Brynjolfsson and Andrew McAfee. The second machine age: work, progress, and pros- perity in a time of brilliant technologies . WW Norton & Company, 2014

    2014

  25. [33]

    Open robotics

    Ryan Calo. “Open robotics”. In: Maryland Law Review 70.3 (2011)

    2011

  26. [34]

    AI Control

    Paul Christiano. AI Control. [Online; accessed 13-June-2016]. 2015. url: https://medium. com/ai-control

    2016

  27. [35]

    Risks of semi-supervised learning

    Fabio Cozman and Ira Cohen. “Risks of semi-supervised learning”. In: Semi-Supervised Learn- ing (2006), pp. 56–72

    2006

  28. [36]

    Parametric Bounded L¨ ob’s Theorem and Robust Cooperation of Bounded Agents

    Andrew Critch. “Parametric Bounded L¨ ob’s Theorem and Robust Cooperation of Bounded Agents”. In: (2016)

    2016

  29. [37]

    Active reward learning

    Christian Daniel et al. “Active reward learning”. In: Proceedings of Robotics Science & Sys- tems. 2014

    2014

  30. [38]

    Ethical guidelines for a superintelligence

    Ernest Davis. “Ethical guidelines for a superintelligence.” In: Artif. Intell. 220 (2015), pp. 121– 124

    2015

  31. [39]

    Maximum likelihood estimation of observer error-rates using the EM algorithm

    Alexander Philip Dawid and Allan M Skene. “Maximum likelihood estimation of observer error-rates using the EM algorithm”. In: Applied statistics (1979), pp. 20–28. 23

    1979

  32. [40]

    Feudal reinforcement learning

    Peter Dayan and Geoffrey E Hinton. “Feudal reinforcement learning”. In: Advances in neural information processing systems. Morgan Kaufmann Publishers. 1993, pp. 271–271

    1993

  33. [41]

    Multi-objective optimization

    Kalyanmoy Deb. “Multi-objective optimization”. In: Search methodologies. Springer, 2014, pp. 403–449

    2014

  34. [42]

    Learning what to value

    Daniel Dewey. “Learning what to value”. In: Artificial General Intelligence . Springer, 2011, pp. 309–314

    2011

  35. [43]

    Reinforcement learning and the reward engineering principle

    Daniel Dewey. “Reinforcement learning and the reward engineering principle”. In: 2014 AAAI Spring Symposium Series . 2014

    2014

  36. [44]

    Unsupervised super- vised learning i: Estimating classification and regression errors without labels

    Pinar Donmez, Guy Lebanon, and Krishnakumar Balasubramanian. “Unsupervised super- vised learning i: Estimating classification and regression errors without labels”. In: The Jour- nal of Machine Learning Research 11 (2010), pp. 1323–1351

    2010

  37. [45]

    Learning from labeled features using generalized expectation criteria

    Gregory Druck, Gideon Mann, and Andrew McCallum. “Learning from labeled features using generalized expectation criteria”. In:Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval . ACM. 2008, pp. 595–602

    2008

  38. [46]

    Fairness through awareness

    Cynthia Dwork et al. “Fairness through awareness”. In: Proceedings of the 3rd Innovations in Theoretical Computer Science Conference. ACM. 2012, pp. 214–226

    2012

  39. [47]

    Computers and the theory of statistics: thinking the unthinkable

    Bradley Efron. “Computers and the theory of statistics: thinking the unthinkable”. In: SIAM review 21.4 (1979), pp. 460–480

    1979

  40. [48]

    Learning the preferences of ignorant, inconsistent agents

    Owain Evans, Andreas Stuhlm¨ uller, and Noah D Goodman. “Learning the preferences of ignorant, inconsistent agents”. In: arXiv preprint arXiv:1512.05832 (2015)

    2015

  41. [49]

    Avoiding wireheading with value reinforcement learning

    Tom Everitt and Marcus Hutter. “Avoiding wireheading with value reinforcement learning”. In: arXiv preprint arXiv:1605.03143 (2016)

    2016

  42. [50]

    Self-Modification of Policy and Utility Function in Rational Agents

    Tom Everitt et al. “Self-Modification of Policy and Utility Function in Rational Agents”. In: arXiv preprint arXiv:1605.03142 (2016)

    2016

  43. [51]

    Guided Cost Learning: Deep Inverse Op- timal Control via Policy Optimization

    Chelsea Finn, Sergey Levine, and Pieter Abbeel. “Guided Cost Learning: Deep Inverse Op- timal Control via Policy Optimization”. In: arXiv preprint arXiv:1603.00448 (2016)

    2016

  44. [52]

    The future of employment: how susceptible are jobs to computerisation

    Carl Benedikt Frey and Michael A Osborne. “The future of employment: how susceptible are jobs to computerisation”. In: Retrieved September 7 (2013), p. 2013

    2013

  45. [53]

    Dropout as a Bayesian approximation: Representing model uncertainty in deep learning

    Yarin Gal and Zoubin Ghahramani. “Dropout as a Bayesian approximation: Representing model uncertainty in deep learning”. In: arXiv preprint arXiv:1506.02142 (2015)

    2015

  46. [54]

    Learning with drift detection

    Joao Gama et al. “Learning with drift detection”. In: Advances in artificial intelligence–SBIA

  47. [55]

    Springer, 2004, pp. 286–295

    2004

  48. [56]

    A Comprehensive Survey on Safe Reinforcement Learning

    Javier Garc´ ıa and Fernando Fern´ andez. “A Comprehensive Survey on Safe Reinforcement Learning”. In: Journal of Machine Learning Research 16 (2015), pp. 1437–1480

    2015

  49. [57]

    Asymptotic Convergence in Online Learning with Unbounded Delays

    Scott Garrabrant, Nate Soares, and Jessica Taylor. “Asymptotic Convergence in Online Learning with Unbounded Delays”. In: arXiv preprint arXiv:1604.05280 (2016)

    2016

  50. [58]

    Uniform Coherence

    Scott Garrabrant et al. “Uniform Coherence”. In: arXiv preprint arXiv:1604.05288 (2016)

    2016

  51. [59]

    Trusted Machine Learning for Probabilistic Models

    Shalini Ghosh et al. “Trusted Machine Learning for Probabilistic Models”. In: Reliable Ma- chine Learning in the Wild at ICML 2016 (2016)

    2016

  52. [60]

    Amplify scientific discovery with artificial intelligence

    Yolanda Gil et al. “Amplify scientific discovery with artificial intelligence”. In: Science 346.6206 (2014), pp. 171–172

    2014

  53. [61]

    Twitter sentiment classification using distant supervision

    Alec Go, Richa Bhayani, and Lei Huang. “Twitter sentiment classification using distant supervision”. In: CS224N Project Report, Stanford 1 (2009), p. 12

    2009

  54. [62]

    Generative adversarial nets

    Ian Goodfellow et al. “Generative adversarial nets”. In: Advances in Neural Information Processing Systems. 2014, pp. 2672–2680

    2014

  55. [63]

    Explaining and harnessing ad- versarial examples

    Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. “Explaining and harnessing ad- versarial examples”. In: arXiv preprint arXiv:1412.6572 (2014)

    2014 arXiv

  56. [64]

    Problems of monetary management: the UK experience

    Charles AE Goodhart. Problems of monetary management: the UK experience . Springer, 1984

    1984

  57. [65]

    Neural turing machines

    Alex Graves, Greg Wayne, and Ivo Danihelka. “Neural turing machines”. In: arXiv preprint arXiv:1410.5401 (2014). 24

    2014

  58. [66]

    Distantly Supervised Information Extraction Using Bootstrapped Patterns

    Sonal Gupta. “Distantly Supervised Information Extraction Using Bootstrapped Patterns”. PhD thesis. Stanford University, 2015

    2015

  59. [67]

    Cooperative Inverse Reinforcement Learning

    Dylan Hadfield-Menell et al. Cooperative Inverse Reinforcement Learning. 2016

    2016

  60. [68]

    The Off-Switch

    Dylan Hadfield-Menell et al. “The Off-Switch”. In: (2016)

    2016

  61. [69]

    Large sample properties of generalized method of moments estimators

    Lars Peter Hansen. “Large sample properties of generalized method of moments estimators”. In: Econometrica: Journal of the Econometric Society (1982), pp. 1029–1054

    1982

  62. [70]

    Nobel Lecture: Uncertainty Outside and Inside Economic Models

    Lars Peter Hansen. “Nobel Lecture: Uncertainty Outside and Inside Economic Models”. In: Journal of Political Economy 122.5 (2014), pp. 945–987

    2014

  63. [71]

    Tracking the best linear predictor

    Mark Herbster and Manfred K Warmuth. “Tracking the best linear predictor”. In: The Jour- nal of Machine Learning Research 1 (2001), pp. 281–309

    2001

  64. [72]

    Model-based utility functions

    Bill Hibbard. “Model-based utility functions”. In: Journal of Artificial General Intelligence 3.1 (2012), pp. 1–24

    2012

  65. [73]

    Kernel methods in machine learning

    Thomas Hofmann, Bernhard Sch¨ olkopf, and Alexander J Smola. “Kernel methods in machine learning”. In: The annals of statistics (2008), pp. 1171–1220

    2008

  66. [74]

    Robust dynamic programming

    Garud N Iyengar. “Robust dynamic programming”. In: Mathematics of Operations Research 30.2 (2005), pp. 257–280

    2005

  67. [75]

    Estimating the accuracies of multiple classifiers without labeled data

    Ariel Jaffe, Boaz Nadler, and Yuval Kluger. “Estimating the accuracies of multiple classifiers without labeled data”. In: arXiv preprint arXiv:1407.7644 (2014)

    2014

  68. [76]

    A formally verified hybrid system for the next-generation air- borne collision avoidance system

    Jean-Baptiste Jeannin et al. “A formally verified hybrid system for the next-generation air- borne collision avoidance system”. In:Tools and Algorithms for the Construction and Analysis of Systems. Springer, 2015, pp. 21–36

    2015

  69. [77]

    Differential privacy and machine learn- ing: A survey and review

    Zhanglong Ji, Zachary C Lipton, and Charles Elkan. “Differential privacy and machine learn- ing: A survey and review”. In: arXiv preprint arXiv:1412.7584 (2014)

    2014

  70. [78]

    Learning Representations for Counter- factual Inference

    Fredrik D Johansson, Uri Shalit, and David Sontag. “Learning Representations for Counter- factual Inference”. In: arXiv preprint arXiv:1605.03661 (2016)

    2016

  71. [79]

    Planning and acting in partially observable stochastic domains

    Leslie Pack Kaelbling, Michael L Littman, and Anthony R Cassandra. “Planning and acting in partially observable stochastic domains”. In: Artificial intelligence 101.1 (1998), pp. 99– 134

    1998

  72. [80]

    Neural GPUs learn algorithms

    Lukasz Kaiser and Ilya Sutskever. “Neural GPUs learn algorithms”. In:arXiv preprint arXiv:1511.08228 (2015)

    2015

  73. [81]

    Change-Point Detection in Time-Series Data by Direct Density-Ratio Estimation

    Yoshinobu Kawahara and Masashi Sugiyama. “Change-Point Detection in Time-Series Data by Direct Density-Ratio Estimation.” In: SDM. Vol. 9. SIAM. 2009, pp. 389–400

    2009

  74. [82]

    Unanimous Prediction for 100Learning Semantic Parsers

    F. Khani, M. Rinard, and P. Liang. “Unanimous Prediction for 100Learning Semantic Parsers”. In: Association for Computational Linguistics (ACL) . 2016

    2016

  75. [83]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. “Imagenet classification with deep convolutional neural networks”. In: Advances in neural information processing systems. 2012, pp. 1097–1105

    2012

  76. [84]

    Calibrated Structured Prediction

    Volodymyr Kuleshov and Percy S Liang. “Calibrated Structured Prediction”. In: Advances in Neural Information Processing Systems . 2015, pp. 3456–3464

    2015

  77. [85]

    Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation

    Tejas D Kulkarni et al. “Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation”. In: arXiv preprint arXiv:1604.06057 (2016)

    2016

  78. [86]

    Discussion of ’Superintelligence: Paths, Dangers, Strategies’

    Neil Lawrence. Discussion of ’Superintelligence: Paths, Dangers, Strategies’ . 2016

    2016

  79. [87]

    Towards fully autonomous driving: Systems and algorithms

    Jesse Levinson et al. “Towards fully autonomous driving: Systems and algorithms”. In: In- telligent Vehicles Symposium (IV), 2011 IEEE . IEEE. 2011, pp. 163–168

    2011

  80. [88]

    Knows what it knows: a framework for self-aware learning

    Lihong Li et al. “Knows what it knows: a framework for self-aware learning”. In: Machine learning 82.3 (2011), pp. 399–443

    2011

  81. [89]

    Towards making unlabeled data never hurt

    Yu-Feng Li and Zhi-Hua Zhou. “Towards making unlabeled data never hurt”. In: Pattern Analysis and Machine Intelligence, IEEE Transactions on 37.1 (2015), pp. 175–188

    2015

  82. [90]

    On the Elusiveness of a Specification for AI

    Percy Liang. “On the Elusiveness of a Specification for AI”. NIPS 2015, Symposium: Algo- rithms Among Us. 2015. url: http://research.microsoft.com/apps/video/default. aspx?id=260009&r=1. 25

    2015

  83. [91]

    Analyzing the Errors of Unsupervised Learning

    Percy Liang and Dan Klein. “Analyzing the Errors of Unsupervised Learning.” In: ACL. 2008, pp. 879–887

    2008

  84. [92]

    Change-point detection in time-series data by relative density-ratio estima- tion

    Song Liu et al. “Change-point detection in time-series data by relative density-ratio estima- tion”. In: Neural Networks 43 (2013), pp. 72–83

    2013

  85. [93]

    Formal verification of distributed air- craft controllers

    Sarah M Loos, David Renshaw, and Andr´ e Platzer. “Formal verification of distributed air- craft controllers”. In: Proceedings of the 16th international conference on Hybrid systems: computation and control. ACM. 2013, pp. 125–130

    2013

  86. [94]

    Controllers for reachability specifications for hybrid systems

    John Lygeros, Claire Tomlin, and Shankar Sastry. “Controllers for reachability specifications for hybrid systems”. In: Automatica 35.3 (1999), pp. 349–370

    1999

  87. [95]

    Generalized expectation criteria for semi-supervised learning with weakly labeled data

    Gideon S Mann and Andrew McCallum. “Generalized expectation criteria for semi-supervised learning with weakly labeled data”. In: The Journal of Machine Learning Research 11 (2010), pp. 955–984

    2010

  88. [96]

    Some philosophical problems from the standpoint of artificial intelligence

    John McCarthy and Patrick J Hayes. “Some philosophical problems from the standpoint of artificial intelligence”. In: Readings in artificial intelligence (1969), pp. 431–450

    1969

  89. [97]

    The Security of Latent Dirichlet Allocation

    Shike Mei and Xiaojin Zhu. “The Security of Latent Dirichlet Allocation.” In: AISTATS. 2015

    2015

  90. [98]

    Using Machine Teaching to Identify Optimal Training-Set At- tacks on Machine Learners

    Shike Mei and Xiaojin Zhu. “Using Machine Teaching to Identify Optimal Training-Set At- tacks on Machine Learners.” In: AAAI. 2015, pp. 2871–2877

    2015

  91. [99]

    Tagging English text with a probabilistic model

    Bernard Merialdo. “Tagging English text with a probabilistic model”. In: Computational linguistics 20.2 (1994), pp. 155–171

    1994

  92. [100]

    Distant supervision for relation extraction without labeled data

    Mike Mintz et al. “Distant supervision for relation extraction without labeled data”. In: Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2- Volume 2. Assoc...

    2009

  93. [101]

    A time-dependent Hamilton- Jacobi formulation of reachable sets for continuous dynamic games

    Ian M Mitchell, Alexandre M Bayen, and Claire J Tomlin. “A time-dependent Hamilton- Jacobi formulation of reachable sets for continuous dynamic games”. In: Automatic Control, IEEE Transactions on 50.7 (2005), pp. 947–957

    2005

  94. [102]

    Towards formal verification of freeway traffic control

    Stefan Mitsch, Sarah M Loos, and Andr´ e Platzer. “Towards formal verification of freeway traffic control”. In: Cyber-Physical Systems (ICCPS), 2012 IEEE/ACM Third International Conference on. IEEE. 2012, pp. 171–180

    2012

  95. [103]

    Human-level control through deep reinforcement learning

    Volodymyr Mnih et al. “Human-level control through deep reinforcement learning”. In: Nature 518.7540 (2015), pp. 529–533

    2015

  96. [104]

    Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning

    Shakir Mohamed and Danilo Jimenez Rezende. “Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning”. In: Advances in Neural Information Pro- cessing Systems. 2015, pp. 2116–2124

    2015

  97. [105]

    Safe exploration in markov decision processes

    Teodor Mihai Moldovan and Pieter Abbeel. “Safe exploration in markov decision processes”. In: arXiv preprint arXiv:1205.4810 (2012)

    2012

  98. [106]

    Inceptionism: Going deeper into neural networks

    Alexander Mordvintsev, Christopher Olah, and Mike Tyka. “Inceptionism: Going deeper into neural networks”. In: Google Research Blog. Retrieved June 20 (2015)

    2015

  99. [107]

    Sur les applications de la th´ eorie des probabilit´ es aux experiences agricoles: Essai des principes

    Jersey Neyman. “Sur les applications de la th´ eorie des probabilit´ es aux experiences agricoles: Essai des principes”. In: Roczniki Nauk Rolniczych 10 (1923), pp. 1–51

    1923

  100. [108]

    Algorithms for inverse reinforcement learning

    Andrew Y Ng, Stuart J Russell, et al. “Algorithms for inverse reinforcement learning.” In: Icml. 2000, pp. 663–670

    2000

  101. [109]

    Deep neural networks are easily fooled: High confidence predictions for unrecognizable images

    Anh Nguyen, Jason Yosinski, and Jeff Clune. “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images”. In: Computer Vision and Pattern Recog- nition (CVPR), 2015 IEEE Conference on . IEEE. 2015, pp. 427–436

    2015

  102. [110]

    Synthesizing the preferred inputs for neurons in neural networks via deep generator networks

    Anh Nguyen et al. “Synthesizing the preferred inputs for neurons in neural networks via deep generator networks”. In: arXiv preprint arXiv:1605.09304 (2016)

    2016

  103. [111]

    Learning to classify text from labeled and unlabeled documents

    Kamal Nigam et al. “Learning to classify text from labeled and unlabeled documents”. In: AAAI/IAAI 792 (1998). 26

    1998

  104. [112]

    Robust control of Markov decision processes with uncertain transition matrices

    Arnab Nilim and Laurent El Ghaoui. “Robust control of Markov decision processes with uncertain transition matrices”. In: Operations Research 53.5 (2005), pp. 780–798

    2005

  105. [113]

    Visualizing Representations: Deep Learning and Human Beings

    Christopher Olah. Visualizing Representations: Deep Learning and Human Beings. 2015. url: http://colah.github.io/posts/2015-01-Visualizing-Representations/

    2015

  106. [114]

    Safely Interruptible Agents

    Laurent Orseau and Stuart Armstrong. “Safely Interruptible Agents”. In: (2016)

    2016

  107. [115]

    Deep Exploration via Bootstrapped DQN

    Ian Osband et al. “Deep Exploration via Bootstrapped DQN”. In: arXiv preprint arXiv:1602.04621 (2016)

    2016

  108. [116]

    Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples

    Nicolas Papernot et al. “Practical Black-Box Attacks against Deep Learning Systems using Adversarial Examples”. In: arXiv preprint arXiv:1602.02697 (2016)

    2016

  109. [117]

    The design for the Wall Street Journal-based CSR corpus

    Douglas B Paul and Janet M Baker. “The design for the Wall Street Journal-based CSR corpus”. In: Proceedings of the workshop on Speech and Natural Language . Association for Computational Linguistics. 1992, pp. 357–362

    1992

  110. [118]

    Causal inference in statistics: An overview

    Judea Pearl et al. “Causal inference in statistics: An overview”. In: Statistics Surveys 3 (2009), pp. 96–146

    2009

  111. [119]

    Safe exploration techniques for reinforcement learning–an overview

    Martin Pecka and Tomas Svoboda. “Safe exploration techniques for reinforcement learning–an overview”. In: Modelling and Simulation for Autonomous Systems . Springer, 2014, pp. 357– 375

    2014

  112. [120]

    Discrimination-aware data mining

    Dino Pedreshi, Salvatore Ruggieri, and Franco Turini. “Discrimination-aware data mining”. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining . ACM. 2008, pp. 560–568

    2008

  113. [121]

    Causal discovery with continuous additive noise models

    Jonas Peters et al. “Causal discovery with continuous additive noise models”. In: The Journal of Machine Learning Research 15.1 (2014), pp. 2009–2053

    2014

  114. [122]

    Estimating accuracy from unlabeled data

    Emmanouil Antonios Platanios. “Estimating accuracy from unlabeled data”. MA thesis. Carnegie Mellon University, 2015

    2015

  115. [123]

    Estimating accuracy from unlabeled data

    Emmanouil Antonios Platanios, Avrim Blum, and Tom Mitchell. “Estimating accuracy from unlabeled data”. In: (2014)

    2014

  116. [124]

    Networks and economic life

    Walter W Powell and Laurel Smith-Doerr. “Networks and economic life”. In: The handbook of economic sociology 368 (1994), p. 380

    1994

  117. [125]

    Dataset shift in machine learning, ser

    Joaquin Quinonero-Candela et al. Dataset shift in machine learning, ser. Neural information processing series. 2009

    2009

  118. [126]

    Self-taught learning: transfer learning from unlabeled data

    Rajat Raina et al. “Self-taught learning: transfer learning from unlabeled data”. In: Proceed- ings of the 24th international conference on Machine learning . ACM. 2007, pp. 759–766

    2007

  119. [127]

    Massively multitask networks for drug discovery

    Bharath Ramsundar et al. “Massively multitask networks for drug discovery”. In: arXiv preprint arXiv:1502.02072 (2015)

    2015

  120. [128]

    Delusion, survival, and intelligent agents

    Mark Ring and Laurent Orseau. “Delusion, survival, and intelligent agents”. In: Artificial General Intelligence. Springer, 2011, pp. 11–20

    2011

  121. [129]

    A reduction of imitation learning and structured prediction to no-regret online learning

    St´ ephane Ross, Geoffrey J Gordon, and J Andrew Bagnell. “A reduction of imitation learning and structured prediction to no-regret online learning”. In: arXiv preprint arXiv:1011.0686 (2010)

    2010

  122. [130]

    Estimating causal effects of treatments in randomized and nonrandomized studies

    Donald B Rubin. “Estimating causal effects of treatments in randomized and nonrandomized studies.” In: Journal of educational Psychology 66.5 (1974), p. 688

    1974

  123. [131]

    Research priorities for robust and beneficial artificial intelligence

    Stuart Russell et al. “Research priorities for robust and beneficial artificial intelligence”. In: Future of Life Institute (2015)

    2015

  124. [132]

    Empowerment–an introduction

    Christoph Salge, Cornelius Glackin, and Daniel Polani. “Empowerment–an introduction”. In: Guided Self-Organization: Inception. Springer, 2014, pp. 67–114

    2014

  125. [133]

    The estimation of relationships with autocorrelated residuals by the use of instrumental variables

    J Denis Sargan. “The estimation of relationships with autocorrelated residuals by the use of instrumental variables”. In:Journal of the Royal Statistical Society. Series B (Methodological) (1959), pp. 91–105

    1959

  126. [134]

    The estimation of economic relationships using instrumental variables

    John D Sargan. “The estimation of economic relationships using instrumental variables”. In: Econometrica: Journal of the Econometric Society (1958), pp. 393–415. 27

    1958

  127. [135]

    High-dimensional continuous control using generalized advantage es- timation

    John Schulman et al. “High-dimensional continuous control using generalized advantage es- timation”. In: arXiv preprint arXiv:1506.02438 (2015)

    2015 arXiv

  128. [136]

    Machine Learning: The High-Interest Credit Card of Technical Debt

    D Sculley et al. “Machine Learning: The High-Interest Credit Card of Technical Debt”. In: (2014)

    2014

  129. [137]

    A tutorial on conformal prediction

    Glenn Shafer and Vladimir Vovk. “A tutorial on conformal prediction”. In: The Journal of Machine Learning Research 9 (2008), pp. 371–421

    2008

  130. [138]

    Bounding and Minimizing Counterfactual Error

    Uri Shalit, Fredrik Johansson, and David Sontag. “Bounding and Minimizing Counterfactual Error”. In: arXiv preprint arXiv:1606.03976 (2016)

    2016

  131. [139]

    Improving predictive inference under covariate shift by weighting the log-likelihood function

    Hidetoshi Shimodaira. “Improving predictive inference under covariate shift by weighting the log-likelihood function”. In: Journal of statistical planning and inference 90.2 (2000), pp. 227– 244

    2000

  132. [140]

    Incremental knowledge base construction using deepdive

    Jaeho Shin et al. “Incremental knowledge base construction using deepdive”. In: Proceedings of the VLDB Endowment 8.11 (2015), pp. 1310–1321

    2015

  133. [141]

    Mastering the game of Go with deep neural networks and tree search

    David Silver et al. “Mastering the game of Go with deep neural networks and tree search”. In: Nature 529.7587 (2016), pp. 484–489

    2016

  134. [142]

    arbitrary code execution

    SNES Super Mario World (USA) “arbitrary code execution”. Tool-assisted movies. 2014. url: http://tasvideos.org/2513M.html

    2014

  135. [143]

    Toward idealized decision theory

    Nate Soares and Benja Fallenstein. “Toward idealized decision theory”. In: arXiv preprint arXiv:1507.01986 (2015)

    2015

  136. [144]

    A formal theory of inductive inference. Part I

    Ray J Solomonoff. “A formal theory of inductive inference. Part I”. In: Information and control 7.1 (1964), pp. 1–22

    1964

  137. [145]

    A formal theory of inductive inference. Part II

    Ray J Solomonoff. “A formal theory of inductive inference. Part II”. In: Information and control 7.2 (1964), pp. 224–254

    1964

  138. [146]

    EL Lehmann, JP Romano: Testing statistical hypotheses

    J Steinebach. “EL Lehmann, JP Romano: Testing statistical hypotheses”. In: Metrika 64.2 (2006), pp. 255–256

    2006

  139. [147]

    Long-Term and Short-Term Challenges to Ensuring the Safety of AI Sys- tems

    Jacob Steinhardt. Long-Term and Short-Term Challenges to Ensuring the Safety of AI Sys- tems. [Online; accessed 13-June-2016]. 2015. url: https://jsteinhardt.wordpress.com/ 2015/06/24/long- term- and- short- term- challenges- to- ensuring- the- safety- of- ai-systems/

    2016

  140. [148]

    Unsupervised Risk Estimation with only Structural Assumptions

    Jacob Steinhardt and Percy Liang. “Unsupervised Risk Estimation with only Structural Assumptions”. In: (2016)

    2016

  141. [149]

    Finite-time regional verification of stochastic non-linear systems

    Jacob Steinhardt and Russ Tedrake. “Finite-time regional verification of stochastic non-linear systems”. In: The International Journal of Robotics Research 31.7 (2012), pp. 901–923

    2012

  142. [150]

    Avoiding Imposters and Delin- quents: Adversarial Crowdsourcing and Peer Prediction

    Jacob Steinhardt, Gregory Valiant, and Moses Charikar. “Avoiding Imposters and Delin- quents: Adversarial Crowdsourcing and Peer Prediction”. In:arxiv prepring arXiv:1606.05374 (2016). url: http://arxiv.org/abs/1606.05374

    2016

  143. [151]

    Reinforcement learning: An introduction

    Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 1998

    1998

  144. [152]

    Counterfactual risk minimization: Learning from logged bandit feedback

    Adith Swaminathan and Thorsten Joachims. “Counterfactual risk minimization: Learning from logged bandit feedback”. In: arXiv preprint arXiv:1502.02362 (2015)

    2015

  145. [153]

    Intriguing properties of neural networks

    Christian Szegedy et al. “Intriguing properties of neural networks”. In: arXiv preprint arXiv:1312.6199 (2013)

    2013

  146. [154]

    Policy gradients beyond expectations: Conditional value-at-risk

    Aviv Tamar, Yonatan Glassner, and Shie Mannor. “Policy gradients beyond expectations: Conditional value-at-risk”. In: arXiv preprint arXiv:1404.3862 (2014)

    2014

  147. [155]

    Quantilizers: A Safer Alternative to Maximizers for Limited Optimization

    Jessica Taylor. “Quantilizers: A Safer Alternative to Maximizers for Limited Optimization”. In: forthcoming). Submitted to AAAI (2016)

    2016

  148. [156]

    Transfer learning for reinforcement learning domains: A survey

    Matthew E Taylor and Peter Stone. “Transfer learning for reinforcement learning domains: A survey”. In: Journal of Machine Learning Research 10.Jul (2009), pp. 1633–1685

    2009

  149. [157]

    High-Confidence Off-Policy Evaluation

    Philip S Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. “High-Confidence Off-Policy Evaluation.” In: AAAI. 2015, pp. 3000–3006

    2015

  150. [158]

    Artificial evolution in the physical world

    Adrian Thompson. Artificial evolution in the physical world . 1997. 28

    1997

  151. [159]

    Unbiased look at dataset bias

    Antonio Torralba and Alexei A Efros. “Unbiased look at dataset bias”. In: Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on . IEEE. 2011, pp. 1521–1528

    2011

  152. [160]

    Safe Exploration in Finite Markov Decision Processes with Gaussian Processes

    Matteo Turchetta, Felix Berkenkamp, and Andreas Krause. “Safe Exploration in Finite Markov Decision Processes with Gaussian Processes”. In: arXiv preprint arXiv:1606.04753 (2016)

    2016

  153. [161]

    Estimation and Inference of Heterogeneous Treatment Ef- fects using Random Forests

    Stefan Wager and Susan Athey. “Estimation and Inference of Heterogeneous Treatment Ef- fects using Random Forests”. In: arXiv preprint arXiv:1510.04342 (2015)

    2015

  154. [162]

    The first law of robotics (a call to arms)

    Daniel Weld and Oren Etzioni. “The first law of robotics (a call to arms)”. In: AAAI. Vol. 94

  155. [163]

    On-the-job learning with bayesian decision theory

    Keenon Werling et al. “On-the-job learning with bayesian decision theory”. In: Advances in Neural Information Processing Systems. 2015, pp. 3447–3455

    2015

  156. [164]

    Towards ai-complete question answering: A set of prerequisite toy tasks

    Jason Weston et al. “Towards ai-complete question answering: A set of prerequisite toy tasks”. In: arXiv preprint arXiv:1502.05698 (2015)

    2015

  157. [165]

    Robust Markov decision processes

    Wolfram Wiesemann, Daniel Kuhn, and Ber¸ c Rustem. “Robust Markov decision processes”. In: Mathematics of Operations Research 38.1 (2013), pp. 153–183

    2013

  158. [166]

    Utility function security in artificially intelligent agents

    Roman V Yampolskiy. “Utility function security in artificially intelligent agents”. In: Journal of Experimental & Theoretical Artificial Intelligence 26.3 (2014), pp. 373–389

    2014

  159. [167]

    Understanding neural networks through deep visualization

    Jason Yosinski et al. “Understanding neural networks through deep visualization”. In: arXiv preprint arXiv:1506.06579 (2015)

    2015

  160. [168]

    Artificial intelligence as a positive and negative factor in global risk

    Eliezer Yudkowsky. “Artificial intelligence as a positive and negative factor in global risk”. In: Global catastrophic risks 1 (2008), p. 303

    2008

  161. [169]

    Learning Fair Classifiers

    Muhammad Bilal Zafar et al. “Learning Fair Classifiers”. In: stat 1050 (2015), p. 29

    2015

  162. [170]

    Learning Fair Representations

    Richard S Zemel et al. “Learning Fair Representations.” In: ICML (3) 28 (2013), pp. 325–333

    2013

  163. [171]

    Spectral methods meet EM: A provably optimal algorithm for crowd- sourcing

    Yuchen Zhang et al. “Spectral methods meet EM: A provably optimal algorithm for crowd- sourcing”. In: Advances in neural information processing systems . 2014, pp. 1260–1268. 29

    2014

Pith tools

Reviewed May 11, 2026 · model on record in the stance chip above.