{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2019:R5VTAKTWDE5JPZRHHJII3RAWIJ","short_pith_number":"pith:R5VTAKTW","schema_version":"1.0","canonical_sha256":"8f6b302a76193a97e6273a508dc416426df405243f227763f9e7b7d97e8765c4","source":{"kind":"arxiv","id":"1906.01820","version":3},"attestation_state":"computed","paper":{"title":"Risks from Learned Optimization in Advanced Machine Learning Systems","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Learned models in machine learning can themselves become optimizers whose objectives diverge from the training loss.","cross_cats":[],"primary_cat":"cs.AI","authors_text":"Chris van Merwijk, Evan Hubinger, Joar Skalse, Scott Garrabrant, Vladimir Mikulik","submitted_at":"2019-06-05T04:43:25Z","abstract_excerpt":"We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer - a situation we refer to as mesa-optimization, a neologism we introduce in this paper. We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be - how will it differ from the loss function it was train"},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"1906.01820","kind":"arxiv","version":3},"metadata":{"license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","primary_cat":"cs.AI","submitted_at":"2019-06-05T04:43:25Z","cross_cats_sorted":[],"title_canon_sha256":"bef97e85af23a1b58be90a7b9e8ecf0a42d495c76af7ada23acf73d323ce9916","abstract_canon_sha256":"fe532e59f6ef9e5e2eb21aa8854b30c924bb21870540f1a3482ecaa5b9e719e9"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:52.564178Z","signature_b64":"F2x/kVMSveT0gyZ/wTRDn/3RHHBwy53dwqkGJLOWuLqfDphbH/rZ3zOtSaM94KumoBDYx2NNMHE6lcMepGzuBw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"8f6b302a76193a97e6273a508dc416426df405243f227763f9e7b7d97e8765c4","last_reissued_at":"2026-05-17T23:38:52.563737Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:52.563737Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Risks from Learned Optimization in Advanced Machine Learning Systems","license":"http://arxiv.org/licenses/nonexclusive-distrib/1.0/","headline":"Learned models in machine learning can themselves become optimizers whose objectives diverge from the training loss.","cross_cats":[],"primary_cat":"cs.AI","authors_text":"Chris van Merwijk, Evan Hubinger, Joar Skalse, Scott Garrabrant, Vladimir Mikulik","submitted_at":"2019-06-05T04:43:25Z","abstract_excerpt":"We analyze the type of learned optimization that occurs when a learned model (such as a neural network) is itself an optimizer - a situation we refer to as mesa-optimization, a neologism we introduce in this paper. We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems. First, under what circumstances will learned models be optimizers, including when they should not be? Second, when a learned model is an optimizer, what will its objective be - how will it differ from the loss function it was train"},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems: under what circumstances will learned models be optimizers, and when a learned model is an optimizer, what will its objective be and how can it be aligned?","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"The analysis assumes that sufficiently capable learned models will contain internal optimization processes whose objectives can be analyzed separately from the outer training loss, without providing formal conditions or empirical thresholds for when this separation becomes load-bearing.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Mesa-optimization arises when learned models act as optimizers with objectives that can differ from their training loss, creating alignment risks in advanced machine learning.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Learned models in machine learning can themselves become optimizers whose objectives diverge from the training loss.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"2720bafcf2225627f4fe10aa4c8bc8687bc38d42ad942670459f997db0b67c42"},"source":{"id":"1906.01820","kind":"arxiv","version":3},"verdict":{"id":"ad4daa52-aae6-44a7-89dd-aabb3516faa4","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-15T12:17:39.308599Z","strongest_claim":"We believe that the possibility of mesa-optimization raises two important questions for the safety and transparency of advanced machine learning systems: under what circumstances will learned models be optimizers, and when a learned model is an optimizer, what will its objective be and how can it be aligned?","one_line_summary":"Mesa-optimization arises when learned models act as optimizers with objectives that can differ from their training loss, creating alignment risks in advanced machine learning.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"The analysis assumes that sufficiently capable learned models will contain internal optimization processes whose objectives can be analyzed separately from the outer training loss, without providing formal conditions or empirical thresholds for when this separation becomes load-bearing.","pith_extraction_headline":"Learned models in machine learning can themselves become optimizers whose objectives diverge from the training loss."},"references":{"count":40,"sample":[{"doi":"","year":2018,"title":"Bottle caps aren’t optimisers, 2018","work_id":"8ecf2977-b6f2-4d72-bc30-102fd9f0f239","ref_index":1,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2018,"title":"TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning","work_id":"54bcc5b2-9cd5-454d-b73d-e4e24db90330","ref_index":2,"cited_arxiv_id":"1710.11417","is_internal_anchor":true},{"doi":"","year":2018,"title":"Universal Planning Networks","work_id":"12d1df6e-e02d-4bc2-9d7e-1adfc87fa16c","ref_index":3,"cited_arxiv_id":"1804.00645","is_internal_anchor":true},{"doi":"","year":2016,"title":"2016 , month = nov, journal =","work_id":"9919eb6a-fde4-4d39-8047-21197365166d","ref_index":4,"cited_arxiv_id":"1606.04474","is_internal_anchor":true},{"doi":"","year":2016,"title":"Bartlett, Ilya Sutskever, and Pieter Abbeel","work_id":"f9b75698-414f-456e-a2e6-dbe568b2693d","ref_index":5,"cited_arxiv_id":"1611.02779","is_internal_anchor":true}],"resolved_work":40,"snapshot_sha256":"b7f5e0419c68a6a7655e64aa0f7f2583f0a3fabadfcde1ed8dc17422c6e4d104","internal_anchors":18},"formal_canon":{"evidence_count":1,"snapshot_sha256":"3187f144d2a547554246567d06b0e45853d60b4f975f40671375d8b31182bf69"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"1906.01820","created_at":"2026-05-17T23:38:52.563807+00:00"},{"alias_kind":"arxiv_version","alias_value":"1906.01820v3","created_at":"2026-05-17T23:38:52.563807+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.1906.01820","created_at":"2026-05-17T23:38:52.563807+00:00"},{"alias_kind":"pith_short_12","alias_value":"R5VTAKTWDE5J","created_at":"2026-05-18T12:33:27.125529+00:00"},{"alias_kind":"pith_short_16","alias_value":"R5VTAKTWDE5JPZRH","created_at":"2026-05-18T12:33:27.125529+00:00"},{"alias_kind":"pith_short_8","alias_value":"R5VTAKTW","created_at":"2026-05-18T12:33:27.125529+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":35,"internal_anchor_count":35,"sample":[{"citing_arxiv_id":"2605.22643","citing_title":"Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety","ref_index":43,"is_internal_anchor":true},{"citing_arxiv_id":"2605.23565","citing_title":"Understanding Goal Generalisation in Sequential Reinforcement Learning","ref_index":29,"is_internal_anchor":true},{"citing_arxiv_id":"2204.06745","citing_title":"GPT-NeoX-20B: An Open-Source Autoregressive Language Model","ref_index":40,"is_internal_anchor":true},{"citing_arxiv_id":"2605.22643","citing_title":"Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety","ref_index":43,"is_internal_anchor":true},{"citing_arxiv_id":"2506.18852","citing_title":"Mechanistic Interpretability Needs Philosophy","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2201.03544","citing_title":"The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16282","citing_title":"Taxonomy and Consistency Analysis of Safety Benchmarks for AI Agents","ref_index":21,"is_internal_anchor":true},{"citing_arxiv_id":"2605.15377","citing_title":"Ensemble Monitoring for AI Control: Diverse Signals Outweigh More Compute","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2605.16035","citing_title":"Who Owns This Agent? Tracing AI Agents Back to Their Owners","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2210.10760","citing_title":"Scaling Laws for Reward Model Overoptimization","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2211.03540","citing_title":"Measuring Progress on Scalable Oversight for Large Language Models","ref_index":45,"is_internal_anchor":true},{"citing_arxiv_id":"2406.10162","citing_title":"Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models","ref_index":14,"is_internal_anchor":true},{"citing_arxiv_id":"2406.10162","citing_title":"Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models","ref_index":182,"is_internal_anchor":true},{"citing_arxiv_id":"2412.04984","citing_title":"Frontier Models are Capable of In-context Scheming","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2603.00678","citing_title":"From Syntax to Semantics: Geometric Stability as the Missing Axis of Perturbation Biology","ref_index":27,"is_internal_anchor":true},{"citing_arxiv_id":"2605.10310","citing_title":"Positive Alignment: Artificial Intelligence for Human Flourishing","ref_index":87,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12809","citing_title":"Correcting Influence: Unboxing LLM Outputs with Orthogonal Latent Spaces","ref_index":217,"is_internal_anchor":true},{"citing_arxiv_id":"2604.01346","citing_title":"Safety, Security, and Cognitive Risks in World Models","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2604.02720","citing_title":"Cognitive Comparability and the Limits of Governance: Evaluating Authority Under Radical Capability Asymmetry","ref_index":53,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11712","citing_title":"Toward Stable Value Alignment: Introducing Independent Modules for Consistent Value Guidance","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2605.11134","citing_title":"Spurious Correlation Learning in Preference Optimization: Mechanisms, Consequences, and Mitigation via Tie Training","ref_index":38,"is_internal_anchor":true},{"citing_arxiv_id":"2605.10310","citing_title":"Positive Alignment: Artificial Intelligence for Human Flourishing","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2605.09773","citing_title":"Exploitation Without Deception: Dark Triad Feature Steering Reveals Separable Antisocial Circuits in Language Models","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2604.24966","citing_title":"Risk Reporting for Developers' Internal AI Model Use","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2604.23338","citing_title":"A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework","ref_index":41,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":1,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ","json":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ.json","graph_json":"https://pith.science/api/pith-number/R5VTAKTWDE5JPZRHHJII3RAWIJ/graph.json","events_json":"https://pith.science/api/pith-number/R5VTAKTWDE5JPZRHHJII3RAWIJ/events.json","paper":"https://pith.science/paper/R5VTAKTW"},"agent_actions":{"view_html":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ","download_json":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ.json","view_paper":"https://pith.science/paper/R5VTAKTW","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=1906.01820&json=true","fetch_graph":"https://pith.science/api/pith-number/R5VTAKTWDE5JPZRHHJII3RAWIJ/graph.json","fetch_events":"https://pith.science/api/pith-number/R5VTAKTWDE5JPZRHHJII3RAWIJ/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ/action/timestamp_anchor","attest_storage":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ/action/storage_attestation","attest_author":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ/action/author_attestation","sign_citation":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ/action/citation_signature","submit_replication":"https://pith.science/pith/R5VTAKTWDE5JPZRHHJII3RAWIJ/action/replication_record"}},"created_at":"2026-05-17T23:38:52.563807+00:00","updated_at":"2026-05-17T23:38:52.563807+00:00"}