{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2022:MTHKQKRGMSXMB7W4XJJO5C66RV","short_pith_number":"pith:MTHKQKRG","schema_version":"1.0","canonical_sha256":"64cea82a2664aec0fedcba52ee8bde8d4838e779018cbd75f2e3cf54ad55cfff","source":{"kind":"arxiv","id":"2209.14375","version":1},"attestation_state":"computed","paper":{"title":"Improving alignment of dialogue agents via targeted human judgements","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Sparrow dialogue agent uses separate human judgments on natural language rules and evidence citations to outperform baselines in preference and safety.","cross_cats":["cs.CL"],"primary_cat":"cs.LG","authors_text":"Abigail See, Amelia Glaese, Boxi Wu, Charlie Chen, Demis Hassabis, Doug Fritz, Fan Yang, Geoffrey Irving, Iason Gabriel, Jaume Sanchez Elias, John Aslanides, John Mellor, Jonathan Uesato, Koray Kavukcuoglu, Laura Weidinger, Lisa Anne Hendricks, Lucy Campbell-Gillingham, Maja Tr\\k{e}bacz, Maribeth Rauh, Martin Chadwick, Nat McAleese, Nicholas Fernando, Phoebe Thacker, Po-Sen Huang, Rachel Foley, Ramona Comanescu, Richard Green, Rory Greig, So\\v{n}a Mokr\\'a, Sumanth Dathathri, Susannah Young, Timo Ewalds, Vlad Firoiu, William Isaac","submitted_at":"2022-09-28T19:04:43Z","abstract_excerpt":"We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":true,"formal_links_present":true},"canonical_record":{"source":{"id":"2209.14375","kind":"arxiv","version":1},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.LG","submitted_at":"2022-09-28T19:04:43Z","cross_cats_sorted":["cs.CL"],"title_canon_sha256":"d2be7c9a82e3da3903cadd379c977e7e799ce3051361fe1ce7286e483432fd5b","abstract_canon_sha256":"04e14b3529d962d858e47fc8446a8e656662bf3ecfafa19e8b3eedf48831a137"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:39:22.268218Z","signature_b64":"qY7CNKMNqwegiHKd2pyL2+iRfQHHJON9/E+Ty7kWoJNinnXM3kknmz2oAmmU+iptwBGMTvU1udNWWiirf1pWBw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"64cea82a2664aec0fedcba52ee8bde8d4838e779018cbd75f2e3cf54ad55cfff","last_reissued_at":"2026-05-17T23:39:22.267587Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:39:22.267587Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"Improving alignment of dialogue agents via targeted human judgements","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Sparrow dialogue agent uses separate human judgments on natural language rules and evidence citations to outperform baselines in preference and safety.","cross_cats":["cs.CL"],"primary_cat":"cs.LG","authors_text":"Abigail See, Amelia Glaese, Boxi Wu, Charlie Chen, Demis Hassabis, Doug Fritz, Fan Yang, Geoffrey Irving, Iason Gabriel, Jaume Sanchez Elias, John Aslanides, John Mellor, Jonathan Uesato, Koray Kavukcuoglu, Laura Weidinger, Lisa Anne Hendricks, Lucy Campbell-Gillingham, Maja Tr\\k{e}bacz, Maribeth Rauh, Martin Chadwick, Nat McAleese, Nicholas Fernando, Phoebe Thacker, Po-Sen Huang, Rachel Foley, Ramona Comanescu, Richard Green, Rory Greig, So\\v{n}a Mokr\\'a, Sumanth Dathathri, Susannah Young, Timo Ewalds, Vlad Firoiu, William Isaac","submitted_at":"2022-09-28T19:04:43Z","abstract_excerpt":"We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and "},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That separate human judgments on the listed natural language rules reliably capture the intended notions of helpfulness, correctness, and harmlessness without introducing new biases or inconsistencies.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"Sparrow uses targeted rule-based human feedback and evidence provision to outperform baselines in preference while violating rules only 8% of the time under adversarial probing.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Sparrow dialogue agent uses separate human judgments on natural language rules and evidence citations to outperform baselines in preference and safety.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"2c106e196e67f56a7b6ad8cbd6cf73426498fd7e2a1e036ac188667f13200a78"},"source":{"id":"2209.14375","kind":"arxiv","version":1},"verdict":{"id":"aec526af-32a9-4a24-81fe-4fe676599e6d","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-14T17:47:41.479414Z","strongest_claim":"Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed.","one_line_summary":"Sparrow uses targeted rule-based human feedback and evidence provision to outperform baselines in preference while violating rules only 8% of the time under adversarial probing.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That separate human judgments on the listed natural language rules reliably capture the intended notions of helpfulness, correctness, and harmlessness without introducing new biases or inconsistencies.","pith_extraction_headline":"Sparrow dialogue agent uses separate human judgments on natural language rules and evidence citations to outperform baselines in preference and safety."},"references":{"count":15,"sample":[{"doi":"10.18653/v1/d19-1176","year":1901,"title":"Supervising strong learners by amplifying weak experts","work_id":"5d6c7456-8e86-44d2-b157-ac0daeb9587c","ref_index":1,"cited_arxiv_id":"1810.08575","is_internal_anchor":true},{"doi":"10.18653/v1/2021.emnlp-main.444","year":2021,"title":"doi: 10.18653/v1/2021.emnlp-main.444","work_id":"842947b7-ec9f-47be-8ca1-1478045daf1c","ref_index":2,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"","year":2022,"title":"Adam: A Method for Stochastic Optimization","work_id":"1910796d-9b52-4683-bf5c-de9632c1028b","ref_index":3,"cited_arxiv_id":"1412.6980","is_internal_anchor":true},{"doi":"10.1145/3359283","year":2019,"title":"URL https://arxiv.org/abs/2203.05115. M. K. Lee, D. Kusbit, A. Kahng, J. T. Kim, X. Yuan, A. Chan, D. See, R. Noothigattu, S. Lee, A. Psomas, and A. D. Procaccia. WeBuildAI: Participatory framework fo","work_id":"3db07906-b7ec-4a98-b4f0-6363b53bfe8f","ref_index":4,"cited_arxiv_id":"","is_internal_anchor":false},{"doi":"10.18653/v1/d19-1244","year":2021,"title":"WebGPT: Browser-assisted question-answering with human feedback","work_id":"e25ef3e1-4848-4cb9-bf28-67a420591165","ref_index":5,"cited_arxiv_id":"2112.09332","is_internal_anchor":true}],"resolved_work":15,"snapshot_sha256":"413187bbeb966182c8b09d3e092f66c9d365b41dc526fa600c791c5492071fb3","internal_anchors":4},"formal_canon":{"evidence_count":2,"snapshot_sha256":"a6b4f56d496566685d14fb2e7609a9b1fe71d11be320a6b8577de2d6dd51c092"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2209.14375","created_at":"2026-05-17T23:39:22.267680+00:00"},{"alias_kind":"arxiv_version","alias_value":"2209.14375v1","created_at":"2026-05-17T23:39:22.267680+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2209.14375","created_at":"2026-05-17T23:39:22.267680+00:00"},{"alias_kind":"pith_short_12","alias_value":"MTHKQKRGMSXM","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_16","alias_value":"MTHKQKRGMSXMB7W4","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_8","alias_value":"MTHKQKRG","created_at":"2026-05-18T12:33:33.725879+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":41,"internal_anchor_count":41,"sample":[{"citing_arxiv_id":"2605.23398","citing_title":"TPMM-DPO: Trajectory-aware Preference-guided Model Merging for Iterative Direct Preference Optimization","ref_index":27,"is_internal_anchor":true},{"citing_arxiv_id":"2301.13688","citing_title":"The Flan Collection: Designing Data and Methods for Effective Instruction Tuning","ref_index":16,"is_internal_anchor":true},{"citing_arxiv_id":"2307.15043","citing_title":"Universal and Transferable Adversarial Attacks on Aligned Language Models","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2312.11805","citing_title":"Gemini: A Family of Highly Capable Multimodal Models","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2504.12501","citing_title":"Reinforcement Learning from Human Feedback","ref_index":42,"is_internal_anchor":true},{"citing_arxiv_id":"2307.06435","citing_title":"A Comprehensive Overview of Large Language Models","ref_index":167,"is_internal_anchor":true},{"citing_arxiv_id":"2506.01937","citing_title":"RewardBench 2: Advancing Reward Model Evaluation","ref_index":6,"is_internal_anchor":true},{"citing_arxiv_id":"2408.00724","citing_title":"Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models","ref_index":129,"is_internal_anchor":true},{"citing_arxiv_id":"2304.06767","citing_title":"RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment","ref_index":119,"is_internal_anchor":true},{"citing_arxiv_id":"2402.13116","citing_title":"A Survey on Knowledge Distillation of Large Language Models","ref_index":120,"is_internal_anchor":true},{"citing_arxiv_id":"2303.17491","citing_title":"Language Models can Solve Computer Tasks","ref_index":22,"is_internal_anchor":true},{"citing_arxiv_id":"2409.12917","citing_title":"Training Language Models to Self-Correct via Reinforcement Learning","ref_index":82,"is_internal_anchor":true},{"citing_arxiv_id":"2310.06987","citing_title":"Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation","ref_index":9,"is_internal_anchor":true},{"citing_arxiv_id":"2308.03958","citing_title":"Simple synthetic data reduces sycophancy in large language models","ref_index":13,"is_internal_anchor":true},{"citing_arxiv_id":"2402.05070","citing_title":"A Roadmap to Pluralistic Alignment","ref_index":215,"is_internal_anchor":true},{"citing_arxiv_id":"2309.17400","citing_title":"Directly Fine-Tuning Diffusion Models on Differentiable Rewards","ref_index":8,"is_internal_anchor":true},{"citing_arxiv_id":"2309.00267","citing_title":"RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback","ref_index":79,"is_internal_anchor":true},{"citing_arxiv_id":"2308.01263","citing_title":"XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models","ref_index":2,"is_internal_anchor":true},{"citing_arxiv_id":"2605.12288","citing_title":"TokenRatio: Principled Token-Level Preference Optimization via Ratio Matching","ref_index":104,"is_internal_anchor":true},{"citing_arxiv_id":"2603.28281","citing_title":"Corruption-robust Offline Multi-agent Reinforcement Learning From Human Feedback","ref_index":4,"is_internal_anchor":true},{"citing_arxiv_id":"2310.03684","citing_title":"SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2303.17760","citing_title":"CAMEL: Communicative Agents for \"Mind\" Exploration of Large Language Model Society","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2309.00614","citing_title":"Baseline Defenses for Adversarial Attacks Against Aligned Language Models","ref_index":17,"is_internal_anchor":true},{"citing_arxiv_id":"2303.17564","citing_title":"BloombergGPT: A Large Language Model for Finance","ref_index":36,"is_internal_anchor":true},{"citing_arxiv_id":"2308.08998","citing_title":"Reinforced Self-Training (ReST) for Language Modeling","ref_index":11,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV","json":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV.json","graph_json":"https://pith.science/api/pith-number/MTHKQKRGMSXMB7W4XJJO5C66RV/graph.json","events_json":"https://pith.science/api/pith-number/MTHKQKRGMSXMB7W4XJJO5C66RV/events.json","paper":"https://pith.science/paper/MTHKQKRG"},"agent_actions":{"view_html":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV","download_json":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV.json","view_paper":"https://pith.science/paper/MTHKQKRG","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2209.14375&json=true","fetch_graph":"https://pith.science/api/pith-number/MTHKQKRGMSXMB7W4XJJO5C66RV/graph.json","fetch_events":"https://pith.science/api/pith-number/MTHKQKRGMSXMB7W4XJJO5C66RV/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV/action/timestamp_anchor","attest_storage":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV/action/storage_attestation","attest_author":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV/action/author_attestation","sign_citation":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV/action/citation_signature","submit_replication":"https://pith.science/pith/MTHKQKRGMSXMB7W4XJJO5C66RV/action/replication_record"}},"created_at":"2026-05-17T23:39:22.267680+00:00","updated_at":"2026-05-17T23:39:22.267680+00:00"}