{"record_type":"pith_number_record","schema_url":"https://pith.science/schemas/pith-number/v1.json","pith_number":"pith:2022:APCCNCY42HHIJXSKVRWKP2A7K3","short_pith_number":"pith:APCCNCY4","schema_version":"1.0","canonical_sha256":"03c4268b1cd1ce84de4aac6ca7e81f56d23f69fada537f6a0c40966b36b8a18f","source":{"kind":"arxiv","id":"2212.12017","version":3},"attestation_state":"computed","paper":{"title":"OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Instruction-tuning on a 2000-task benchmark produces models that generalize to held-out categories, tasks, and instances.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Asli Celikyilmaz, Brian O'Horo, Christopher Dewan, Daniel Simig, Gabriel Pereyra, Jeff Wang, Kurt Shuster, Luke Zettlemoyer, Ping Yu, Punit Singh Koura, Qing Liu, Ramakanth Pasunuru, Srinivasan Iyer, Tianlu Wang, Todor Mihaylov, Ves Stoyanov, Xian Li, Xi Victoria Lin","submitted_at":"2022-12-22T19:56:09Z","abstract_excerpt":"Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, "},"verification_status":{"content_addressed":true,"pith_receipt":true,"author_attested":false,"weak_author_claims":0,"strong_author_claims":0,"externally_anchored":false,"storage_verified":false,"citation_signatures":0,"replication_records":0,"graph_snapshot":true,"references_resolved":false,"formal_links_present":true},"canonical_record":{"source":{"id":"2212.12017","kind":"arxiv","version":3},"metadata":{"license":"http://creativecommons.org/licenses/by/4.0/","primary_cat":"cs.CL","submitted_at":"2022-12-22T19:56:09Z","cross_cats_sorted":[],"title_canon_sha256":"50f144c76cd8960a3a65c93eec8753ba5b9444987ea030fc5c004d82595e8a06","abstract_canon_sha256":"025164ad055811875b0f90849498f9de497302ffe14bb73802b0234deb1bbe90"},"schema_version":"1.0"},"receipt":{"kind":"pith_receipt","key_id":"pith-v1-2026-05","algorithm":"ed25519","signed_at":"2026-05-17T23:38:14.863845Z","signature_b64":"q88eL8fWs7g/vjoDqCzAFfMcEbFOCPpS+Q7/i6LNPXSAS2YWSI1JTrkbLv3D/w8XgEKc9cyzWYdDzqB2sASUDw==","signed_message":"canonical_sha256_bytes","builder_version":"pith-number-builder-2026-05-17-v1","receipt_version":"0.3","canonical_sha256":"03c4268b1cd1ce84de4aac6ca7e81f56d23f69fada537f6a0c40966b36b8a18f","last_reissued_at":"2026-05-17T23:38:14.863298Z","signature_status":"signed_v1","first_computed_at":"2026-05-17T23:38:14.863298Z","public_key_fingerprint":"8d4b5ee74e4693bcd1df2446408b0d54"},"graph_snapshot":{"paper":{"title":"OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization","license":"http://creativecommons.org/licenses/by/4.0/","headline":"Instruction-tuning on a 2000-task benchmark produces models that generalize to held-out categories, tasks, and instances.","cross_cats":[],"primary_cat":"cs.CL","authors_text":"Asli Celikyilmaz, Brian O'Horo, Christopher Dewan, Daniel Simig, Gabriel Pereyra, Jeff Wang, Kurt Shuster, Luke Zettlemoyer, Ping Yu, Punit Singh Koura, Qing Liu, Ramakanth Pasunuru, Srinivasan Iyer, Tianlu Wang, Todor Mihaylov, Ves Stoyanov, Xian Li, Xi Victoria Lin","submitted_at":"2022-12-22T19:56:09Z","abstract_excerpt":"Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, "},"claims":{"count":4,"items":[{"kind":"strongest_claim","text":"OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG.","source":"verdict.strongest_claim","status":"machine_extracted","claim_id":"C1","attestation":"unclaimed"},{"kind":"weakest_assumption","text":"That the consolidation of tasks from eight existing benchmarks into 2000 tasks and the defined held-out category/task/instance splits provide a representative and unbiased measure of generalization to truly unseen NLP problems.","source":"verdict.weakest_assumption","status":"machine_extracted","claim_id":"C2","attestation":"unclaimed"},{"kind":"one_line_summary","text":"OPT-IML 30B and 175B models, trained on a new 2000-task instruction benchmark, demonstrate generalization to held-out categories, tasks, and instances while outperforming base OPT and competing with benchmark-specific models.","source":"verdict.one_line_summary","status":"machine_extracted","claim_id":"C3","attestation":"unclaimed"},{"kind":"headline","text":"Instruction-tuning on a 2000-task benchmark produces models that generalize to held-out categories, tasks, and instances.","source":"verdict.pith_extraction.headline","status":"machine_extracted","claim_id":"C4","attestation":"unclaimed"}],"snapshot_sha256":"2695d9dc9fc29a142a4ce0c2d84f1b74dbad1ff924d64ab2ee33d1847edfb483"},"source":{"id":"2212.12017","kind":"arxiv","version":3},"verdict":{"id":"09c7bd3e-91cb-47fd-abf3-052f2bcd2891","model_set":{"reader":"grok-4.3"},"created_at":"2026-05-17T06:02:41.527056Z","strongest_claim":"OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG.","one_line_summary":"OPT-IML 30B and 175B models, trained on a new 2000-task instruction benchmark, demonstrate generalization to held-out categories, tasks, and instances while outperforming base OPT and competing with benchmark-specific models.","pipeline_version":"pith-pipeline@v0.9.0","weakest_assumption":"That the consolidation of tasks from eight existing benchmarks into 2000 tasks and the defined held-out category/task/instance splits provide a representative and unbiased measure of generalization to truly unseen NLP problems.","pith_extraction_headline":"Instruction-tuning on a 2000-task benchmark produces models that generalize to held-out categories, tasks, and instances."},"references":{"count":0,"sample":[],"resolved_work":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57","internal_anchors":0},"formal_canon":{"evidence_count":2,"snapshot_sha256":"bb8120a1cd8778e55f62cce5ef6748e50bb11ab03f5b17f1a6932fcb64af250c"},"author_claims":{"count":0,"strong_count":0,"snapshot_sha256":"258153158e38e3291e3d48162225fcdb2d5a3ed65a07baac614ab91432fd4f57"},"builder_version":"pith-number-builder-2026-05-17-v1"},"aliases":[{"alias_kind":"arxiv","alias_value":"2212.12017","created_at":"2026-05-17T23:38:14.863396+00:00"},{"alias_kind":"arxiv_version","alias_value":"2212.12017v3","created_at":"2026-05-17T23:38:14.863396+00:00"},{"alias_kind":"doi","alias_value":"10.48550/arxiv.2212.12017","created_at":"2026-05-17T23:38:14.863396+00:00"},{"alias_kind":"pith_short_12","alias_value":"APCCNCY42HHI","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_16","alias_value":"APCCNCY42HHIJXSK","created_at":"2026-05-18T12:33:33.725879+00:00"},{"alias_kind":"pith_short_8","alias_value":"APCCNCY4","created_at":"2026-05-18T12:33:33.725879+00:00"}],"events":[],"event_summary":{},"paper_claims":[],"inbound_citations":{"count":27,"internal_anchor_count":27,"sample":[{"citing_arxiv_id":"2301.13688","citing_title":"The Flan Collection: Designing Data and Methods for Effective Instruction Tuning","ref_index":24,"is_internal_anchor":true},{"citing_arxiv_id":"2410.06431","citing_title":"Functional-level Uncertainty Quantification for Calibrated Fine-tuning on LLMs","ref_index":12,"is_internal_anchor":true},{"citing_arxiv_id":"2507.05660","citing_title":"Optimus: A Robust Defense Framework for Mitigating Toxicity while Fine-Tuning Conversational AI","ref_index":34,"is_internal_anchor":true},{"citing_arxiv_id":"2605.17152","citing_title":"Multilingual and Multimodal LLMs in the Wild: Building for Low-Resource Languages","ref_index":129,"is_internal_anchor":true},{"citing_arxiv_id":"2307.06435","citing_title":"A Comprehensive Overview of Large Language Models","ref_index":97,"is_internal_anchor":true},{"citing_arxiv_id":"2505.20638","citing_title":"Music Audio-Visual Question Answering Requires Specialized Multimodal Designs","ref_index":7,"is_internal_anchor":true},{"citing_arxiv_id":"2507.11687","citing_title":"MetaLint: Easy-to-Hard Generalization for Code Linting","ref_index":18,"is_internal_anchor":true},{"citing_arxiv_id":"2303.09540","citing_title":"SemDeDup: Data-efficient learning at web-scale through semantic deduplication","ref_index":44,"is_internal_anchor":true},{"citing_arxiv_id":"2512.09292","citing_title":"Identifying Bias in Machine-generated Text Detection","ref_index":5,"is_internal_anchor":true},{"citing_arxiv_id":"2309.08532","citing_title":"EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers","ref_index":145,"is_internal_anchor":true},{"citing_arxiv_id":"2309.08532","citing_title":"EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizers","ref_index":96,"is_internal_anchor":true},{"citing_arxiv_id":"2306.13549","citing_title":"A Survey on Multimodal Large Language Models","ref_index":98,"is_internal_anchor":true},{"citing_arxiv_id":"2605.13369","citing_title":"Query-Conditioned Test-Time Self-Training for Large Language Models","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2605.13369","citing_title":"Query-Conditioned Test-Time Self-Training for Large Language Models","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2303.17760","citing_title":"CAMEL: Communicative Agents for \"Mind\" Exploration of Large Language Model Society","ref_index":53,"is_internal_anchor":true},{"citing_arxiv_id":"2303.17580","citing_title":"HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face","ref_index":11,"is_internal_anchor":true},{"citing_arxiv_id":"2406.00515","citing_title":"A Survey on Large Language Models for Code Generation","ref_index":116,"is_internal_anchor":true},{"citing_arxiv_id":"2401.10020","citing_title":"Self-Rewarding Language Models","ref_index":25,"is_internal_anchor":true},{"citing_arxiv_id":"2311.05232","citing_title":"A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions","ref_index":134,"is_internal_anchor":true},{"citing_arxiv_id":"2510.18234","citing_title":"DeepSeek-OCR: Contexts Optical Compression","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2305.15334","citing_title":"Gorilla: Large Language Model Connected with Massive APIs","ref_index":15,"is_internal_anchor":true},{"citing_arxiv_id":"2402.06196","citing_title":"Large Language Models: A Survey","ref_index":103,"is_internal_anchor":true},{"citing_arxiv_id":"2305.14314","citing_title":"QLoRA: Efficient Finetuning of Quantized LLMs","ref_index":29,"is_internal_anchor":true},{"citing_arxiv_id":"2309.07864","citing_title":"The Rise and Potential of Large Language Model Based Agents: A Survey","ref_index":266,"is_internal_anchor":true},{"citing_arxiv_id":"2304.08485","citing_title":"Visual Instruction Tuning","ref_index":22,"is_internal_anchor":true}]},"formal_canon":{"evidence_count":2,"sample":[],"anchors":[]},"links":{"html":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3","json":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3.json","graph_json":"https://pith.science/api/pith-number/APCCNCY42HHIJXSKVRWKP2A7K3/graph.json","events_json":"https://pith.science/api/pith-number/APCCNCY42HHIJXSKVRWKP2A7K3/events.json","paper":"https://pith.science/paper/APCCNCY4"},"agent_actions":{"view_html":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3","download_json":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3.json","view_paper":"https://pith.science/paper/APCCNCY4","resolve_alias":"https://pith.science/api/pith-number/resolve?arxiv=2212.12017&json=true","fetch_graph":"https://pith.science/api/pith-number/APCCNCY42HHIJXSKVRWKP2A7K3/graph.json","fetch_events":"https://pith.science/api/pith-number/APCCNCY42HHIJXSKVRWKP2A7K3/events.json","actions":{"anchor_timestamp":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3/action/timestamp_anchor","attest_storage":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3/action/storage_attestation","attest_author":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3/action/author_attestation","sign_citation":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3/action/citation_signature","submit_replication":"https://pith.science/pith/APCCNCY42HHIJXSKVRWKP2A7K3/action/replication_record"}},"created_at":"2026-05-17T23:38:14.863396+00:00","updated_at":"2026-05-17T23:38:14.863396+00:00"}