{"work":{"id":"dd464d2b-4adb-40de-9148-c19470e7533b","openalex_id":"https://openalex.org/W4312205996","doi":"10.48550/arxiv.2212.12017","arxiv_id":"2212.12017","raw_key":null,"title":"OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization","authors":null,"authors_text":"Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Daniel Simig, Ping Yu","year":2022,"venue":"cs.CL","abstract":"Recent work has shown that fine-tuning large pre-trained language models on a collection of tasks described via instructions, a.k.a. instruction-tuning, improves their zero and few-shot generalization to unseen tasks. However, there is a limited understanding of the performance trade-offs of different decisions made during the instruction-tuning process. These decisions include the scale and diversity of the instruction-tuning benchmark, different task sampling strategies, fine-tuning with and without demonstrations, training using specialized datasets for reasoning and dialogue, and finally, the fine-tuning objectives themselves. In this paper, we characterize the effect of instruction-tuning decisions on downstream task performance when scaling both model and benchmark sizes. To this end, we create OPT-IML Bench: a large benchmark for Instruction Meta-Learning (IML) of 2000 NLP tasks consolidated into task categories from 8 existing benchmarks, and prepare an evaluation framework to measure three types of model generalizations: to tasks from fully held-out categories, to held-out tasks from seen categories, and to held-out instances from seen tasks. Through the lens of this framework, we first present insights about instruction-tuning decisions as applied to OPT-30B and further exploit these insights to train OPT-IML 30B and 175B, which are instruction-tuned versions of OPT. OPT-IML demonstrates all three generalization abilities at both scales on four different evaluation benchmarks with diverse tasks and input formats -- PromptSource, FLAN, Super-NaturalInstructions, and UnifiedSKG. Not only does it significantly outperform OPT on all benchmarks but is also highly competitive with existing models fine-tuned on each specific benchmark. We release OPT-IML at both scales, together with the OPT-IML Bench evaluation framework.","external_url":"https://arxiv.org/abs/2212.12017","cited_by_count":85,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2212.12017","created_at":"2026-05-10T22:46:39.814559+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization","render_title":"OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization"},"hub":{"state":{"work_id":"dd464d2b-4adb-40de-9148-c19470e7533b","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":27,"external_cited_by_count":85,"distinct_field_count":7,"first_pith_cited_at":"2023-01-31T15:03:44+00:00","last_pith_cited_at":"2026-07-08T05:44:33+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-21T23:59:37.519220+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":12},{"context_role":"method","n":1}],"polarity_counts":[{"context_polarity":"background","n":13}],"runs":{},"summary":{},"graph":{},"authors":[]}}