{"work":{"id":"aebc3613-3aa4-4807-b7a7-c7496f1d7f34","openalex_id":"https://openalex.org/W4391631876","doi":"10.48550/arxiv.2402.03898","arxiv_id":"2402.03898","raw_key":null,"title":"DistiLLM: Towards Streamlined Distillation for Large Language Models","authors":null,"authors_text":"Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se-Young Yun","year":2024,"venue":"cs.CL","abstract":"Knowledge distillation (KD) is widely used for compressing a teacher model to a smaller student model, reducing its inference cost and memory footprint while preserving model capabilities. However, current KD methods for auto-regressive sequence models (e.g., large language models) suffer from missing a standardized objective function. Moreover, the recent use of student-generated outputs to address training-inference mismatches has significantly escalated computational costs. To tackle these issues, we introduce DistiLLM, a more effective and efficient KD framework for auto-regressive language models. DistiLLM comprises two components: (1) a novel skew Kullback-Leibler divergence loss, where we unveil and leverage its theoretical properties, and (2) an adaptive off-policy approach designed to enhance the efficiency in utilizing student-generated outputs. Extensive experiments, including instruction-following tasks, demonstrate the effectiveness of DistiLLM in building high-performing student models while achieving up to 4.3$\\times$ speedup compared to recent KD methods.","external_url":"https://arxiv.org/abs/2402.03898","cited_by_count":4,"metadata_source":"pith","metadata_fetched_at":"2026-08-05T02:28:24.338817+00:00","pith_arxiv_id":"2402.03898","created_at":"2026-05-10T00:54:49.026589+00:00","updated_at":"2026-08-05T02:28:24.338817+00:00","title_quality_ok":true,"display_title":"Distillm: Towards streamlined distillation for large language models.ArXiv, abs/2402.03898","render_title":"Distillm: Towards streamlined distillation for large language models.ArXiv, abs/2402.03898"},"hub":{"state":{"work_id":"aebc3613-3aa4-4807-b7a7-c7496f1d7f34","tier":"hub","tier_reason":"10+ Pith inbound or 1,000+ external citations","pith_inbound_count":17,"external_cited_by_count":4,"distinct_field_count":4,"first_pith_cited_at":"2026-04-16T05:13:57+00:00","last_pith_cited_at":"2026-07-06T17:59:58+00:00","author_build_status":"not_needed","summary_status":"needed","contexts_status":"needed","graph_status":"needed","ask_index_status":"not_needed","reader_status":"not_needed","recognition_status":"not_needed","updated_at":"2026-08-23T07:09:59.189874+00:00","tier_text":"hub"},"tier":"hub","role_counts":[{"context_role":"background","n":1},{"context_role":"method","n":1}],"polarity_counts":[{"context_polarity":"background","n":2}],"runs":{},"summary":{},"graph":{},"authors":[]}}