{"id":"1a48a566-357c-452b-871c-0125eae7dee1","arxiv_id":"2505.20888","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"EasyDistill packages established LLM knowledge-distillation techniques into a single modular toolkit with released distilled models, datasets, and Alibaba Cloud integration.","lead":"EasyDistill is a new open-source toolkit that bundles data synthesis, supervised fine-tuning, ranking optimization, and reinforcement learning into one pipeline for knowledge distillation of large language models. It also ships pretrained 'DistilQwen' models, datasets, and a cloud integration on Alibaba Cloud's PAI platform.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'effective' claim rests on Table 2's LiveCodeBench scores without baselines, error bars, teacher comparisons, or speedup methodology; without a controlled evaluation the central claim is unsupported.","rationale":"The reader's overall CONDITIONAL verdict is appropriate, and I agree with the concern about thin benchmarking. However, I identify a different load-bearing concern than the reader's stated weakest assumption. The top-k logits approximation (Section 2.1.2) is a potential implementation risk, but it is optional ('offers users options') and not directly tied to the headline effectiveness evidence. The single most load-bearing concern is that the only quantitative support for the central claim that EasyDistill produces effective models is Table 2, which lacks baselines, error bars, and methodology. If the numbers cannot be reproduced or if the distilled models do not beat their own base models, the central value proposition collapses. This concern is about unverifiable evidence rather than internal inconsistency. A controlled reproduction would settle it. The reader's rationale already mentions the absence of baselines, so agreement is partial: the weakest_assumption field points to top-k rather than the evaluation gap, but the overall verdict remains unchanged.","tokens_in":10010,"tokens_out":6208,"duration_ms":62711,"concrete_test":"Reproduce Table 2 with a controlled protocol: take the released DistilQwen code models and their base counterparts (Qwen2.5-3B-Instruct, Qwen2.5-3B-Code, Qwen2.5-7B-Instruct, Qwen2.5-7B-Code) plus the teacher (DeepSeek-R1), run the LiveCodeBench V2 harness with a fixed number of samples per problem (e.g., 100, temperature 0.2), and compute Pass@1 with 95% confidence intervals. Also measure end-to-end inference latency on identical hardware (e.g., A100) with the same batch size and max tokens. If the distilled models do not significantly outperform their base models, or if the speedup is not reproduced, the effectiveness claim for the toolkit fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that EasyDistill is an effective toolkit for LLM knowledge distillation, demonstrated by the released DistilQwen models and datasets. The only in-paper quantitative evidence for this claim is Table 2 (Section 2.3.2), reporting LiveCodeBench V2 Pass@1 and 'Speedup' for four variants (Qwen2.5-3B/7B Instruct/Code). This table has no baselines: no teacher model score, no pre-distillation base-model score, no error bars or sample counts, no evaluation harness description, and no measurement methodology for speedup. Two rows (7B) do not even report a speedup. The text says 'we distill two models' but lists four rows, leaving ambiguity about which models are distilled and which are base models. Without a controlled comparison, the reader cannot determine whether the improvement of Qwen2.5-3B-Code (16.62 vs 11.35) is due to EasyDistill or to the base model's existing code specialization, nor whether the 2.3x speedup is real. No comparison against existing toolkits (e.g., DistillKit) is provided, so the 'comprehensive and effective' characterization is unverified. If Table 2 is shown to be non-reproducible, or if the distilled models fail to beat their own base models on identical benchmarks, the core claim of the paper would be substantially weakened.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents EasyDistill, an open-source toolkit for knowledge distillation (KD) of large language models. It supports data synthesis and augmentation, black-box SFT, white-box logit-based KD (forward and reverse KLD), preference rank optimization (DPO, CogPO), reinforcement learning (PPO, GRPO), and multi-modal KD, and it ships a series of DistilQwen models plus open datasets. The toolkit is modular, exposes a command-line interface with JSON configurations, includes practical recipes for general and domain-specific KD, and is integrated into Alibaba Cloud's PAI platform. The central claim is that EasyDistill is a comprehensive and effective toolkit for producing lightweight LLMs, with the main quantitative evidence being a single LiveCodeBench V2 table for code-distilled models.","tokens_in":10259,"tokens_out":3582,"duration_ms":39148,"significance":"If the effectiveness claim were properly substantiated, EasyDistill would be a useful engineering contribution: it covers a broad set of KD algorithms, provides released checkpoints and datasets, and gives reproducible configuration examples. The authors deserve credit for open-sourcing code, models, and datasets, and for integrating a full KD pipeline with cloud deployment. However, as it stands the paper reads more as a system description than as an empirical validation. The central effectiveness claim rests on a single benchmark table with no baselines, no error bars, no teacher comparisons, and no speedup methodology. The paper therefore currently does not establish that EasyDistill is effective in a way that is measurable against existing toolkits or against the base models it starts from.","major_comments":[{"comment":"The only in-paper quantitative evidence for the central 'effective' claim is Table 2, which reports LiveCodeBench V2 Pass@1 and speedup for four models. This table has no teacher-model scores, no pre-distillation base-model scores, no error bars or sample counts, no evaluation-harness description, and no measurement methodology for the speedup. The text says 'we distill two models' but the table lists four rows, so it is unclear whether the Code variants are EasyDistill products or the original Qwen2.5-Code base models. Without a controlled comparison on identical benchmarks, the reader cannot determine whether the reported gains are attributable to EasyDistill or to the base model's existing capabilities.","section":"Section 2.3.2, Table 2"},{"comment":"The white-box KD implementation restricts the loss to the top-k (k=10) teacher logits, motivated by the statement that 'the sum of the probabilities of the top-10 tokens is almost equal to 1.' This is cited from the authors' own prior report, Wang et al. (2025), and no experiment or error analysis is provided in this paper. Since this approximation is load-bearing for the white-box KD functionality, the paper should either include a measurement of top-10 probability-mass coverage on the target tasks or a comparison between full-vocabulary and top-k distillation to show that the approximation does not degrade student quality.","section":"Section 2.1.2"},{"comment":"The effectiveness evidence is largely self-referential: the DistilQwen models produced by EasyDistill are cited as proof that the pipeline works, while the underlying method papers (Wang et al. 2025, Cai et al. 2025a, Cai et al. 2025b) are also by the same group. The paper does not compare EasyDistill against existing toolkits such as DistillKit, nor does it include external or third-party evaluation results. A concrete comparison on a common benchmark, using the same student initialization and the same teacher, would be needed to support the claim that EasyDistill is both comprehensive and effective relative to the current state of the art.","section":"Section 2.3 and Section 4"},{"comment":"The reported '2.3x' speedup is not defined: it is unclear whether this compares the distilled student to the teacher, to the base student without distillation, or to some other reference, and no hardware, batch size, or decoding configuration is given. The two 7B rows report no speedup at all, which further obscures the comparison. The speedup claim should be restated with a precise experimental protocol.","section":"Section 2.3.2"}],"minor_comments":[{"comment":"The word 'fauces' should be 'focuses' in the description of OmniThought-0528.","section":"Section 2.3.3"},{"comment":"The entry 'DeepSeek-V3-0234' appears to be a typo for 'DeepSeek-V3-0324', which is used elsewhere in the text.","section":"Table 1"},{"comment":"The JSON configuration examples are not always valid JSON: Code 1 is missing a comma after the dataset block, and the examples contain ellipses in place of required fields. For a paper whose contribution is a user-friendly toolkit, complete runnable examples would be more helpful.","section":"Section 2.2, Code 1-3"},{"comment":"Table 3 uses '[URL]' placeholders for the dataset links; these should be actual URLs or the table should be removed.","section":"Table 3"},{"comment":"The heading 'Ethic Considerations' should be 'Ethical Considerations'.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The paper is a system/toolkit description with a heavy reliance on the authors' own prior work for the underlying methods. The central empirical claim would need a substantially expanded evaluation section, including baselines, provenance of all reported models, and a comparison with an existing toolkit. This may be a scope issue for a research venue unless the evaluation is strengthened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"EasyDistill is a real, working toolkit: it integrates SFT, logit-based KLD, DPO/CogPO, PPO/GRPO, data synthesis/augmentation, and ships a family of DistilQwen models and open datasets, with PAI integration. For a practitioner who wants one codebase to run black-box and white-box KD, this is a useful package, and the JSON-config CLI looks pragmatic. Credit where due: the modular architecture is coherent, the release artifacts are substantial, and the recipes give concrete starting points.\n\nThe problem is the evidence. The paper's central claim—that EasyDistill is 'effective'—rests almost entirely on Table 2, which reports LiveCodeBench V2 Pass@1 and speedup for four Qwen variants. The table has no teacher-model scores, no pre-distillation baseline for the student's base model, no error bars or sample sizes, and no description of the evaluation harness or the speedup measurement. Two rows leave the speedup blank. The text says 'we distill two models' while listing four rows, so it's unclear which rows are distilled products and which are the starting checkpoints. Without controlling for the base model's existing code specialization, the 16.62 vs 11.35 gap could be inherited, not produced by the toolkit. No comparison to DistillKit or other toolkits is provided, so 'comprehensive and effective' is asserted, not demonstrated.\n\nA secondary soft spot: the white-box KD implementation truncates teacher logits to top-k (k=10) based on a claim from a self-cited report, with no error analysis in this paper. That may be fine in practice, but it is an unvalidated assumption.\n\nNone of this is fatal. The weaknesses are fillable: run controlled experiments with teacher and base-model baselines, report variance and evaluation details, clarify the speedup methodology, and add a toolkit comparison. The self-citation pattern is heavy but the underlying components are standard and the released artifacts are verifiable; I don't see incoherence or bad faith.\n\nThis paper is for practitioners and tool-builders, not for readers seeking new algorithmic insights. It deserves a serious referee; the right outcome is conditional acceptance with a demand for a real evaluation section.","headline":"A genuinely useful KD toolkit whose effectiveness claim is left unsupported by a single uncontrolled benchmark table.","tokens_in":10838,"tokens_out":2253,"would_cite":true,"duration_ms":21906,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One toolkit now covers black-box and white-box distillation of large language models.","keywords":["knowledge distillation","large language models","black-box distillation","white-box distillation","chain-of-thought","DPO","reinforcement learning","toolkit"],"falsifier":"Train the same student with full-vocabulary KLD and with top-10 KLD on identical data, and compare downstream benchmarks such as LiveCodeBench V2; a material performance gap would falsify the top-$k$ approximation assumption.","tokens_in":9793,"feed_emoji":"🧰","tokens_out":7543,"duration_ms":63724,"temperature":0.7,"pith_summary":"This paper introduces EasyDistill, a modular toolkit that spans the full knowledge-distillation (KD) pipeline for large language models: synthetic data creation, supervised fine-tuning on teacher outputs, logit-based white-box distillation, DPO/CogPO preference optimization, and PPO/GRPO reinforcement learning. The authors claim that these pieces can be combined in a single command-line configuration for both fast System 1 students and reasoning-oriented System 2 students. The claim is anchored by released artifacts — the DistilQwen model series and open datasets such as DistilQwen 100K/1M and OmniThought — plus a code-generation recipe whose distilled models score higher on LiveCodeBench V2 and run faster than their base counterparts. If the recipes reproduce, EasyDistill lowers the barrier to making smaller, cheaper models that behave like much larger teachers.","feed_headline":"This toolkit distills giant LLMs into small fast students","feed_subtitle":"EasyDistill bundles data synthesis, SFT, preference optimization, and RL with released DistilQwen models and datasets.","key_machinery":"The load-bearing mechanism is the layered EasyDistill pipeline, in which seed instructions are expanded, refined, or extracted from raw text, CoT operations simplify or extend reasoning traces, and a training stage then applies black-box SFT or white-box logit matching. The white-box path precomputes teacher logits and evaluates the loss only on the top $k=10$ tokens, on the premise that their probabilities sum to almost 1; this reduces computation and logit storage. Preference and RL stages (DPO, CogPO, PPO, GRPO) refine the student's behavior after SFT, and the released DistilQwen models plus OmniThought datasets serve as reusable validation artifacts for the whole pipeline.","core_discovery":"The paper's central claim is that a single extensible toolkit can make LLM knowledge distillation practical in both black-box and white-box settings. The authors argue that their pipeline — teacher-driven data synthesis, black-box SFT, white-box top-k logit matching with KLD and reverse KLD, DPO/CogPO ranking optimization, and PPO/GRPO reinforcement learning — is enough to produce lightweight students that inherit capabilities from proprietary and open-source teachers. They offer the DistilQwen series as evidence: System 1 models distilled from GPT-4, Qwen-max, Qwen2.5, DeepSeek-V3-0324, and System 2 models distilled from DeepSeek-R1, DeepSeek-R1-0528, and QwQ-32B. The code-distillation recipe shows Qwen2.5-3B-Code reaching 16.62 on LiveCodeBench V2 versus 11.35 for the base 3B instruct model, with a 2.3x inference speedup, indicating that KD can push small models closer to large-model code generation ability.","pith_inferences":["The top-10 logit assumption is empirically testable beyond the toolkit: varying $k$ under different temperatures and data domains would show whether the approximation holds for long-tail and multilingual vocabulary, which the paper does not examine.","The OmniThought verbosity and difficulty scores could be reused outside KD as a curriculum for training-time or inference-time token budgets, a direction the paper only implies.","The code-generation recipe sets up a clean controlled comparison — same base model with and without distillation on LiveCodeBench V2 — that practitioners could extend to other domains such as math, instruction following, or tool use.","The paper's evaluation evidence is scattered across model cards and prior reports; a single reproducible benchmark comparing EasyDistill recipes with standard SFT would be a strengthening next step, but that is my inference, not a claim the paper makes."],"forward_implications":["Users can run the same KD pipeline against proprietary teachers through any OpenAI-compatible API or against local open-source teachers through vLLM, with one JSON config and one command.","The top-$k$ logit approximation means white-box KD can be run with substantially less GPU memory and logit I/O than full-vocabulary matching, under the stated assumption that $k=10$ captures nearly all probability mass.","Reasoning-oriented student models can inherit chain-of-thought behavior from large reasoning teachers through synthetic CoT data, token-length simplification, and CogPO alignment.","The released DistilQwen models and DistilQwen 100K/1M and OmniThought datasets give the community reproducible starting points and baselines for further KD experiments.","Because the toolkit is integrated into a cloud machine-learning platform, the same recipes can move from local experimentation to large-scale industrial deployment."],"supporting_citations":[{"why":"Supplies the prior finding that top-10 token probabilities sum to almost 1, which justifies the white-box top-k logit approximation, and the DistilQwen2.5 training recipe.","marker":"Wang et al., 2025"},{"why":"Provides the task-aware curriculum planning method used to balance instruction distributions before black-box SFT for DistilQwen2.","marker":"Yue et al., 2024b"},{"why":"Defines the KLD loss for LLM knowledge distillation used in the white-box training path.","marker":"Gu et al., 2024"},{"why":"Supplies the reverse KLD loss as an alternative white-box distillation objective.","marker":"Wu et al., 2025"},{"why":"Introduces DPO, the preference optimization method integrated for stable alignment.","marker":"Rafailov et al., 2023"},{"why":"Provides the DPO training pipeline that EasyDistill adapts for KD preference optimization.","marker":"Tunstall et al., 2023"},{"why":"Supplies PPO, the reinforcement learning algorithm used for System 1 students.","marker":"Schulman et al., 2017"},{"why":"Introduces GRPO, the reinforcement learning algorithm used for System 2 reasoning students.","marker":"Shao et al., 2024"},{"why":"Introduces CogPO, the cognitive preference optimization algorithm used to align reasoning students with their capacities.","marker":"Cai et al., 2025b"},{"why":"Introduces the OmniThought dataset and the Reasoning Verbosity and Cognitive Difficulty scores used to select CoT training data.","marker":"Cai et al., 2025a"}],"fun_headline_variants":["Toolkit distills LLMs into fast small models","One toolkit for black-box and white-box LLM distillation","DistilQwen: small models learn from GPT-4 and DeepSeek","KD toolkit cuts 3B model speedup 2.3x on code tasks","Streamline distillation with EasyDistill's modular pipeline"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The white-box distillation path assumes that the ten most likely next tokens carry essentially all of the teacher's probability mass, so ignoring the rest of the vocabulary does not hurt the student.","fun_headline_variants_meta":{"raw":{"variants":["Toolkit distills LLMs into fast small models","One toolkit for black-box and white-box LLM distillation","DistilQwen: small models learn from GPT-4 and DeepSeek","KD toolkit cuts 3B model speedup 2.3x on code tasks","Streamline distillation with EasyDistill's modular pipeline"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000122,"raw_usage":{"total_tokens":1100,"prompt_tokens":952,"completion_tokens":148,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":68}},"tokens_in":568,"tokens_out":148,"duration_ms":2583,"temperature":1.0,"reasoning_tokens":68,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:44:57.827225+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same student with full-vocabulary KLD and with top-10 KLD on identical data, and compare downstream benchmarks such as LiveCodeBench V2; a material performance gap would falsify the top-$k$ approximation assumption.","supporting_citations":[{"cited_title":"Manning, Stefano Ermon, and Chelsea Finn","cited_arxiv_id":null,"evidence_quote":"Introduces DPO, the preference optimization method integrated for stable alignment."}],"review_version":1}