{"id":"4730f1ab-9868-4f66-b4b7-31e09ab2db1f","arxiv_id":"2506.05695","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"POCL wraps LLM knowledge distillation in a curriculum that increases data difficulty and temperature over stages, improving Rouge-L on small GPT-2 and OPT students, though ablations show temperature drives the gains.","lead":"A new training recipe orders instruction data from easy to hard, then raises the distillation temperature, and reports better Rouge-L scores for small student models. The gains, however, come mostly from the temperature schedule, not from the easy-to-hard ordering itself.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own ablation (Table 3) removes the temperature and ratio schedules and POCL no longer beats plain KLD (23.21 vs 23.49 on DollyEval), so 'easy-to-hard ordering' is not established as the driver of the reported gains; the temperature schedule is confounded with it.","rationale":"The reader's conditional verdict is appropriate, and the reader's weakest assumption is the same load-bearing issue I identify: the curriculum ordering is confounded with the concurrently applied temperature schedule. The full-recipe results in Tables 1 and 4 are plausible and show consistent average gains, so REJECT would be too strong. However, the paper's framing is over-claimed: the curriculum ordering is the named contribution, and Tables 3 and 6 show that removing the temperature schedule (and ratio) eliminates or reverses the advantage over plain KD. This does not falsify the empirical result that 'POCL as a whole' helps, but it does mean the abstract and Section 5.2 conclusion about sorted samples cannot be accepted as the mechanism. A single shuffled-subset control would settle the attribution. The absence of error bars and the ambiguous epoch statement in Section A.2.2 are secondary; the temperature/curriculum confound is the load-bearing concern. Since the reader already recommended CONDITIONAL, my read does not change that verdict.","tokens_in":16960,"tokens_out":5564,"duration_ms":54971,"concrete_test":"Decisive check: run GPT-2 1.5B -> 0.1B KLD distillation with POCL's exact four-stage Baby Step schedule and the same temperature (1 -> 2) and alpha (0.3 -> 0) curves, but assign the 12.5K samples to the four subsets by a random permutation instead of by Eq. (2) difficulty ranking. Keep subset sizes, per-stage update counts, and checkpoints identical to the reported POCL run. If random-subset POCL matches the reported POCL Rouge-L on DollyEval and out-of-domain sets within about 0.1, difficulty ranking is not what drives the gain; if random-subset collapses, the ranking is doing real work. A second arm fixing tau=1 and alpha=0.3 with the same random subsets would directly reproduce the Table 3 'w/o temp & ratio' condition.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that sorting distillation samples from easy to hard (the curriculum component of POCL) improves student Rouge-L. The paper's own ablations undercut this. In Table 3, KLD+POCL 'w/o temp. & ratio'—the pure difficulty-scheduler variant, retaining only staged easy-to-hard exposure—scores 23.21 on DollyEval, below the 23.49 of plain KLD, and it is also worse on S-NI (18.97 vs 19.70) and UnNI (21.85 vs 22.01). The same pattern appears in Table 6 for GKD (24.05 vs 24.67 with temperature removed) and JSD (22.73 vs 23.79 for the scheduler alone). The full POCL recipe only surpasses baselines when the rising temperature schedule is added (24.87 / 26.60 / 25.97 in Tables 3 and 6), with the decreasing alpha ratio contributing a smaller increment. So the full package may work, but the component the paper motivates and names—curriculum ordering by difficulty—is not the causal ingredient in these experiments. The abstract's claim that 'sorted training samples' are effective is therefore not supported by the data as reported. The concern is mechanistic attribution, not fabrication: the numbers in Tables 1/4 can stand while the explanation for them is misidentified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes POCL, a curriculum-learning plug-in for white-box knowledge distillation of LLMs. POCL ranks training samples by difficulty using reciprocal rank fusion of the student's Rouge-L and cross-entropy scores, partitions the data into n difficulty subsets, and then trains the student in n stages, adding harder subsets while linearly increasing the distillation temperature and (for off-policy methods) decreasing the SFT ratio. The authors evaluate POCL on GPT-2 and OPT with KLD, RKL, JSD, TVD, SKL, SRKL, and GKD, reporting Rouge-L improvements on five instruction-following benchmarks over a plain-SFT/SeqKD/white-box-KD baseline set.","tokens_in":40,"tokens_out":5994,"duration_ms":105510,"significance":"If the full POCL recipe is robust, it would be a simple, plug-in training schedule that improves white-box KD across several loss functions and two model families, and the public code is a strength. The paper's breadth of base KD methods and its component-wise ablations are useful. However, the paper's own ablation isolating the curriculum-ordering component does not beat plain KLD, so the significance as a curriculum-learning contribution is not currently established; the paper would need to be reframed around the adaptive-temperature schedule or supported by a controlled experiment showing that difficulty ordering alone helps.","major_comments":[{"comment":"The variant that isolates the curriculum-ordering component—KLD+POCL (w/o temp. & ratio)—does not improve over plain KLD on DollyEval (23.21 vs 23.49), S-NI (18.97 vs 19.70), or UnNI (21.85 vs 22.01), and the same pattern appears for JSD (22.73 vs 23.79) and for GKD when temperature is removed (24.05 vs 24.67). Section 5.2 itself states that the pure POCL framework \"underperforms compared to baselines without POCL,\" and Table 7 shows that hard-to-easy ordering also improves over the baseline in most cells. This directly contradicts the abstract's claim that \"sorted training samples\" are effective and the mechanism stated in Section 3.2. The paper must either reframe the contribution as an adaptive-temperature/ratio training schedule or provide a controlled experiment showing that difficulty ordering alone yields gains (for example, staged easy-to-hard without temperature versus staged random-order without temperature).","section":"Section 5.2, Tables 3 and 6"},{"comment":"The conclusion that \"all variants of KLD with POCL outperform standard KLD without POCL, demonstrating the benefit of curriculum-based sample ordering\" is invalid because those POCL variants include the adaptive temperature and ratio schedules. This comparison cannot separate the ranking method from the rest of the POCL pipeline, so it does not establish a benefit of curriculum-based sample ordering per se. The ranking-method comparison within POCL is informative, but the ordering-benefit claim is conflated with the adaptive-parameter schedules.","section":"Section 5.1, Tables 2 and 5"},{"comment":"The paper reports means over five random seeds but provides no standard deviations, confidence intervals, or significance tests. Many of the reported gains are small (for example, OPT RKL +0.48 average, OPT JSD +0.77, OPT TVD +0.61), and several per-dataset cells decrease (for example, OPT JSD on SelfInst 12.57 vs 12.97 and on S-NI 23.30 vs 23.51; OPT RKL on S-NI 23.57 vs 23.92). Without variance estimates or a paired significance test, the \"consistently improves\" claim is not supported at the reported granularity. Please add per-seed results or error bars and perform paired significance tests, or explicitly downgrade the claim to \"often improves in these experiments.\"","section":"Tables 1 and 4"}],"minor_comments":[{"comment":"The base models are described as \"GPT-2 (120M) and OPT (350B),\" but the actual student models are GPT-2 0.1B and OPT 0.3B; \"350B\" should be \"350M.\" The same typo appears in Table 1's caption where \"GPT2-120M\" is used.","section":"Section 4.1"},{"comment":"Algorithm 1 says \"while not converged for p epochs\" but p is not defined in the input list, and \"converged\" is not operationalized. Appendix A.2.2 later says the framework is trained for 8 epochs per stage and for 40% of baseline epochs; please make the training-length accounting explicit and consistent between the algorithm, Section 4.1, and the appendix.","section":"Algorithm 1 and Appendix A.2.2"},{"comment":"The displayed formula for FRscore is typeset incorrectly (\"nX i\") and should be a sum over the ranked lists i; please fix the notation so that ri is clearly the rank of the sample in list i.","section":"Equation (2)"},{"comment":"The table notes attribute GKD to reference [1], but the GKD method is described in the text as reference [4]; the citation is inconsistent. Please correct the reference.","section":"Tables 1 and 4, notes"},{"comment":"In the caption for Figure 4, \"STF ratio\" should be \"SFT ratio.\"","section":"Figure 4 caption"},{"comment":"The text says \"Our theoretical analysis suggests\" that CL addresses catastrophic forgetting and training-inference mismatch, but no formal theoretical analysis appears in the paper; the section contains only intuitive arguments. Either remove the phrase \"theoretical analysis\" or provide a precise statement with supporting argument.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"This is a borderline case. The full POCL recipe appears to give consistent gains in many settings, but the paper's central attribution to curriculum ordering is contradicted by its own Table 3, and the significance claim as a curriculum-learning result is therefore not established. If the authors can either demonstrate an ordering benefit in a properly controlled comparison or reframe the contribution as an adaptive-temperature distillation schedule with appropriate statistical support, the result may be publishable; in the current form the abstract and conclusions overclaim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this paper reports a real, reproducible gain from a temperature-annealing schedule for white-box KD, but it misattributes that gain to easy-to-hard curriculum ordering. The ablations the authors themselves report show the ordering alone hurts or helps negligibly; the rising temperature is the active ingredient. So treat the headline claim with suspicion, but don't throw the paper out.\n\nWhat's new and good: they combine reciprocal rank fusion (Rouge-L + CE) to rank samples, stage training into four subsets, and anneal temperature from 1 to 2 (and alpha from 0.3 to 0). They test this plug-in across six KD losses plus GKD, on two model families, with consistent Rouge-L gains (mostly 0.5-2.6 points). Code is public. They also honestly report the ablation that gives away the game: in Table 3, KLD+POCL without temperature and ratio is 23.21 vs 23.49 for plain KLD on DollyEval, and worse on S-NI and UnNI. Table 6 shows the same for GKD and JSD. The paper's own text (Section 5.2) admits temperature has a stronger effect than the SFT ratio, but the abstract and title still sell the curriculum as the star.\n\nSoft spots: (1) No error bars or significance tests, though they report means over five seeds. (2) The training budget description is confusing: baselines get 20 epochs, POCL gets 8, yet they claim equal total steps; that needs a clear accounting of per-stage iterations. (3) Some citation numbering slips (e.g., GKD cited as [1] in Table 4 note). (4) The novelty of the effective component is thin: Curriculum Temperature for Knowledge Distillation [41] already does rising temperature for KD. The RRF ranking over student outputs is a new combination, but its contribution looks marginal.\n\nNet: this is a competent empirical study with a misleading frame. I'd accept it for peer review, but the referee should ask the authors to reframe the contribution around adaptive temperature, add variance reporting, and fix the training description. The curriculum part can remain as a minor component if the data support it, but not as the headline.\n\nIf you're working on KD, worth a skim for the temperature annealing results across losses. I wouldn't cite it as evidence for curriculum learning, though.\n\nBest","headline":"The paper's own ablations show the curriculum ordering is not the causal ingredient—the rising temperature schedule is—so the headline claim is overstated, but the empirical package is real and worth a referee's time.","tokens_in":34,"tokens_out":2954,"would_cite":false,"duration_ms":62234,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an easy-to-hard curriculum schedule with rising temperature consistently improves white-box knowledge distillation of large language models.","keywords":["knowledge distillation","curriculum learning","large language models","white-box distillation","difficulty ranking","temperature scheduling","instruction following","reciprocal rank fusion"],"falsifier":"Train the same student on random-order sample subsets while keeping the identical rising-temperature and alpha schedules; if ROUGE-L matches POCL, the easy-to-hard ordering contributes nothing. A cheaper check is already in the paper: POCL with temperature and ratio removed scores 23.21 versus 23.49 for plain KLD on DollyEval, so an experiment that fixes the temperature at 1 while varying only the order would determine whether the ordering claim is right.","tokens_in":16752,"feed_emoji":"📈","tokens_out":9817,"duration_ms":92695,"temperature":0.7,"pith_summary":"The paper claims that ordering the data used in white-box knowledge distillation from easy to hard, a curriculum it calls POCL, makes distilled language models consistently better on five instruction-following benchmarks, across two model families and six distillation losses. The framework ranks each training sample using a fusion of the student model's ROUGE-L and cross-entropy scores, partitions the data into four difficulty tiers, and feeds the tiers to the student progressively while the distillation temperature rises from 1 to 2. Because POCL is a wrapper around the existing KD loss, the authors argue it is cheap and portable: it does not change the loss or the teacher, only the order and schedule of training. The larger point is that data structure inside distillation is itself a performance lever.","feed_headline":"Easy-to-hard training lifts distilled LLM Rouge-L","feed_subtitle":"A plug-in curriculum with rising temperature improves student models trained by six white-box distillation losses.","key_machinery":"The load-bearing machinery is POCL, a two-part framework: a difficulty measurer and a training scheduler. The difficulty measurer computes two rankings per training sample, one by ROUGE-L between student output and ground truth and one by student cross-entropy, and merges them through reciprocal rank fusion, $FR_{\\mathrm{score}} = \\sum_i 1/(k + r_i)$ with $k = 60$, partitioning the 12.5K training samples into four subsets from easiest to hardest. The scheduler, called Baby Step, starts training on the easiest subset and adds the next subset at fixed intervals or convergence, while the distillation temperature $\\tau$ grows linearly from 1 to 2 and the SFT ratio $\\alpha$ falls from 0.3 to 0 for off-policy methods. The framework's role is to control the order and sharpness of the distributions the student is matched against, which the paper argues prevents abrupt distribution shifts during distillation.","core_discovery":"The central claim is that a curriculum schedule, not a new loss function, is what stabilizes and improves white-box KD of LLMs. In the paper's terms, the difficulty measurer ranks samples by reciprocal rank fusion over ROUGE-L and cross-entropy; the Baby Step scheduler then trains on the easiest subset first and adds harder subsets at successive stages, while the distillation temperature is linearly raised from 1 to 2 and off-policy methods shift their SFT weighting from 0.3 down to 0. Across KLD, RKL, JSD, TVD, SKL, SRKL, and GKD, POCL raises average ROUGE-L relative to plain baselines, with average gains between roughly 0.5 and 2.6 points depending on method and student model. The authors interpret this as evidence that progressive overload reduces teacher-student distribution shift, mitigates catastrophic forgetting and mode collapse, and eases training-inference mismatch.","pith_inferences":["My inference: the active ingredient may be the temperature schedule rather than the curriculum order, since the paper's own Table 3 shows ordering-only POCL loses to plain KLD on DollyEval; a randomized-order control with the same temperature schedule would settle this.","A directly testable extension is to hold the rising-temperature schedule fixed and randomize sample order within stages; if ROUGE-L is unchanged, the easy-to-hard ordering is not carrying the effect.","Because POCL's difficulty measure uses the student model's own confidence, the ranking is capacity-dependent; as student models grow, the perceived difficulty spread may narrow and the gains may shrink, a possibility the paper itself raises in its limitations.","The same staging idea could transfer to black-box distillation by substituting a proxy difficulty signal such as teacher output entropy, but the paper does not demonstrate that transfer."],"forward_implications":["Adding POCL to any of the tested white-box KD losses raises average ROUGE-L across five instruction-following benchmarks for both the GPT-2 0.1B and OPT 0.3B students.","Easy-to-hard ordering outperforms hard-to-easy ordering, although hard-to-easy still improves over plain training in most settings.","The rising temperature component is necessary for the reported gains; removing it leaves POCL at or below the unmodified baseline.","Because POCL is a wrapper, it can be dropped onto existing distillation loops without changing the loss or the teacher, at roughly the same total training steps.","Staged training reaches higher validation ROUGE-L earlier than the corresponding KD baseline, indicating faster convergence as well as better final performance."],"supporting_citations":[{"why":"Provides the token-level KLD distillation loss and temperature scaling that POCL wraps.","marker":"[1]"},{"why":"Supplies the on-policy GKD baseline and student-generated-output data strategy that POCL augments.","marker":"[4]"},{"why":"Supplies the reverse-KLD baseline and the instruction-following evaluation setup used throughout the experiments.","marker":"[24]"},{"why":"Supplies the curriculum-learning background motivating the claim that easy-to-hard exposure stabilizes training.","marker":"[27]"},{"why":"Defines ROUGE-L, used both as the evaluation metric and as one of the two difficulty-ranking signals.","marker":"[31]"},{"why":"Provides the reciprocal rank fusion formula that combines the two difficulty rankings.","marker":"[32]"},{"why":"Is the origin of the Baby Step curriculum schedule that POCL's training scheduler adopts.","marker":"[33]"},{"why":"Motivates the least-to-most prompting schedule underlying the decreasing SFT-ratio schedule.","marker":"[34]"},{"why":"Provides the Dolly instruction-response dataset used for training and the DollyEval evaluation set.","marker":"[35]"}],"fun_headline_variants":["Curriculum beats new loss for LLM distillation","Progressive overload boosts distilled LLM scores","Easy-first training stabilizes LLM knowledge transfer","Rising temperature curriculum lifts distilled LLM quality","Sorted samples improve white-box LLM distillation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that easy-to-hard sample ordering is the active ingredient, but the paper's own Table 3 shows that ordering without the temperature schedule does not beat plain KLD, so the mechanism attributed to the curriculum is not cleanly isolated.","fun_headline_variants_meta":{"raw":{"variants":["Curriculum beats new loss for LLM distillation","Progressive overload boosts distilled LLM scores","Easy-first training stabilizes LLM knowledge transfer","Rising temperature curriculum lifts distilled LLM quality","Sorted samples improve white-box LLM distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000326,"raw_usage":{"total_tokens":1845,"prompt_tokens":983,"completion_tokens":862,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":599,"completion_tokens_details":{"reasoning_tokens":794}},"tokens_in":599,"tokens_out":862,"duration_ms":7510,"temperature":1.0,"reasoning_tokens":794,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:12:59.411840+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the same student on random-order sample subsets while keeping the identical rising-temperature and alpha schedules; if ROUGE-L matches POCL, the easy-to-hard ordering contributes nothing. A cheaper check is already in the paper: POCL with temperature and ratio removed scores 23.21 versus 23.49 for plain KLD on DollyEval, so an experiment that fixes the temperature at 1 while varying only the order would determine whether the ordering claim is right.","supporting_citations":[{"cited_title":"A survey on curriculum learning","cited_arxiv_id":null,"evidence_quote":"Supplies the curriculum-learning background motivating the claim that easy-to-hard exposure stabilizes training."},{"cited_title":"Reciprocal rank fusion outperforms condorcet and individual rank learning methods","cited_arxiv_id":null,"evidence_quote":"Provides the reciprocal rank fusion formula that combines the two difficulty rankings."},{"cited_title":"A comparison of most-to-least and least-to-most prompting on the acquisition of solitary play skills","cited_arxiv_id":null,"evidence_quote":"Motivates the least-to-most prompting schedule underlying the decreasing SFT-ratio schedule."}],"review_version":1}