{"id":"850eef27-8e05-4ac8-99cf-ae0072db57ad","arxiv_id":"2505.06270","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A WiP paper claims a mathematical rationale for dynamically adjusting the KD loss balancing parameter, but the derivation only shows the loss-reduction size depends on the weighting.","lead":"This paper derives a first-order Taylor expansion for the loss decrease in knowledge distillation and argues that the balancing parameter between the teacher and task losses should change during training. The derivation is standard, and the paper does not demonstrate that dynamic adjustment improves performance.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The derivation proves only that per-step loss decrease depends on λ; it never shows dynamic adjustment is necessary or improves any stated objective, so the central normative claim is a non sequitur.","rationale":"The paper's only concrete result is an identity for the first-order loss decrease. Considered as algebra, it is correct up to the dropped O(η²) term. But the advertised contribution is a \"mathematical rationale\" for dynamic balancing-parameter control. A rationale must connect the formula to a precise sense in which dynamic control is necessary or beneficial. The paper never does this: \"desired learning behavior\" is not defined, no baseline is compared, and the Conclusion admits future work will investigate an algorithm. Thus even if every equation were exact, the central claim would not be established. The Taylor remainder is a genuine but secondary concern: it affects whether Eq. (7) is the true loss change, whereas the normative gap affects whether the paper has argued for its headline claim at all. For these reasons I concur with the reader's REJECT verdict, while locating the decisive weakness in the missing optimality argument rather than in the truncation error.","tokens_in":3419,"tokens_out":6257,"duration_ms":63249,"concrete_test":"Formalize the claimed benefit as follows: minimize ∑_{t=1}^T L(θ_t) under gradient descent with loss Eq. (1) on a two-parameter convex quadratic instance L(θ) = λ||θ-a||² + (1-λ)||θ-b||², with a and b fixed and random θ_0. Grid-search the best constant λ* and compare it with any dynamic schedule that at each step picks λ_t maximizing the first-order decrease in Eq. (7). If the best fixed λ* yields equal or lower cumulative loss over random initializations, the paper's \"should be dynamically adjusted\" claim is refuted in the very setting the paper analyzes; if the dynamic schedule is better, the authors still need to state the criterion and prove it generally.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section II-B's algebra is internally consistent: Eqs. (9)-(14) amount to ∆L_{i+1} ≈ -η||λ∇L_dist + (1-λ)∇L_cls||², so the first-order loss decrease is a quadratic function of λ. The abstract and Discussion, however, conclude that \"more effective knowledge distillation requires dynamic adjustment of λ at each step.\" That conclusion does not follow. The formula only shows that the size of the one-step decrease depends on λ; a fixed λ also decreases the loss, and no objective function (final loss, convergence speed, gradient conflict, etc.) is specified by which \"dynamic is better\" could be judged. No comparison against a fixed-λ baseline is made, and no schedule or update rule for λ is proposed. The paper even defers such an algorithm to future work. The load-bearing premise is therefore not the Taylor expansion but the unstated entailment from \"λ affects the decrease\" to \"λ must be adjusted dynamically\"; that entailment is false without additional optimality assumptions. A secondary technical issue: Eqs. (5)-(6) drop the O(η²) term with no small-learning-rate condition, so the approximate sign and magnitude in Eq. (7) are not guaranteed for the learning rates used in practice.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript analyzes a simple knowledge-distillation objective L = λL_dist + (1−λ)L_cls and uses a first-order Taylor expansion (Eqs. (5)–(6)) to express the per-step loss decrease as −η times the squared norm of the weighted gradient combination (Eq. (14)). The authors conclude from this expression that the balancing parameter λ should be dynamically adjusted during training. No experiments, baselines, or update rules for λ are presented; the Conclusion defers such an algorithm to future work.","tokens_in":3644,"tokens_out":2943,"duration_ms":26230,"significance":"The algebraic derivation is transparent and appears internally consistent; it correctly captures the first-order dependence of the loss decrease on λ. If the normative claim were supported, it would give a theoretical justification for time-varying KD loss weights. However, as argued below, the conclusion does not follow from the equations, and no evidence is provided that dynamic adjustment outperforms any fixed λ. The practical significance in its current form is therefore limited.","major_comments":[{"comment":"The conclusion that 'more effective knowledge distillation requires dynamic adjustment of λ at each step' is a non sequitur relative to Eq. (14). That equation only shows that the magnitude of a single first-order loss decrease depends on λ; it does not compare fixed versus dynamic schedules, specify an objective (e.g., final loss, convergence speed, generalization) with respect to which dynamic adjustment is optimal, or rule out the possibility that some fixed λ is optimal throughout training. Without such an optimality argument or an empirical comparison, the central claim is unsupported.","section":"Abstract and Section III"},{"comment":"The first-order Taylor expansion discards the O(η²) remainder without stating any smallness condition on η or any bound on the Hessian of L. The subsequent sign and magnitude statements for ∆L_{i+1} (Eqs. (7)–(14)) are therefore not guaranteed for finite learning rates used in practice; the second-order term could change the sign of the loss difference. The paper must either state and justify a small-η assumption or bound the remainder.","section":"Section II-B, Eqs. (5)–(6)"},{"comment":"The paper acknowledges that 'future work focuses and investigates an algorithm' for dynamically adjusting λ. Since no update rule for λ is proposed and no objective for 'desired learning behavior' (Section III) is formalized, the manuscript does not deliver a testable prediction or method; the stated contribution is only a heuristic motivation.","section":"Section IV (Conclusion)"}],"minor_comments":[{"comment":"There is a typo: 'downsteam-task loss' should be 'downstream-task loss'.","section":"Abstract"},{"comment":"The phrase 'the term in square brackets equals the squared sum of the two gradients' is inaccurate; the term equals the squared norm of the weighted sum λ∇L_dist + (1−λ)∇L_cls, not the sum of the two squared-norm terms.","section":"Section II-B, after Eq. (14)"},{"comment":"The caption does not specify how the plotted curves are computed from Eq. (14) or what exactly the red and green curves represent beyond 'cases'.","section":"Figure 2 caption"},{"comment":"Reference [10] is cited for 'methods for determining an optimal balancing parameter', but [10] concerns gradient surgery in multi-task learning; the intended reference may be [12] or another work.","section":"References"}],"recommendation":"reject","confidential_remarks":"The manuscript reads as a preliminary WiP note. The central claim is an overreach of the algebra, and the authors themselves defer the actual algorithm to future work. For a journal, this is well below the bar in its current form; a revision would need to either add a controlled comparison of fixed and dynamic λ schedules or substantially weaken the claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper is exactly what it says on the tin: a WiP preprint with clean notation and a correct but textbook derivation that the one-step loss decrease for a weighted-sum KD loss is -η times a quadratic form in λ. Second, the abstract's conclusion—that λ should be dynamically adjusted—does not follow from that derivation. Nothing in the algebra compares a dynamic schedule to a fixed λ, and the paper itself defers the algorithm to future work.\n\nCredit where due: the authors do not overclaim empirical support; they say explicitly this is a Work in Process and the Discussion is framed as a suggestion. Fig. 2 is a reasonable way to visualize the quadratic dependence on λ. The references look normal, no self-citation problem.\n\nThe soft spots are real but not obscure. The main one is the entailment: Eq. (14) shows the magnitude of the one-step decrease varies with λ, but a fixed λ also decreases the loss. To get 'dynamic adjustment is required,' you need an objective (maximize per-step decrease, trade off gradient conflict, hit a target convergence curve) and a comparison. None is provided. So the title and abstract overreach.\n\nSecondary: the Taylor expansion drops the O(η²) term without a small-learning-rate condition. For the learning rates used in practice, the sign of the loss difference is not guaranteed by this argument. That matters because their conclusion is explicitly conditional on 'when the loss is decreasing.' It's a load-bearing gap, though not the main one.\n\nThe paper has no experiments, no algorithm, and no falsifiable prediction. It is a short position piece with one identity. I don't think it deserves referee time as a full paper. A workshop could take it as an extended abstract, provided the authors narrow the claim to 'λ affects per-step decrease' or actually run the comparison with a dynamic schedule against a fixed baseline. As it stands, I'd desk-reject or return for major revision. The citation pattern is fine and the thinking is clear, just the conclusion is unsupported.","headline":"Short WiP whose algebra is fine but whose headline conclusion doesn't follow from it; no experiments or algorithm, and the only novel-sounding claim is unearned.","tokens_in":4140,"tokens_out":2264,"would_cite":false,"duration_ms":21978,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a simple knowledge-distillation setting, the balancing parameter between distillation and classification losses should change during training rather than stay fixed.","keywords":["knowledge distillation","balancing parameter","dynamic loss weighting","gradient geometry","Taylor expansion","loss decrease","model compression","student-teacher training"],"falsifier":"Train a tiny student with two fixed quadratic losses for a single gradient step at several learning rates; if there is a learning rate and gradient configuration where the exact total loss increases even though the quadratic expression in Eq. (10) is positive, then the first-order approximation that carries the conclusion has failed.","tokens_in":3233,"feed_emoji":"⚖️","tokens_out":8394,"duration_ms":79513,"temperature":0.7,"pith_summary":"The paper tries to prove that the parameter balancing a distillation loss and a downstream classification loss in knowledge distillation should be adjusted dynamically while training, rather than held at a constant value. It derives, in a simplified setting, an expression for how much the total loss falls after one gradient step, and shows that this fall is a quadratic function of the balancing parameter with coefficients set by the current gradient magnitudes and the angle between them. Since those quantities shift during training, a fixed weight cannot be optimal at every step. If the derivation holds, it gives a theoretical justification for time-varying loss weights in knowledge distillation, which most existing recipes treat as a constant hyperparameter.","feed_headline":"Knowledge distillation's loss weight should move each step","feed_subtitle":"A Taylor-expansion argument ties the per-step loss drop to gradient geometry, making a fixed weight suboptimal.","key_machinery":"The central object is the total loss $L(\\theta_s) = \\lambda L_{\\text{dist}}(\\theta_s) + (1-\\lambda)L_{\\text{cls}}(\\theta_s)$ and its one-step update under gradient descent. The argument's workhorse is the first-order Taylor expansion of $L(\\theta_s - \\eta(g_{\\text{dist}} + g_{\\text{cls}}))$, following the stiffness-based expansion used in [13]. Expanding the squared norm of the weighted gradient sum gives $$\\$\\Delta$ L_{i+1} \\approx -\\eta\\big[\\$lambda^{2}$\\|\\nabla L_{\\text{dist}}\\|^2 + 2\\$\\lambda$(1-\\$\\lambda$)\\|\\nabla L_{\\text{dist}}\\|\\|\\nabla L_{\\text{cls}}\\|\\cos\\varphi + (1-\\$\\lambda$)^2\\|\\nabla L_{\\text{cls}}\\|^2\\big],$$ where $\\varphi$ is the angle between the two gradient vectors. This quadratic-in-$\\lambda$ expression is the component that carries the conclusion: the size of the loss decrease is controlled by $\\lambda$ through a quantity that changes with the gradient geometry.","core_discovery":"The paper's central claim is that in a simple knowledge-distillation setup, where a student is trained by a weighted sum of a distillation loss and a classification loss, the balancing parameter lambda should change from step to step rather than stay fixed. The claim is grounded in a first-order Taylor expansion of the total loss after one gradient step. The resulting formula expresses the one-step loss change as minus eta times a quadratic polynomial in lambda whose coefficients are the gradient norms of the two losses and the cosine of the angle between them. Because those geometric quantities evolve during training, the lambda that gives a fast loss decrease at one step is not generally the best at the next. The paper therefore concludes that a dynamic lambda, chosen from the current gradients and the desired rate of learning, is needed for effective knowledge distillation.","pith_inferences":["Beyond the paper's explicit claim, the quadratic form in lambda points toward an explicit per-step optimizer the paper does not derive: setting the derivative of the bracket to zero gives the lambda that maximizes one-step loss decrease from the current gradient norms and cos(phi).","A testable extension beyond the paper's scope is to run the same student on the same data with a fixed lambda and with a per-step lambda chosen from this formula, then compare the loss curves; the dynamic schedule should dominate at every step if the approximation is reliable.","The same quadratic structure may apply to any two-term training objective, not just knowledge distillation, so dynamic weighting might be a general multi-task training principle rather than a distillation-specific fix."],"forward_implications":["A fixed lambda is, on this account, a compromise: it will be too aggressive at some steps and too timid at others as gradient magnitudes and the angle between the two losses evolve.","The lambda that gives the largest one-step loss decrease is the one that maximizes the quadratic bracket in the derived expression, so per-step lambda selection reduces to a scalar optimization once the two gradients are known.","When the distillation and classification gradients point in the same direction, the cross term contributes positively to the loss drop, so weighting both terms matters differently than when the gradients oppose each other.","Comparing fixed-lambda schedules only by final accuracy can hide their different loss-decrease paths, since each lambda produces a different per-step loss curve."],"supporting_citations":[{"why":"Introduces knowledge distillation and the weighted two-loss objective this paper analyzes.","marker":"[6]"},{"why":"One of the prior studies the paper cites for the empirical and theoretical point that distillation loss should outweigh the task loss.","marker":"[8]"},{"why":"Supports the same premise by characterizing distillation as partial variance reduction.","marker":"[9]"},{"why":"Gradient-surgery work the paper groups with evidence that balancing the two loss terms matters.","marker":"[10]"},{"why":"Shows that relying only on the distillation loss can hurt performance, motivating the need to regulate lambda.","marker":"[11]"},{"why":"Proposes methods for choosing an optimal balancing parameter, the line of work this paper tries to justify theoretically.","marker":"[12]"},{"why":"Supplies the Taylor-expansion treatment of the loss after a gradient step that the derivation follows.","marker":"[13]"}],"fun_headline_variants":["Distillation loss weight should track gradients","Dynamic lambda trumps fixed in distillation","One-step math: adjust KD's balancing parameter","Gradient geometry dictates per-step KD weight","Fixed KD weight is suboptimal—update it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The calculation keeps only the first-order term of the loss change and drops the remainder without stating when the learning rate is small enough for that remainder to be negligible; if the remainder is not small, the predicted sign and size of the loss decrease can be wrong.","fun_headline_variants_meta":{"raw":{"variants":["Distillation loss weight should track gradients","Dynamic lambda trumps fixed in distillation","One-step math: adjust KD's balancing parameter","Gradient geometry dictates per-step KD weight","Fixed KD weight is suboptimal—update it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1283,"prompt_tokens":853,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":469,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":469,"tokens_out":430,"duration_ms":4754,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:57:40.134493+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a tiny student with two fixed quadratic losses for a single gradient step at several learning rates; if there is a learning rate and gradient configuration where the exact total loss increases even though the quadratic expression in Eq. (10) is positive, then the first-order approximation that carries the conclusion has failed.","supporting_citations":[{"cited_title":"Towards understanding knowl- edge distillation","cited_arxiv_id":null,"evidence_quote":"One of the prior studies the paper cites for the empirical and theoretical point that distillation loss should outweigh the task loss."},{"cited_title":"Knowledge distilla- tion performs partial variance reduction","cited_arxiv_id":null,"evidence_quote":"Supports the same premise by characterizing distillation as partial variance reduction."},{"cited_title":"Gradient surgery for multi-task learning","cited_arxiv_id":null,"evidence_quote":"Gradient-surgery work the paper groups with evidence that balancing the two loss terms matters."},{"cited_title":"Can students beyond the teacher? distilling knowledge from teacher’s bias","cited_arxiv_id":null,"evidence_quote":"Shows that relying only on the distillation loss can hurt performance, motivating the need to regulate lambda."},{"cited_title":"Even your teacher needs guidance: Ground-truth targets dampen regularization imposed by self-distillation","cited_arxiv_id":null,"evidence_quote":"Proposes methods for choosing an optimal balancing parameter, the line of work this paper tries to justify theoretically."}],"review_version":1}