{"id":"21b6f358-4482-484f-a71e-7221854dfc62","arxiv_id":"2607.26246","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A strong LLM is improved by distilling from the logit difference of two weaker models instead of from a stronger teacher.","lead":"Weak-to-strong on-policy distillation (W2S-OPD) trains a stronger LLM not from a single weaker teacher, but from the logit-space difference between two weaker models. A reader would care because it offers a way for frontier models to keep improving when no stronger teacher exists.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing OPD baselines in the pure weak-to-strong settings (Tables 3–4) leave the central attribution unestablished.","rationale":"The paper presents a clear, plausible mechanism and includes a controlled OPD baseline in the pre-RL/post-RL setting, where W2S-OPD improves over OPD by 5.3 average points on math. The theoretical derivation in Appendix B (Eqs. 8–10) is internally consistent: the proxy teacher is an exponential tilting of the base model, and the reverse-KL objective is equivalent to reward maximization with a KL trust region. However, the strongest and most novel claim—improvement from sources that are all weaker than the student—is only demonstrated against the student's own base, not against the most natural alternative: directly applying OPD to the positive weak model. This omission matters because the method's central assumption is that the logit difference z+ - z- isolates a transferable direction that plain imitation of the positive model would not provide. Without this control, the observed gains in Tables 3 and 4 could be due to the weak positive model's signal alone, to an entropy/smoothing effect from the α-weighted mixture, or to selection of α on the evaluation benchmarks. The reader's verdict of CONDITIONAL is therefore appropriate; the concern strengthens the condition rather than overturning the verdict. A single targeted ablation would resolve whether the contrast subtraction is load-bearing.","tokens_in":24726,"tokens_out":5964,"duration_ms":60495,"concrete_test":"Run the smaller/larger setting with plain OPD from the Qwen3-4B positive model alone, using the identical 100-step training config, top-K=32, prompts, and data as Table 4, and compare against W2S-OPD. If plain OPD attains the same or larger math/code gains over the 8B student base, then the contrast-pair subtraction is not the active ingredient and the central claim would need to be revised; if plain OPD fails to improve (or degrades), the weak-to-strong contrast explanation is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim is that W2S-OPD 'keeps improving the student even when every supervision source is weaker' and 'outperforms OPD.' The OPD comparison, however, is only reported in the pre-RL/post-RL setting (Table 2), where the positive model is not actually weaker than the student: Qwen3-4B-RL scores 48.8 vs 17.0 on math and 61.5 vs 57.6 on code, so it is a strong domain expert despite being smaller. In the two settings where the positive model is genuinely weaker (smaller/larger, Table 4; correct/wrong hints, Table 3), the only reported baseline is the student's own base; no OPD-from-the-positive-model baseline is shown. Consequently, the observed improvement is not yet attributable to the contrast-pair subtraction z+ - z-: it could come from plain on-policy distillation toward the weak positive model, from a distributional-smoothing effect, or from the α hyperparameter tuned on the evaluation benchmarks. The claimed 'capability direction' mechanism requires a control that separates the effect of the contrast from the effect of the positive model alone. Without it, the paper's own design goal—distilling a direction rather than imitating a weak supervisor—is not empirically supported in the pure weak-to-strong regime.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces W2S-OPD, a method to distill a strong student LLM from one or more weaker models. A proxy teacher is synthesized in logit space as softmax(z_base + α(z_+ − z_−)), where the contrast pair (m_+, m_−) is intended to isolate a capability direction; the student is trained by on-policy reverse-KL distillation against this proxy teacher. Three contrast instantiations are evaluated: a post-RL 4B expert against its pre-RL initialization, two off-the-shelf base models (4B and 0.6B), and a single 4B model conditioned on correct versus wrong hints. The student is Qwen3-8B. The authors report gains over the student base on math and code benchmarks, claim W2S-OPD outperforms OPD, and present token-level analyses of which reasoning episodes are reinforced.","tokens_in":25058,"tokens_out":3428,"duration_ms":32950,"significance":"If the empirical claims hold, the idea is significant: it offers a way to improve frontier models without a stronger teacher, using direction/subtraction in logit space rather than direct imitation. The formal connection in Appendix B (exponential tilting / product-of-experts view) is elegant and makes the method easy to adopt. The three contrast instantiations are practical and well motivated, and the paper includes useful analysis of learning signals and runtime overhead. However, the central attribution is currently under-supported: the OPD baseline is only reported in the pre-RL/post-RL setting, where the positive teacher is not actually weaker than the student, and the pure weak-to-strong settings lack the OPD control that would isolate the effect of the contrast subtraction. In addition, the main results rely on α tuned on the evaluation benchmarks without multiple seeds or error bars, making the claimed superiority to OPD and the 'weak-to-strong' interpretation fragile.","major_comments":[{"comment":"In the two genuinely weak-to-strong settings (smaller/larger and correct/wrong hints), the only baseline is the student's base model; no OPD-from-the-positive-model baseline is reported. Without that control, the observed improvement cannot be attributed to the contrast subtraction z_+ − z_−. It could equally come from plain on-policy distillation toward the weak positive model, from a distributional-smoothing effect, or from the α hyperparameter. Please add an OPD baseline with the positive model (Qwen3-4B) as teacher under the identical configuration for these settings, and also an α=0 control.","section":"§4.2, Tables 3 and 4"},{"comment":"The main results are reported after selecting α on the same evaluation benchmarks (Figure 4), and Table 7 fixes α=1.0 for math and 0.75 for code. Figure 4 shows that the optimal α varies across benchmarks (e.g., on LiveCodeBench α=0.75 gives 26.8 while α=1.25 gives 19.7, below the OPD reference of 22.0). This makes the headline gains optimistic and does not support the abstract's claim of consistent outperformance. Report results with a validation-selected α, or provide a robustness table across α values and multiple seeds/error bars for at least the main settings.","section":"§4.3.1, Figure 4 and Table 7"},{"comment":"The proxy teacher is anchored at the student's own base logits, so the target is partially defined by the student. The paper's interpretation that the logit difference 'isolates a capability direction' is not independently tested; the method could reduce to a tuned interpolation between the student and a weak model. Provide a control that separates the contrast from the anchor, e.g., α=0, a negative-direction control, or a random/permuted logit difference. Also, the abstract says 'keeps improving the student even when every supervision source is weaker,' but the only setting with an OPD comparison (pre-RL/post-RL, Table 2) has a positive model that is much stronger on math (48.8 vs 17.0), so it is smaller but not weaker. The pure weak settings have no OPD baseline, leaving the central claim empirically unverified.","section":"§3.1, Eq. (2) and Abstract"},{"comment":"The claim that 'W2S-OPD enables the student to surpass the domain teacher' is only demonstrated in the pre-RL/post-RL setting, where the positive model is a strong domain expert (Qwen3-4B-RL: 48.8 math vs student 17.0). That is not a weak-to-strong transfer in the sense claimed in the abstract and introduction. Please clearly separate the strong-expert setting from the genuinely weak settings, and state that the OPD comparison applies only to the former. This is not a fatal issue if the weak-setting results are supported with proper baselines, but the current presentation overstates the scope of the finding.","section":"§4.2 and Table 2"}],"minor_comments":[{"comment":"Minor grammar/wording issues: 'which token-level supervision efficiently' appears to be missing a verb; the sentence should be revised.","section":"§3.2, third bullet"},{"comment":"'W2S-OPD address both challenges' should be 'addresses'.","section":"§3.1, after Eq. (3)"},{"comment":"The caption contains an incomplete phrase ('divide and ...') and would benefit from clearer labels for the logit-offset path.","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The central idea is promising and the formalization in Appendix B is a strength. The main empirical gap is the missing OPD baseline in the genuinely weak-to-strong settings (Tables 3–4), which is load-bearing for the paper's central claim. The α-tuning-on-test-set issue compounds this. I would be willing to accept after these are addressed experimentally; the scope of the revision is more than minor because the missing comparisons are essential to the stated contribution."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the method is simple and the empirical package is mostly honest, but the paper's central claim—improving a strong student from purely weaker sources—is the one part that lacks the key control. Read it if you work on distillation, but don't take the abstract's framing at face value.\n\nWhat's new: applying decoding-time logit arithmetic (z_base + α(z+ − z−), the DExperts/Tuning-by-Proxy recipe) as a frozen proxy teacher for on-policy distillation, with three instantiations of the contrast pair. That is a natural extension, not a conceptual leap, and the authors are upfront about the lineage. The math in Appendix B is clean: the proxy teacher is an exponential tilting of the student's base with a KL trust region, and the compositional view (product of experts) is helpful. Implementation details are thorough: top-K OPD, runtime overhead, OOD transfer.\n\nStrengths: in the pre-RL/post-RL setting, W2S-OPD beats OPD under identical configurations on all benchmarks, with a solid margin on math. The gains are consistent across seven benchmarks, and the token-level episode analysis (Table 6) is a genuine attempt to characterize what each contrast reinforces. The case studies show real qualitative differences.\n\nSoft spots: the OPD baseline only appears in the setting where the 'weak' positive model is actually a strong domain expert (Qwen3-4B-RL: 48.8 avg math vs 17.0 for the 8B base). In the two settings where every supervision source is genuinely weaker—smaller/larger and correct/wrong hints—there is no OPD-from-the-positive-model baseline. Without it, the observed improvement is not attributable to the contrast subtraction; it could come from plain on-policy distillation toward the positive model, from a distributional smoothing effect, or from α tuned on the eval benchmarks. Figure 4 shows α selected per benchmark, Table 7 gives the chosen values, and there are no error bars or multiple seeds. The abstract's 'surpass the teacher' is also false on code. These are addressable rather than fatal, but they shift the paper's contribution from 'proven weak-to-strong transfer' to a 'promising recipe that needs a control.'\n\nWho it's for: people working on LLM post-training, distillation, and weak supervision. It deserves a serious referee; the idea is clear and the evidence is substantial enough to merit the extra experiments.\n\nRecommendation: send to peer review, but require the missing OPD baselines in the pure weak-to-strong settings and a report of variance across seeds before acceptance.","headline":"A clean, likeable distillation paper whose headline weak-to-strong claim is undercut by missing OPD baselines in the very settings where all sources are genuinely weaker.","tokens_in":25512,"tokens_out":3297,"would_cite":true,"duration_ms":31135,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a strong LLM can improve by distilling from the logit difference between two weaker models, even when both are less capable than the student.","keywords":["weak-to-strong","on-policy distillation","proxy teacher","logit arithmetic","LLM reasoning","knowledge distillation","reverse KL","math reasoning"],"falsifier":"A randomized-direction control: take the same positive and negative models but replace z+ − z− at every token with a random vector of identical norm, and distill that. If the student improves as much as with the true direction, the direction itself is not the carrier; if it does not, the identity of the contrast direction is causal.","tokens_in":24665,"feed_emoji":"🧠","tokens_out":4850,"duration_ms":47356,"temperature":0.7,"pith_summary":"This paper tries to establish that weak-to-strong transfer need not imitate a weak teacher. Instead, the difference between two weak models' pre-softmax scores (logits) can be treated as a capability direction, added to the student's own base logits, and then distilled on the student's own rollouts. On math and code benchmarks, this proxy-teacher distillation improves an 8B student over direct on-policy distillation and, in some settings, above the domain expert itself. The claim matters because it offers a path for frontier models to keep improving when no stronger teacher exists.","feed_headline":"Weaker models can teach a stronger AI to improve","feed_subtitle":"Subtracting two weak models' logits isolates a capability direction that, added to the student, beats direct distillation on math and code.","key_machinery":"The load-bearing object is the proxy teacher, a product-of-experts composition in logit space: πT,α(a|s) ∝ πbase(a|s) · (π+(a|s) / π−(a|s))^α. The ratio π+/π− is the capability reward carried by the contrast pair; subtracting logits cancels shared weakness; α controls how far the resulting distribution may move from the student's base. Distillation minimizes reverse KL on student-generated prefixes, so the teacher corrects exactly the states the student visits. A multi-teacher extension sums several such directions in one run.","core_discovery":"The central claim is that the logit difference z+(s) − z−(s) between a positive and a negative weak model cancels what the two share and preserves the direction along which the positive improves over the negative. Adding this direction with a coefficient α to the student's base logits yields a proxy teacher πT,α = softmax(zbase + α(z+ − z−)) that is simultaneously more capable in the target skill and distributionally close to the student. The student minimizes per-token reverse KL toward this proxy on its own rollouts. The paper argues this isolates and transfers capability rather than imitating a weak supervisor, and reports gains across three contrast types: pre-RL vs post-RL, larger vs sm","pith_inferences":["The same recipe could mine other contrast pairs — before/after safety tuning, generalist vs specialist, native vs translated — as long as their logit difference is directionally meaningful, making weak-to-strong a general post-training primitive.","Since the method requires only logits, it could be run with the student's own checkpoints from different training steps as the positive/negative pair, turning self-distillation into a weak-to-strong signal without any extra model.","A testable extension is per-token or per-example α: the paper uses a single scalar and shows performance peaks at moderate values, so an adaptive schedule might push gains further."],"forward_implications":["A frontier model can keep improving without any stronger teacher, as long as a contrast pair with a separable capability gap exists.","The method transfers RL-acquired skills to a large student without ever running RL at the student's scale.","Contrast directions from different sources (RL, scale, hints) can be merged in a single distillation run by adding their logit differences.","Out-of-domain evaluation suggests the transferred skill does not erode general ability; in the paper's experiments it improves the student on scientific reasoning and instruction following."],"fun_headline_variants":["Weak models teach a stronger AI to improve","Subtract weak logits to add skill to a strong AI","Weak contrasts create proxy teachers for strong AI","Distill from weak pairs to beat weak teachers","On-policy weak-to-strong distillation via logit subtraction"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The assumption that carries the paper is that the logit difference between two weak models is a clean, transferable capability direction; if it mostly encodes style, format, or benchmark artifacts, the proxy teacher would just be a tuned interpolation and the weak-to-strong story collapses.","fun_headline_variants_meta":{"raw":{"variants":["Weak models teach a stronger AI to improve","Subtract weak logits to add skill to a strong AI","Weak contrasts create proxy teachers for strong AI","Distill from weak pairs to beat weak teachers","On-policy weak-to-strong distillation via logit subtraction"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000234,"raw_usage":{"total_tokens":1406,"prompt_tokens":890,"completion_tokens":516,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":441}},"tokens_in":634,"tokens_out":516,"duration_ms":5717,"temperature":1.0,"reasoning_tokens":441,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T00:20:54.460422+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A randomized-direction control: take the same positive and negative models but replace z+ − z− at every token with a random vector of identical norm, and distill that. If the student improves as much as with the true direction, the direction itself is not the carrier; if it does not, the identity of the contrast direction is causal.","supporting_citations":[],"review_version":1}