{"id":"1ae1ae00-ae8e-4ea2-8971-d826cb9dda3f","arxiv_id":"2608.01263","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"FP-OPD projects the teacher-student prediction gap onto a student visual tangent space before distillation, improving compact vision-language models over standard on-policy distillation.","lead":"This paper proposes a distillation method that lets a large vision-language teacher model teach a smaller student only the corrections the student can actually represent from its visual input, instead of pushing it to match every teacher prediction. On math and general multimodal benchmarks, the method beats standard on-policy distillation in 8B-to-2B transfers and improves most benchmarks in a 32B-to-8B transfer.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FP-OPD's reported gains are not separated from target-strength scaling; the paper's own α=0.5 result and the random-probe ablation leave open that a scalar-scaled target with matched norm reproduces the improvement.","rationale":"The reader's verdict is CONDITIONAL, and the condition is appropriate: the paper lacks reproducibility details (the literal 'Appendix??' placeholder, no error bars, no released code), and the hand-chosen probe basis is a methodological risk. My stress-test identifies a distinct but related risk: the paper does not control for target strength. The projection always reduces the Fisher norm, so the improvement over OPD could be a target-scaling effect. The paper's own Figure 1c demonstrates that a weaker target beats the full teacher target in a one-epoch experiment, which makes this confounding especially plausible. The random-probe ablation, where a random 4-dimensional visual subspace preserves most of the improvement, suggests the spatial structure of the probes is not critical, weakening the 'locally realizable visual correction' interpretation. However, the Euclidean-projection ablation (42.48, far below base) shows that the choice of metric matters enormously, so the result is not a trivial scaling artifact. The decisive missing experiment is an α-scaled target with matched norm. Because this test can settle the concern, the correct verdict remains CONDITIONAL rather than ACCEPT; since the reader already assigned CONDITIONAL, I mark the verdict as UNCHANGED. In good faith, the method is plausible and the empirical gains are consistent across settings, but the mechanism is not yet established.","tokens_in":11164,"tokens_out":7695,"duration_ms":81233,"concrete_test":"Run the 8B-to-2B setting (Qwen3-VL-8B-Instruct teacher, 2B student) with target q_α = softmax(stopgrad(ℓθ) + α g) for α values spanning 0.2 to 0.8, using identical data, schedule, and evaluation as the FP-OPD run. For each α, compute the average Fisher norm of the applied correction and compare it with the norm of bĝ_R from a matched FP-OPD run. If the best α-scaled target achieves an average score at least as high as 51.27, then the reported FP-OPD gains are not evidence for the local-reachability mechanism. If FP-OPD outperforms the best α-scaled baseline, the central claim is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that Fisher projection onto a perturbation-estimated visual tangent space explains the 1.60-point average improvement over standard OPD. However, the paper never compares against a scalar-scaled target. Its own target-interpolation study (Figure 1c) shows that the complete teacher target (α=1) is harder to fit and worse downstream than an intermediate target (α=0.5). FP-OPD always produces a projected correction bĝ_R whose Fisher norm is no larger than the full gap ‖g‖ (Pythagorean theorem for orthogonal projection), so it is a form of target weakening. Unless the specific subspace direction matters beyond the norm reduction, the gains could be reproduced by q_α = softmax(stopgrad(ℓθ) + α g) with α tuned to a comparable norm. The ablation in Table 3 partially supports this worry: random visual probes with Fisher projection reach 50.90 versus 51.27 with structured probes, only 0.37 points apart, while both are far above the 49.67 OPD baseline. A random 4-dimensional visual subspace captures most of the gain, which suggests that the precise 'locally realizable' direction is not the dominant factor. The unresolved 'Appendix??' and the absence of error bars further prevent verification, but the missing α-scaled baseline is the decisive scientific gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Fisher-Projected On-Policy Distillation (FP-OPD), an on-policy distillation variant for vision--language models that replaces the complete teacher distribution p_T with a capacity-aware target q_R. The target is built by writing the OPD teacher target as softmax(ell_theta + g), estimating a student visual tangent space via finite-difference perturbations of visual features, and projecting the centered teacher--student gap g onto that space under the Fisher metric. Experiments on Qwen3-VL 8B-to-2B and 32B-to-8B distillation over seven benchmarks report an average improvement of +1.60 points over standard OPD in the main 8B-to-2B setting, with ablations over the finite-difference step, the number of probes, and the projection geometry.","tokens_in":11552,"tokens_out":9411,"duration_ms":118384,"significance":"The core idea is appealing and clearly presented: instead of uniformly weakening the teacher, keep only the component of the teacher correction that the student's visual representation can locally realize. The algebraic decomposition in Eqs. (6)--(7) and the local reverse-KL/Fisher justification in Eq. (12) are correct, and the experimental results are internally consistent. The paper also deserves credit for a relatively clean ablation chain that isolates the finite-difference step, the number of probes, and the projection geometry. However, the central mechanistic claim--that the specific student-supported visual subspace is what matters--is not yet supported, because no comparison is made against a scalar-scaled target with matched norm, and the random-probe ablation in Table 3 suggests that most of the gain may come from norm reduction rather than subspace direction. If the matched-norm control confirms the subspace story, this would be a solid contribution to distillation for multimodal models; as it stands, the mechanism is underdetermined.","major_comments":[{"comment":"The paper does not compare FP-OPD against a scalar-scaled target of the form q_alpha = softmax(stopgrad(ell_theta) + alpha g) with alpha chosen to match the Fisher norm of the projected correction bhat_g_R. Since the Fisher projection is norm-reducing by the Pythagorean theorem applied to Eq. (18), the reported gains are consistent with simple target weakening. Indeed, the paper's own interpolation experiment in Figure 1(c) shows that alpha = 0.5 is both easier to fit and better on MathVista Geometry than alpha = 1. To establish that the direction of the correction, rather than only its norm, is what matters, the authors should include this matched-norm scalar baseline. The random-probe ablation in Table 3 amplifies the concern: Fisher + random probes reaches 50.90 versus 51.27 for structured visual probes, only 0.37 points apart and 1.23 points above the standard OPD baseline of 49.67, suggesting that the specific visual tangent-space structure is not the dominant factor.","section":"Section 4.3, Table 3; Section 3.4, Eqs. (13)--(18); Figure 1(c)"},{"comment":"The sentence \"Complete training, method, and evaluation configurations are provided in Appendix??\" references a non-existent appendix. Without these details, the empirical claims cannot be reproduced or even fully checked. The appendix must be included, or the reference must be removed and the details provided in a supplementary document, before the manuscript is in publishable form.","section":"Section 4.1, Training paragraph"},{"comment":"No error bars, number of seeds, or significance tests are reported. Several headline differences are small (for example, +0.17 on HallusionBench and +0.30 on HallusionBench in Table 1, and -0.53 and -0.21 in the 32B-to-8B setting), and the claims of \"consistent gains\" and of not sacrificing general performance depend on treating these as meaningful. Please report standard deviations over at least three runs, or otherwise provide an uncertainty estimate, for the main results and ablations.","section":"Tables 1--3"}],"minor_comments":[{"comment":"There is a missing space in \"proposeFisher-Projected\" in the abstract.","section":"Abstract"},{"comment":"The centering operator display is visually confusing because \"a - <a,1>_p 1\" is written on the same line as the definition; use a clearer display with braces to indicate that the scalar is subtracted as a vector of ones.","section":"Section 3.1, Eq. (5)"},{"comment":"The metric \"AVG@8\" is used throughout but not defined; state explicitly whether it is the average accuracy over eight sampled rollouts and give the sampling protocol.","section":"Section 4.1, Evaluation paragraph"},{"comment":"The axes of Figure 3 are not labeled; specify what is on the x-axis (steps) and y-axis for gap energy and target-shift JS.","section":"Figure 3"},{"comment":"The phrase \"the first and last 32 updates\" should specify the step range and whether these are gradient updates or sampled-batch evaluations.","section":"Section 4.4, Mechanism Analysis"},{"comment":"The We-Math reference contains a duplicated name \"MinhuiWu MinhuiWu\"; this should be corrected to a single author name.","section":"References, We-Math entry"}],"recommendation":"major_revision","confidential_remarks":"The manuscript appears to be an incomplete preprint: the Appendix?? reference indicates a missing appendix, and the central mechanism claim lacks a matched-norm scalar-target control. Both issues are concrete and fixable. I recommend major revision rather than rejection because the core idea and the derived objective are sound, and the requested experiments are well defined. If the authors add the scalar-scaled baseline and the appendix, and if the baseline does not reproduce the gains, I would be willing to support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: FP-OPD is a real idea, cleanly formulated, and the math checks out. The paper replaces the full teacher target in on-policy distillation with a target projected onto a student visual tangent space under the Fisher metric. That is new relative to the OPD literature, and the local reverse-KL expansion and the centering identity are correct. The reported gains are consistent across seven benchmarks and two model scales, which is more than most distillation papers show. Credit where it is due: this is a solid, well-motivated technique, not a repackaging of existing tricks.\n\nThe soft spots are real but not fatal. The biggest issue, which the stress-test note nails, is the missing scalar-scaled baseline. The paper shows that α=0.5 beats α=1, and FP-OPD always weakens the target by projection. Without comparing against q_α = softmax(ℓθ + αg) with matched norm, we do not know if the specific subspace direction matters or whether any moderate target-strength reduction reproduces the gains. The random-probe ablation strengthens that worry: random probes with Fisher projection reach 50.90 versus 51.27 with structured probes, only 0.37 apart. So the precise 'visual tangent space' may be doing little work beyond providing a low-dimensional, norm-reducing subspace. The Fisher versus Euclidean contrast (8.79 points) shows the metric matters, but that is a separate claim from the subspace direction being right.\n\nThe other weaknesses are the usual suspects: hyperparameters (εfd, K, γ, ρ) are tuned on the evaluation benchmarks and reported without error bars or multi-seed statistics, and the unresolved 'Appendix??' is an unacceptable hole in a paper that claims reproducibility. These are fixable but need to be addressed before the central claim is accepted.\n\nWho is this for? Anyone working on distillation for VLMs or on capacity-aware training targets. It is a worthwhile contribution that deserves a serious referee, but the referee should push on the scalar-scaling baseline and the probe-basis sensitivity. I would not desk-reject it, and I would want to see the revision.\n\nRecommendation: engage with it, send it to review, and insist on the missing baseline and proper reproducibility material.","headline":"A clean, novel distillation method with a correct algebraic core, but the empirical claim is undercut by a missing scalar-scaled baseline and weak probe ablations; deserves serious review after revision.","tokens_in":11987,"tokens_out":1100,"would_cite":true,"duration_ms":73716,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Replacing the full teacher distillation target with a Fisher projection of the teacher–student gap onto the student's visual tangent space improves all seven vision-language benchmarks in 8B-to-2B transfer, by 2.77 points on average over…","keywords":["on-policy distillation","Fisher projection","vision-language models","capacity-aware distillation target","reverse-KL distillation","visual tangent space","multimodal reasoning","knowledge distillation"],"falsifier":"Take a small benchmark where the correct answer depends on a fine visual mark at the image center, far from all four boundary bands, and first verify that the small student can learn that correction under standard OPD. If FP-OPD then fails to transfer the same correction — because the center-only change is orthogonal to all four probes and is discarded — while standard OPD succeeds, the four-band tangent space does not contain the student-supported corrections and the central claim collapses.","tokens_in":10913,"feed_emoji":"🎯","tokens_out":10946,"duration_ms":109728,"temperature":0.7,"pith_summary":"On-policy distillation supervises a compact student at prefixes sampled from its own generation, but it still asks the student to match the complete next-token distribution of the teacher. In vision-language reasoning, part of the teacher's correction encodes visual distinctions the student's representation cannot express, so the full target contains unreachable supervision. This paper's target-scaling study supports that diagnosis: moving the target all the way to the teacher makes the prescribed shift harder to fit and degrades downstream reasoning. The paper's proposal, Fisher-Projected On-Policy Distillation (FP-OPD), estimates the student's local visual tangent space with perturbed visual features and projects the teacher-student gap onto that space under the Fisher metric, producing a capacity-aware target. In 8B-to-2B distillation FP-OPD improves all seven benchmarks and raises the average by 2.77 points over the base student and 1.60 points over standard OPD, which the paper takes as evidence that locally realizable teacher corrections are the right target for compact vision-language models.","feed_headline":"Filtering what a student can't see lifts distillation by 2.77","feed_subtitle":"On seven benchmarks, distilling only reachable teacher corrections beats full-target on-policy distillation.","key_machinery":"The load-bearing object is the Fisher projection of the centered teacher–student log-probability gap onto a finite-difference approximation of the student's local visual tangent space. The student's visual embeddings are perturbed along four fixed binary spatial bands (top, bottom, left, right, each spanning 25% of the grid axis) to give response directions $d_k$, and their span approximates the first-order output changes the student can express through its current visual state. Projection uses the Fisher inner product $\\langle a, b\\rangle_p = \\sum_{v\\in V} p(v) a_v b_v$, which is the geometry induced by the OPD loss itself because the local expansion of reverse KL is $\\tfrac{1}{2}\\langle \\delta,\\delta\\rangle_p + o(\\|\\delta\\|^2)$. A ridge-stabilized solve produces $\\hat{g}_R$, and the reachable target $q_R = \\mathrm{softmax}(\\mathrm{stopgrad}(\\ell_\\theta) + \\hat{g}_R)$ anchors at the detached student distribution; the loss keeps the standard OPD form $\\mathrm{KL}(p_\\theta \\,\\|\\, q_R)$ with full-vocabulary reverse KL on student-generated trajectories.","core_discovery":"The central claim is that the appropriate distillation target for a compact vision-language student is not the complete teacher distribution but the component of the teacher correction contained in the student's locally reachable output space. Writing the teacher target as $p_T = \\mathrm{softmax}(\\ell_\\theta + g)$ with $g$ the centered teacher–student log-probability gap, FP-OPD replaces $g$ by its Fisher projection $\\hat{g}_R$ onto a tangent space estimated from finite-difference responses to four spatial perturbations of the student's visual embeddings, and optimizes full-vocabulary reverse KL against $q_R = \\mathrm{softmax}(\\mathrm{stopgrad}(\\ell_\\theta) + \\hat{g}_R)$. If the gap lies entirely in the estimated tangent space the target coincides with the teacher, and if it is orthogonal the target produces no update at all. The authors report that this capacity-aware target improves all seven benchmarks in 8B-to-2B distillation and five of seven benchmarks over standard OPD in 32B-to-8B distillation, with gains that persist under greedy decoding, and conclude that locally realizable teacher corrections are more effective supervision for compact vision-language models.","pith_inferences":["My inference: the reachability principle should extend beyond vision — any capacity gap between teacher and student (tool use, long-context reasoning, a second language) creates teacher corrections outside the student's local response space, and projecting the correction under the loss's own metric is a general recipe, not a vision-specific fix.","My inference: the four fixed boundary bands are a hand-chosen, domain-specific proxy for the tangent space, so a data-driven or adaptive perturbation basis is the natural stress test; whether such a basis improves results would reveal whether the specific spatial bands are essential or merely convenient.","My inference: FP-OPD should show its largest edge exactly where the teacher's advantage is spatially localizable (diagram labels, angles, marks near image edges) and its smallest edge where the decisive visual detail is central or texture-based; a per-task breakdown against the image region that carries the evidence would test this.","My inference: if the residual $g_\\perp$ is truly unlearnable by the current student, it becomes a natural training signal for a later, larger stage of the same model family, since it is by construction orthogonal to what the smaller student can absorb."],"forward_implications":["With the 8B-Instruct teacher, FP-OPD beats standard OPD on all seven benchmarks for the 2B student, raising the average by 1.60 points over OPD and 2.77 points over the base student.","FP-OPD compounds with a stronger teacher: distilling from an 8B Geo3K-GRPO teacher yields a 52.78 average for the 2B student, the best in every column and 0.73 points above standard OPD with the same teacher.","The gains are not sampling artifacts: under greedy decoding the 32B-to-8B student reaches 63.95 on average, 1.12 points above standard OPD, improving six of seven benchmarks.","The projection metric carries most of the benefit: replacing the Fisher metric with Euclidean projection collapses the 8B-to-2B average by 8.79 points, whereas swapping the four spatial probes for random probes costs only 0.37 points.","The student actually absorbs the projected signal during training: median gap energy falls by 43.3% and median target-shift Jensen–Shannon divergence by 35.8% over two epochs, indicating the target shrinks toward the student rather than remaining a fixed unreachable demand."],"supporting_citations":[{"why":"Introduces on-policy distillation (GKD), the training framework FP-OPD inherits: matching teacher distributions at student-visited states.","marker":"[Agarwal et al., 2024]"},{"why":"Supplies the Fisher metric as the natural geometry of predictive distributions, which FP-OPD uses as its projection metric.","marker":"[Amari, 1998]"},{"why":"Foundational knowledge-distillation setup that FP-OPD refines from full-target matching to capacity-aware targets.","marker":"[Hinton et al., 2015]"},{"why":"The PG-OPD formulation of on-policy distillation at student-generated trajectories, serving as a direct baseline.","marker":"[Lu and Lab, 2025]"},{"why":"Connects Fisher information to trust-region and local geometry of policy optimization, supporting the use of the Fisher inner product.","marker":"[Schulman et al., 2015]"},{"why":"Documents the prefix-mismatch problem in off-policy distillation, motivating the on-policy trajectory alignment FP-OPD retains.","marker":"[Xu et al., 2025]"},{"why":"Evidence that larger capacity gaps between teacher and student can hurt distilled performance, motivating capacity-aware targets.","marker":"[Zhang et al., 2025]"},{"why":"Shows a teacher must adapt its knowledge to the student, the qualitative principle FP-OPD formalizes with projection.","marker":"[Qian et al., 2025]"}],"fun_headline_variants":["Fisher-projected distillation distills only what the student can see","FP-OPD: Distill only locally realizable teacher corrections","Project teacher corrections onto student's visual space to lift distillation","Distill what the student can represent, not the full teacher","Capacity-aware distillation: only reachable corrections, better results"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that four fixed spatial bands of visual tokens (top, bottom, left, right, each a quarter of the image grid) span every teacher correction the student could actually learn, so that whatever the Fisher projection discards is genuinely unreachable.","fun_headline_variants_meta":{"raw":{"variants":["Fisher-projected distillation distills only what the student can see","FP-OPD: Distill only locally realizable teacher corrections","Project teacher corrections onto student's visual space to lift distillation","Distill what the student can represent, not the full teacher","Capacity-aware distillation: only reachable corrections, better results"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000575,"raw_usage":{"total_tokens":2766,"prompt_tokens":1051,"completion_tokens":1715,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":1630}},"tokens_in":667,"tokens_out":1715,"duration_ms":13366,"temperature":1.0,"reasoning_tokens":1630,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:59:50.560352+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a small benchmark where the correct answer depends on a fine visual mark at the image center, far from all four boundary bands, and first verify that the small student can learn that correction under standard OPD. If FP-OPD then fails to transfer the same correction — because the center-only change is orthogonal to all four probes and is discarded — while standard OPD succeeds, the four-band tangent space does not contain the student-supported corrections and the central claim collapses.","supporting_citations":[],"review_version":2}