Computer Vision and Pattern Recognition · arXiv:2610.12355 v2 · submitted 8 Oct 2026
Distilling Routed 3D Privilege for Spatial Reasoning in Vision-Language Models
The paper says
“Spatial reasoning remains a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence.”
more of the abstract
Existing remedies either inject 3D into the model at inference, paying architecture and latency costs, or train with outcome rewards that supervise only the final answer. Spatial errors originate in perception: a misjudged depth or direction can be corrected only by the scene's true geometry, which the 3D-scanned sources of spatial training corpora already provide. We propose GPD (Geometry-Privileged Distillation), which makes geometric evidence the privilege in on-policy self-distillation (OPSD). For each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text and routed to the teacher alongside the reference answer; a privileged KL, applied only to incorrect trajectories, augments GRPO, and the deployed model remains RGB-only. On the 4B backbone, GPD achieves 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming both GRPO and answer-privileged OPSD across spatial reasoning benchmarks. Ablations confirm the complementarity of 3D and answer privilege, the advantage of question-conditioned routing over full-context injection, and the benefit of restricting distillation to incorrect trajectories.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 9 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A cleanly specified training-only recipe whose central claim holds up in spirit, but the unvalidated single-sample KL estimator and missing error bars keep the measured gains softer than the summary claims. This line is wrong
3 major objections · 6 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Sec. 3.3, Eqs. (4)-(6), and Fig. 6
The review did not quote this passage.
Pith says: Objection 1 of 3 · major
The distillation signal d_i,j = exp(Delta_i,j) - Delta_i,j - 1 is exactly unbiased in expectation over the student's sampled tokens (E_{t_j ~ pi_S}[d_i,j] = KL[pi_S(\cdot | t_<j, p_S) || pi_T(\cdot | t_<j, p_T)]), but its per-token value and gradient are unbounded and asymmetric: for a token the teacher prefers over the student, Delta > 0 gives a contribution growing as exp(Delta), and Eq.
more of objection 1
(6) clips only the micro-batch scalar at 1.5, so a single such token can dominate the whole batch gradient, or, when the batch average exceeds 1.5, the min() saturates and the distillation gradient for the entire micro-batch vanishes. In this setup the failure is concrete: when the reference answer is in the teacher prompt and the student has sampled a wrong answer token, the teacher's probability for the reference token can exceed the student's by orders of magnitude, so the answer-token term, not the 3D cues, becomes the dominant gradient contributor. The paper reports no diagnostics on how often the 1.5 cap is hit or what gradient norms look like, reports a single seed for every number in Tables 1-2 and Figure 4, and Figure 6 itself explains GPD's early advantage through a higher entropy peak, which is a generic exploration effect rather than a geometric one. These gaps make it impossible to distinguish the stated mechanism (routed 3D privilege) from GRPO plus a tuned KL/entropy regularizer; please add per-token gradient control or a bounded KL estimate, report clip-hit rates and gradient statistics, give multi-seed means with standard deviations, or explicitly weaken the internalization claim (b25).
The paper says, at Abstract (b4) and Conclusion (b113) vs. Tables 1-2
The review did not quote this passage.
Pith says: Objection 2 of 3 · major
The claim that GPD 'outperforms answer-privileged OPSD across spatial reasoning benchmarks' holds only for the reported averages, not benchmark-by-benchmark. In Table 2, GPD at 2B is below OPSD on MMSI (21.1 vs 21.7), GPD at 4B is below OPSD on ViewSpatial (39.8 vs 40.4) with a tie at 2B (36.8 vs 36.8), and GPD at 2B is below GRPO on SPAR (38.2 vs 38.5) and below the base model on MMSI (21.1 vs 24.3).
more of objection 2
The Abstract and Conclusion should be rephrased to the aggregate level, the regressions should be discussed in Sec. 4.2, and the large OPSD collapse on MindCube 4B (21.9, near the 21.5 base, against 33.0 for GRPO) should be examined as a possible symptom of instability in the answer-privileged baseline rather than presented only as evidence that answer privilege cannot correct perception errors.
The paper says, at Sec. 3.2 and Sec. 4.4
The review did not quote this passage.
Pith says: Objection 3 of 3 · major
The question-conditioned router r(q) is a named contribution, but the paper never specifies how 'the base VLM selects a relevant subset' - no prompt, inference procedure, or likelihood criterion is given - and its validation rests on a 1000-question VSI-590K subset plus the single 1.7-point routed-versus-full-context gap in Figure 4 (41.5 vs 39.8), which is within the noise level allowed by the missing error bars.
more of objection 3
Please specify the routing mechanism and report a routing-quality metric, for example agreement of r(q) with the task-derived modality needs shown in Figure 5, or an ablation with a random router.
6 minor objections
The paper says, at References
Pith says: minor objection 1 of 6
The reference list contains duplicate entries for the same works: Depth Anything 3 appears as Lin et al. 2025a and 2025b, and Grounded SAM as Ren et al. 2024a and 2024b; merge them into single entries.
The paper says, at Fig. 4 (left)
Pith says: minor objection 2 of 6
The bars carry no numeric labels in the reproduced figure although Sec. 4.3 quotes exact values (40.2, 40.5, 41.5, 39.8, 40.4); please annotate the bars with their values.
show 4 more
The paper says, at Sec. 4.2
Pith says: minor objection 3 of 6
The MMSI 2B results deserve one sentence of context: even GRPO regresses below the base model (19.8 vs 24.3), so the small GPD numbers on that benchmark are not obviously informative about GPD's effect.
The paper says, at Fig. 7
Pith says: minor objection 4 of 6
Figure 7 injects routed 3D text at test time into a model the paper repeatedly describes as RGB-only; please state explicitly that this is a diagnostic analysis, not part of the evaluation protocol.
The paper says, at Eq. (3)
Pith says: minor objection 5 of 6
Equation (3) writes GRPO with a PPO-style min/clip on the importance ratio, whereas the original DeepSeekMath formulation uses the unclipped ratio; please cite the source of this clipped-ratio variant so that the GRPO baseline matches a clearly identified reference.
The paper says, at General
Pith says: minor objection 6 of 6
The manuscript contains no limitations paragraph; given the dependence on estimated depth and semantic cues for MindCube, the benchmark-level regressions, and the estimator behavior discussed above, a short limitations statement would be appropriate and would help readers weigh the claims.
What would settle it
Re-run the 4B training on MindCube/SPAR/VSI with the estimator replaced by the same single-sample d but with per-token clipping: d_clip = min(exp(delta)-delta-1, C) for C in {2.0, 5.0}, including seeds and 95% bootstrapped deltas; then compare the VSI/MindCube/SPAR deltas against the manuscript's Tables 1-2. Additionally report the per-token index of max d across incorrect trajectories at step ~50 to show whether the gradient is concentrated in spikes. A stable 0.5-2 point gain across these variants keeps the claim; a swing of more than ~1 point or a collapse when C is small would mean Eq. 5's unbounded term, not the 3D cue content, is carrying the signal.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 9 Oct 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 3 major objections, 6 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Honest take: this is a well-specified training recipe with clean ablations, and the central claim is not circular. The genuinely new element is question-conditioned routing of depth, semantic, and BEV text into the teacher context, combined with error-gated token-level KL within OPSD. That specific combination is absent from the cited prior work, so novelty is real. The paper earns credit for isolating components: answer-only privilege gives +0.4, routed 3D text +0.7, both together +1.7, while full-text injection hurts and routed text beats routed images. The MindCube pattern is the most supportive internal evidence: answer-only OPSD collapses to near base (21.9 vs. 21.5) while GPD reaches 38.0. That fits the authors' claim that perception errors need geometric privilege, and it is not something the reference answer alone can explain.
The soft spots are real but proportional. The load-bearing piece is Eqs. 4-5: the per-token d = exp(delta) - delta - 1 is a single-sample reverse-KL estimate with unbounded terms, and Eq. 6 clips only the batch scalar, not the per-token values. No bias or variance analysis is given. On long multi-view trajectories, a few tokens where the teacher is much more confident than the student could dominate the gradient, so the risk that GPD reduces to GRPO plus an entropy-raising KL term is not retired. The authors do not flag this; it is a genuine gap. Second, there are no seeds or error bars while the headline deltas are 0.4 to 1.9 points. Third, the summary overstates: 'outperforms answer-privileged OPSD across benchmarks' holds on averages, not at benchmark level (OPSD wins ViewSpatial 4B and MMSI 2B). Fourth, both the routing function and the KL weight were selected on the evaluation benchmark family (VSI-590K subset and the Fig. 4 sweep), so selection is baked in. The internalization claim in Fig. 7 is indirect: a smaller test-time privilege gap is suggestive, not proof. These are limitations, not fatal flaws, and they are mostly in the reporting rather than in the method.
The citation pattern looks honest: OPSD, answer-privileged distillation, and 3D-cue injection are all cited. The duplicated Depth Anything 3 reference is cosmetic. The paper ships no code or data in the submission, only a link, so the numbers cannot be independently verified.
Who this is for: people working on distillation or RL for spatial VLMs, and embodied agents where RGB-only inference matters. It deserves a serious referee: the recipe is clear, the ablations are informative, and the question of whether geometric privilege can be distilled is worth resolving. My recommendation is to send it to peer review with the request that the authors add seeds, error bars, and a stability or variance check of the KL estimator before publication.
The strongest objection
The central claim is that question-routed 3D privilege, not the KL term itself, drives the gains. The teacher-student signal rests on Eqs. 4-5: delta = log(pi_theta(t_j | p_T)) - log(pi_theta(t_j | p_S)), and d = exp(delta) - delta - 1, a single-sample estimate of the reverse KL on the student-sampled token, with Eq. 6 clipping only the batch scalar at 1.5. The per-token term exp(delta) is unbounded: whenever the teacher assigns one token a probability several times larger than the student does, a single trajectory token injects a gradient as large as the whole micro-batch average. The paper provides no bias, variance, or stability analysis for this estimator, and reports no seeds or error bars (Sec. 4.2), so one cannot tell whether the reported +0.7/+1.9 average deltas reflect geometric signal. If the estimator is unstable on long multi-view trajectories, GPD internally reduces to GRPO plus a KL regularizer that happens to raise entropy (Fig. 6), a known confound in RL fine-tuning, not evidence that geometric privilege was internalized. The paper's own summary also overstates: Tables 1-2 show OPSD beats GPD on ViewSpatial 4B (40.4 vs 39.8) and MMSI 2B (21.7 vs 21.1), so claim b113 ('outperforms answer-privileged OPSD' across benchmarks) only holds on averages, not at benchmark level.
The reader's weakest_assumption is the unvalidated single-sample reverse-KL estimator, and I agree it is the load-bearing premise. The method has a clear logical spine: error-gated token-level KL from a question-routed geometric teacher should correct perception errors that outcome rewards cannot (Figure 3, case study), and the MindCube contrast (GPD 38.0 vs answer-only OPSD 21.9, Table 2) is the strongest supportive fact. But the estimator in Eqs. 4-5 is the only channel through which geometric privilege reaches the student, and it is not characterized. The per-token exp(delta) term is unbounded; Eq. 6 clips only the batch scalar, so a single token can dominate a micro-batch. In long multi-view trajectories (VSI), this makes the OPSD loss a noisy function of teacher peakedness rather than a faithful reverse-KL of geometric content. The paper gives no bias/variance/stability analysis, no seeds, and no error bars, while the headline deltas are only 0.4-1.9 points. The ablation sweep (Fig. 4 right) and the zero-shot routing study on a 1000-question subset (Sec. 4.4) are informative, and the entropy curve (Fig. 6) suggests a plausible regularizer mechanism, but that same entropy dynamics is exactly what a generic KL entropy bonus would produce, so it does not discriminate geometric privilege from a tuning effect. I do not claim the method is wrong; I claim the central identification is unproven, and the concrete test above would settle it. I also note, per the review rules, that the summary wording 'outperforms ... across spatial reasoning benchmarks' (b4, b23, b113) does not hold at benchmark level, since OPSD exceeds GPD on ViewSpatial 4B and MMSI 2B. This is a precision issue in the claim, not an attack on the authors. Verdict stays CONDITIONAL: the method is coherent and potentially reproducible, and the MindCube result gives real reason to believe the geometric cue matters, but the estimator stability needs a direct check before the causal attribution is accepted.
The referee's report
If the attribution holds, this is a useful and inexpensive recipe: it converts 3D annotations (or off-the-shelf depth and semantic estimates) into token-level process supervision for spatial reasoning while keeping inference RGB-only. The paper has real strengths: controlled comparisons on identical backbones, training data, and precomputed cues; a clean component ablation in Figure 4 that distinguishes 3D text, answer, and their combination; a zero-shot privilege study in Figure 5 that genuinely motivates the routed-text form; and a distillation estimator that is exactly unbiased in expectation over the student's own sampled tokens. The headline aggregate gains (+0.7 to +1.9 over GRPO on benchmark averages, and +3.4 to +6.1 over answer-privileged OPSD, largely driven by MindCube) are plausible, but they are modest and single-run. The paper's own entropy analysis in Figure 6 and the lack of any seeds or variance reporting leave open the question of whether the improvement comes from the geometric content of the privilege or from a generic KL/exploration effect; that question is the main risk to the paper's stated mechanism.
Pith's reading of the paper (a machine's summary, not the authors')
GPD claims that spatial errors in vision-language models are perceptual before they are logical: a model that misjudges depth or direction can reason coherently over the wrong facts, and neither outcome rewards nor the reference answer can repair that. The paper proposes putting the scene's true geometry, which 3D-scanned spatial training corpora already provide, into the training signal alone, through a teacher conditioned on question-routed depth, semantic, and bird's-eye-view text plus the answer, distilling token-level corrections only on incorrect trajectories. The deployed model stays RGB-only, so no architecture or inference cost is paid. If the reported results hold, question-routed geometric privilege is an effective way to supervise spatial reasoning where final-answer supervision is blind.
What carries the argument. The argument turns on three mechanisms. First, offline privilege construction: depth, semantic segmentation, and bird's-eye-view layout are converted by fixed templates into textual summaries, and a question-conditioned router selects the relevant subset $r(q) \subseteq \{depth, semantics, BEV\}$, so the teacher sees only the geometric evidence a question needs. Second, token-level teacher scoring: the teacher (same weights, extra conditioning) re-scores the student's own sampled tokens through the per-token log-probability gap $\Delta_{i,j} = \log \pi_\theta(t_j \mid t_{<j}, p_T) - \log \pi_\theta(t_j \mid t_{<j}, p_S)$, estimated as a single-sample reverse KL $d_{i,j} = \exp(\Delta_{i,j}) - \Delta_{i,j} - 1$. Third, error gating: the distillation term is aggregated only over incorrect trajectories ($R_i < \tau = 0.5$), the resulting batch scalar is clipped at 1.5, and the term is added to GRPO with weight $\lambda = 0.003$, giving the joint objective $L = L_{\mathrm{GRPO}} + \lambda \tilde{L}_{\mathrm{OPSD}}$ that the paper optimizes.
Core claim. The central claim is that one VLM can play both roles: an RGB-only student that samples trajectories from question and image, and a privileged teacher that additionally sees routed textual 3D cues and the reference answer and re-scores those trajectories token by token. On the Qwen3-VL-4B backbone, GPD achieves 57.1 on VSI-Bench and a 37.6 average across MindCube, SPAR-Bench, MMSI-Bench, and ViewSpatial, beating GRPO by 0.7 and 1.9 points respectively and beating answer-privileged OPSD by larger margins, with the same ordering at the 2B scale. A zero-shot study grounded the design choice: routed text was the only privilege form that exceeded the RGB-only baseline (52.2 vs. 49.1), while full-context text and image overlays hurt. Ablations attribute the gain to three design choices: the complementarity of 3D and answer privilege (41.5 combined vs. 40.2 answer-only and 40.5 3D-only), question-conditioned routing over full-context injection (41.5 vs. 39.8), and restricting the KL to incorrect trajectories (41.5 vs. 40.6 for all-trajectory application). Test-time privilege injection helps GPD less than it helps GRPO (+0.6 vs. +2.1 points), which the paper reads as evidence that the geometric cues have been internalized by the student.
The premise it rests on. The load-bearing premise is that the single-sample token gap $d_{i,j} = \exp(\Delta_{i,j}) - \Delta_{i,j} - 1$, computed on the student's own tokens, is a faithful and stable estimate of the teacher-student reverse KL on long multi-view trajectories; the paper gives no bias, variance, or stability analysis, and a token where the teacher is far more confident than the student injects an exponentially large per-token gradient that the batch-level clip does not touch. A secondary premise is that the question routing validated on a 1000-question subset transfers to the full training mixture.
If the paper is right (Pith's inferences, not the authors')
Spatial training no longer needs 3D at inference: geometry enters only through the training signal, so the deployed model keeps the same architecture, inputs, and latency as a plain RGB VLM.
Corpora built from 3D scans become a source of process-level geometric supervision, and for datasets without aligned ground truth, off-the-shelf depth and segmentation estimators can supply the cues instead.
Answer privilege and geometric privilege fix different failure modes—target outcome versus perceptual facts—so combining them is better than either alone.
Question-conditioned routing is a genuine component of the gain: indiscriminate full-context injection performs worse despite containing more information, and the learned routing pattern matches geometric intuition (counting to semantic maps, distance to depth, room size to BEV).
Privilege internalization is measurable: GPD's smaller test-time gain from privilege injection (+0.6 vs. +2.1 for GRPO) is a signature that the student has learned the geometry rather than depending on the teacher.
A robustness check the paper does not run: replacing the teacher's 3D text with question-irrelevant text of equal length would separate 'geometry drives the gain' from 'a generic KL regularizer drives the gain'.
The mechanism looks transferable to any privileged modality that can be serialized as text, such as estimated depth for driving scenes, optical flow for video QA, or map priors for navigation, since the correction targets perceptual rather than logical mistakes.
The router is itself a hidden selection policy validated only on a 1000-question subset; training routing end-to-end or verifying it across the full training mixture would test whether the routing distribution, and not just the presence of 3D text, carries the benefit.
Scaling is untested: the advantage over GRPO is a few points on a 4B backbone after one epoch, and whether it survives longer training or larger backbones, where outcome supervision is stronger, would show whether the effect saturates.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Spatial reasoning remains a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence. Existing remedies either inject 3D into the model at inference, paying architecture and latency costs, or train with outcome rewards that supervise only the final answer. Spatial errors originate in perception: a misjudged depth or direction can be corrected only by the scene's true geometry, which the 3D-scanned sources of spatial training corpora already provide. We propose GPD (Geometry-Privileged Distillation), which makes geometric evidence the privilege in on-policy self-distillation (OPSD). For each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text and routed to the teacher alongside the reference answer; a privileged KL, applied only to incorrect trajectories, augments GRPO, and the deployed model remains RGB-only. On the 4B backbone, GPD achieves 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming both GRPO and answer-privileged OPSD across spatial reasoning benchmarks. Ablations confirm the complementarity of 3D and answer privilege, the advantage of question-conditioned routing over full-context injection, and the benefit of restricting distillation to incorrect trajectories.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-09
The authors' main claim, as the classifier picked it out 0.98
“On the 4B backbone, GPD achieves 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming both GRPO and answer-privileged OPSD across spatial reasoning benchmarks.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.07 · states limitation 0.24 · human participants 0.03 · controlled comparison 0.94 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.97 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Spatial reasoning remains a persistent weakness of vision-language models (VLMs), because RGB inputs do not directly provide geometric evidence. | background 1.00 | 0.06 | 0.03 | asserted 1.00 |
| Existing remedies either inject 3D into the model at inference, paying architecture and latency costs, or train with outcome rewards that supervise only the final answer. | background 0.55 | 0.04 | 0.02 | asserted 1.00 |
| Spatial errors originate in perception: a misjudged depth or direction can be corrected only by the scene's true geometry, which the 3D-scanned sources of spatial training corpora already provide. | background 0.87 | 0.11 | 0.07 | asserted 0.99 |
| We propose GPD (Geometry-Privileged Distillation), which makes geometric evidence the privilege in on-policy self-distillation (OPSD). | contribution 0.92 | 0.73 | 0.16 | asserted 0.99 |
| For each question, depth, semantic, and bird's-eye-view (BEV) cues are rendered as compact text and routed to the teacher alongside the reference answer; a privileged KL, applied only to incorrect trajectories, augments GRPO, and the deployed model remains RGB-only. | method 1.00 | 0.91 | 0.12 | asserted 0.99 |
| On the 4B backbone, GPD achieves 57.1 on VSI-Bench and 37.6 average across MindCube, SPARBench, MMSI-Bench, and ViewSpatial, outperforming both GRPO and answer-privileged OPSD across spatial reasoning benchmarks. | result 1.00 | 0.96 | 0.55 | asserted 1.00 |
| Ablations confirm the complementarity of 3D and answer privilege, the advantage of question-conditioned routing over full-context injection, and the benefit of restricting distillation to incorrect trajectories. | result 0.76 | 0.95 | 0.19 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.97
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-09: 1 request, 5,836 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Showing 7 of the paper's 8 figures. Figures are the authors', reproduced from the paper.
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment