IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

Dongming Zhang; Hao Wu; Jian Xue; Ke Lu; Siwen Jiao; Xiuyuan Zhu; Zijin Du

IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Ground-truth boxes, shown to a teacher during training, improve visual grounding beyond serving as coordinate labels.

desk verdict A modest but real empirical contribution: the privileged-box claim survives the ablations, but the paper needs a wrong-box control, multi-seed runs, and code before the exact size of the effect is fully trustworthy. read the letter →

arxiv 2607.15732 v3 pith:OSFT67S7 submitted 2026-07-17 cs.CV

Xiuyuan Zhu , Ke Lu , Hao Wu , Siwen Jiao , Zijin Du , Dongming Zhang , Jian Xue This is my paper · ORCID

classification cs.CV

keywords visualgroundingprivilegeddistillationcoordinategenerationmultimodallargelanguagemodelIoU-awareweightingreferringexpressionknowledgeon-policyself-distillation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Visual grounding with multimodal large language models is usually trained to output bounding-box coordinates as text, but scored by geometric overlap between predicted and true boxes. IoU-PD uses ground-truth boxes twice: as the supervised coordinate target, and as a privileged visual hint drawn on the image for a frozen teacher model. The teacher sees the box-marked image plus a short hint, while the student sees only the original image and prompt; the student is trained with a supervised fine-tuning anchor and an IoU-aware distillation loss. At inference the teacher, hint, and box overlay are all removed, so the method changes only training, not the deployed interface. Across standard referring-expression benchmarks, the full setting improves mIoU by 4.03 points and Acc@0.7 by 4.25 points over a strong 4B base model, with ablations attributing the gain to the box-marked teacher and the IoU-aware weighting.

What carries the argument

The central mechanism is a frozen privileged teacher paired with an on-policy distillation loss. The teacher receives the original image with the ground-truth box overlaid in green, plus the original prompt augmented with the hint 'The answer is located within the green rectangle'; the student receives only the original image and prompt. The student's own decoded box is scored under both branches, and the KL term is token-weighted by a product of factors — exponential in (1 − IoU of the sampled box), softmax over coordinate-level errors, digit-position significance, a sigmoid on teacher–student log-probability agreement, and teacher entropy — so that geometrically important and reliable toke

What would settle it

Retrain the same pipeline with a teacher whose green rectangle is placed at a random location (same size, same hint text); if the student's mIoU and Acc@0.7 gains stay close to the full IoU-PD numbers, the claim that ground-truth location information drives the improvement is falsified, whereas a drop to the original-teacher baseline would confirm that the privileged location is the active component.

Watch

Extended reading notes

Core claim

The paper claims that ground-truth boxes carry useful training-time guidance beyond their role as output labels. When a frozen teacher conditions on the image with the target box drawn in green and a text hint saying the answer lies inside the green rectangle, its output distribution contains region-aware information that a student, seeing only the original image, can absorb through distillation. The student is anchored by supervised fine-tuning on the true coordinate string and shaped by a token-weighted KL loss whose weights combine a sample-level IoU factor, per-coordinate error, digit-position significance, teacher–student agreement, and teacher confidence. Experiments show consistent re

Load-bearing premise

The load-bearing premise is that the frozen teacher's distribution, computed on an input that literally contains the ground-truth box, remains a faithful distillation target for a student that will never see that box—that the conditional mismatch between the privileged and standard inputs is small enough that the teacher's logits teach region awareness rather than reliance on the green rectangle.

Editorial extensions

If this is right

Training-time-only privileged visual hints can be layered onto coordinate-generating multimodal LLMs without changing the inference input format or adding a localization head.
The gains increase under stricter overlap thresholds: P@0.9 rises 7.75 points and P@0.95 by 12.84 over the base model, indicating the method pushes many predictions into the high-overlap range.
The method helps small and medium objects more than large ones, suggesting privileged box hints are especially useful when coordinate errors cover a larger fraction of the target region.
Removing the SFT anchor sharply reduces the gain, so privileged distillation complements direct coordinate supervision rather than replacing it.
The approach requires grounding annotations at training time; without ground-truth boxes, the privileged teacher input cannot be constructed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A direct test of the mechanism would be to replace the ground-truth overlay with a randomly placed green box of the same size; if the gain persists, the active ingredient is the visual marker itself rather than the location information.
The same privileged-distillation scheme could transfer to other structured-output tasks with geometric supervision and spatial metrics, such as keypoint detection, dense captioning, or layout generation.
Because the teacher is frozen and sees the base distribution plus the overlay, the method is a form of on-policy self-distillation with a visual hint; it may combine naturally with verifiable rewards that directly optimize IoU.
The paper's own limitation—that the hint is a simple overlay and gains are moderate—points to richer privileged inputs, such as a cropped region or segmentation mask, as a way to reduce the remaining conditional mismatch.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a genuine but incremental training method that improves region-level grounding by 3-4 points over a Qwen3-VL-4B coordinate-generating baseline. The core claim — that a frozen teacher seeing the ground-truth box on its input provides useful guidance beyond plain distillation — is supported by the ablation table. The gain over SFT alone is about one point of Acc@0.7, and the full recipe adds about four points over the base. That is useful progress for REF grounding.

What's actually new: the combination of a box-marked privileged teacher with on-policy distillation and a five-factor token weighting for referring-expression grounding. Prior work already used visually enriched teacher guidance for GUI grounding, so the delta is the application to REC and the explicit IoU-aware weights. The ablation design is mostly careful: B vs D separates the box teacher from self-distillation, and C shows the SFT anchor is needed.

Soft spots, in proportion. First, the B-to-D comparison changes two things at once: the teacher gets both the green box and the text hint. A wrong-box control and a hint-only control would make the "privileged box" claim much cleaner. The paper argues in App. C that a box overlay is a conservative hint, but that is reasoning, not data. Second, all numbers are single-run; no error bars. The tau_r comparison picks 0.5 as best on what looks like the same test set; a validation split would be more reassuring. Third, no code or checkpoints are released yet. Fourth, the conditional-mismatch risk — teacher's distribution under the box may be far from anything the student can express — is acknowledged in App. C but never measured. The agreement factor likely mitigates it, but the paper doesn't show that.

None of this breaks the central claim. The ablations are consistent, the gains survive stricter IoU thresholds and object-size splits, and the limitations section is honest about the gains being moderate and requiring annotations. This is a paper that should go to peer review: a serious referee can ask for the missing controls and multi-seed numbers, and the contribution should hold up if they do.

Referee Report

3 major / 4 minor

Summary. The paper proposes IoU-PD, a training-time privileged-distillation method for coordinate-generating multimodal large language models. The student is trained on the original image and referring expression, while a frozen teacher receives a box-marked image plus an appended hint. The total loss is an SFT anchor plus a token-weighted KL distillation over student-generated responses, with weights informed by IoU, coordinate error, digit position, teacher-student agreement, and teacher entropy. At inference the teacher, box overlay, and hint are removed. Experiments on RefCOCO, RefCOCO+, and RefCOCOg with Qwen3-VL-4B report consistent gains over the base model (+4.03 mIoU, +4.25 Acc@0.7 in the main setting), with ablations isolating SFT, original-teacher distillation, box-marked teacher input, and IoU-aware weighting. The paper honestly lists limitations, including the requirement for grounding annotations and the condition-mismatch risk of privileged distillation.

Significance. If the reported gains are robust, the paper makes a practical and conceptually useful contribution: it improves region-level grounding without changing the inference-time interface, adding a teacher branch, or requiring extra inference-time annotations. The ablation structure is a genuine strength: the SFT anchor, the non-privileged self-distillation baseline, and the IoU-aware weighting are separately tested, and the threshold, object-size, and scaling analyses give a more complete picture than a single benchmark number would. The central claim—that ground-truth boxes can serve as privileged guidance beyond coordinate labels—is plausible and largely well supported by the component ablations. However, the key isolation of the box-teacher effect is confounded by the simultaneously introduced text hint, and every result is single-run with no error bars. These issues are fixable within the manuscript's scope but need to be addressed before the central causal claim can be accepted.

major comments (3)

[§4.3, Table 7, Appendix E] The comparison that isolates the privileged box teacher is Variant B ('Original teacher') vs Variant D ('Box teacher'). Moving from B to D changes two variables at once: the green box overlay and the appended hint text 'The answer is located within the green rectangle.' There is no box-only, hint-only, wrong-box, or random-box control. The observed difference (+0.65 Acc@0.7, +0.0051 mIoU) is therefore not uniquely attributable to the semantic content of the ground-truth box. Since this is the central claim of the paper, please add a wrong-box teacher (same overlay and hint but with a non-target box) and a box-overlay-without-hint variant; if a wrong-box teacher produces a similar gain, the effect would be consistent with a peaked/regularizing target rather than with privileged visual guidance about the referred region.
[§4.3, Table 7; §4.2, Table 3] All results are reported from single training runs with no error bars or significance tests. The load-bearing component differences are small: B-to-D is 0.65 Acc@0.7, D-to-E is 0.27, E-to-F is 0.08, and F-to-Full is 0.13. These differences are within the range of typical run-to-run variation for LLM fine-tuning, so the finer-grained claims (e.g., that the sample-level IoU factor with tau_r=0.5 helps over tau_r=1.0) are not reliably supported. Please provide multiple seeds with standard deviations, or an equivalent statistical analysis, for at least the Table 7 ablation rows and the main Table 6 comparison.
[§3.3, Eq. (2); Appendix C] The method's mechanism requires that the frozen teacher's distribution p_T(·|x_t,·), which is conditioned on the box-marked image, be a learnable and beneficial target for the student's p_θ(·|x_s,·), which does not see the box. Appendix C acknowledges the conditional-mismatch risk but offers only a qualitative argument. No quantitative measurement is provided for how large the mismatch is, how it evolves during training, or whether the KL term functions as genuine guidance rather than a regularizer. Measuring teacher-student KL divergence on held-out prompts before and after training, or reporting teacher accuracy under x_t and student accuracy under x_s, would directly address the load-bearing assumption and materially strengthen the paper.

minor comments (4)

[Table 2, row 'Original teacher'] Variant B improves mIoU and Acc@0.7 over SFT-only (0.8492 vs 0.8470; 85.63 vs 85.32) but degrades Acc@0.5 (89.80 vs 90.62). The paper does not discuss this cross-metric inconsistency; a brief explanation would help.
[§3.4, Eq. (3)] The normalization expression for w̄_t is ambiguous as typeset ('|Y|w_t P_j...'). It should read w̄_t = |Y| w_t / (Σ_j w_j + ε); please fix the notation.
[Table 3 and Figure 4] The 2B scaling point is only reported at 30k/1 epoch, so the comparison between backbone sizes is incomplete. Adding a 2B/300k/3-epoch run (or explicitly stating why it is omitted) would make the scaling conclusion cleaner.
[Algorithm 1, line 5] The notation 'ŷ ← Decode(p_θ(·|x_s))' is underspecified. Clarify whether this is greedy decoding, sampling, or beam search, and whether the same decoding is used for the teacher's condition.

Circularity Check

0 steps flagged · score 2.0 of 10

Empirical training method; gains measured on held-out splits — no circular reduction. Score 2 reflects one minor non-load-bearing self-citation (Smooth Operator) and the acknowledged-but-unquantified teacher-student conditional mismatch.

full rationale

The central claim—that ground-truth boxes provide useful privileged guidance beyond coordinate labels—is an empirical effect, not a derivation that restates its inputs. The training objective (Section 3.3, L = L_kd + αL_sft) uses the GT box both as the SFT target and as the teacher's box overlay; the claimed gains (+4.03 mIoU, +4.25 Acc@0.7 over Qwen3-VL-4B) are measured on the held-out RefCOCO/RefCOCO+/RefCOCOg splits under a unified evaluation protocol (Table 6), so no fitted value is renamed as a prediction. The IoU-aware weights w_t = r(u)·c_t·d_t·a_t·e_t (Section 3.4) are per-sample training reweightings computed from the student's own decoded box; aligning the loss with the evaluation metric is metric-aligned optimization, not a by-construction guarantee of test-set improvement, and the ablations (Table 7, rows D->E->Full) show small, non-forced increments. The only load-bearing assumption—that the frozen box-marked teacher's distribution p_T(·|x_t) transfers to the student that lacks the box—is an acknowledged risk, not a circularity: the paper explicitly flags the conditional-mismatch danger in Appendix C ('increasing the conditional mismatch between teacher and student'), and the B-vs-D ablation is confounded (the hint text changes alongside the box) and lacks a wrong-box control, which is a validity/robustness limitation rather than an equivalence-by-construction. The paper's own Limitations section acknowledges that the method requires grounding annotations and that gains are moderate. The one self-citation is Smooth Operator (Jiao et al., 2026), a co-authored related-work baseline that is not load-bearing. No equation is equivalent to its inputs by definition.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The method introduces six hand-chosen weighting hyperparameters and relies on the standard privileged-distillation assumption that teacher-side conditional distributions transfer to the student. No new physical or architectural entities are postulated; the box overlay is an input transform, not an invented entity. The free-parameter count is moderate and the lack of sensitivity analysis beyond tau_r is a weakness.

free parameters (6)

SFT coefficient alpha = 1.0
Chosen by hand to balance distillation and supervised fine-tuning; no sensitivity analysis reported (Appendix A.1).
Sample-level IoU temperature tau_r = 0.5
Ablated on test splits (Variant F vs Full) and the better value selected; tau_r=1.0 gives a nearly identical result (+4.12 vs +4.25 Acc@0.7).
Coordinate softmax temperature tau_c = 1.0
Chosen for the coordinate-error factor c_t; no sensitivity analysis reported (Appendix A.3).
Digit-position weight lambda_d = 0.5
Scales the digit-position factor d_t; chosen by hand, no sensitivity analysis reported (Appendix A.3).
Agreement sigmoid slope beta = 3.0
Controls the teacher-student agreement factor a_t; chosen by hand, no sensitivity analysis reported (Appendix A.3).
Teacher-confidence entropy temperature tau_e = 1.0
Scales the entropy-based teacher confidence factor e_t; chosen by hand, no sensitivity analysis reported (Appendix A.3).

assumptions (4)

domain assumption The teacher distribution p_T(.|x_t, y_hat_<t) computed under the box-marked privileged input is a useful distillation target for the student conditioned on the original input x_s.
Section 3.3, Eq. (2); if the privileged input makes the teacher target too different from the student's inference-time input, distillation could mislead; the paper relies on this transfer.
domain assumption The token-to-coordinate parsing and the fallback rule that counts malformed responses as invalid are fair across all compared models.
Appendix B.2; the unified evaluation protocol assumes the single parser does not systematically penalize any baseline.
domain assumption Digit-position significance (hundreds > tens > ones) and the multiplicative weighting factors reflect the true effect of token errors on IoU.
Section 3.4; the weighting formula w_t = r(u).c_t.d_t.a_t.e_t assumes these heuristics align with geometric importance.
domain assumption The frozen same-architecture teacher initialized from the same base checkpoint provides reliable logits without EMA updates.
Appendix A.2; no EMA, stop-gradient; assumes no catastrophic drift in teacher quality during training.

how reviews work

0 comments

Cite this review

Pith. "Pith review of IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/OSFT67S7

@misc{pith2026260715732,
  author       = {Pith},
  title        = {Pith review of: IoU-PD: IoU-Aware Privileged Distillation for Visual Grounding with Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OSFT67S7}},
  note         = {Machine review of arXiv:2607.15732}
}

read the original abstract

Visual grounding with multimodal large language models is commonly formulated as autoregressive coordinate generation, where a model outputs bounding-box coordinates as text given an image and a referring-expression prompt. While this interface is simple and compatible with instruction following, it introduces a mismatch between training and evaluation: training optimizes token-level likelihood over coordinate strings, whereas grounding quality is measured by geometric overlap. We propose IoU-PD, an IoU-aware privileged distillation method for coordinate-generating multimodal large language models. IoU-PD uses ground-truth boxes not only as coordinate targets, but also as privileged training-time guidance. During training, the student receives the original image and prompt, while a frozen teacher receives a box-marked image and an augmented prompt that indicates the marked region. The student is trained with a supervised fine-tuning anchor and a privileged distillation loss whose token weights reflect both geometric importance and teacher reliability. At inference time, IoU-PD requires no box overlay, privileged hint, teacher branch, or additional prediction module. Experiments on standard referring-expression grounding benchmarks show consistent region-level improvements over strong coordinate-generating baselines, demonstrating that ground-truth boxes can provide useful privileged guidance beyond serving as coordinate labels. Project page: https://xyzzzh.github.io/IoU-PD/