{"id":"a57f92c5-e542-4600-8b1c-5f62512bd7e3","arxiv_id":"2607.15732","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Training a coordinate-generating VLM with an IoU-aware distillation loss from a teacher that sees the ground-truth box marked on the image improves referring-expression grounding by ~3-4 accuracy points.","lead":"IOU-PD is a training scheme that improves visual grounding in multimodal language models by letting a frozen teacher model see the ground-truth box drawn on the image and distilling that knowledge into a student that only sees the original image and text prompt. It gains a few accuracy points on standard referring-expression benchmarks without changing what the model receives at inference time.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim depends on the frozen box-marked teacher's distribution being a learnable target for the student, but the paper never measures that conditional mismatch; the B-vs-D ablation also confounds the visual box with the text hint and lacks a wrong-box control.","rationale":"The reader's weakest assumption matches the load-bearing concern: the privileged teacher's distribution may be too different or too peaked relative to what the student can produce, and the paper provides only the B-vs-D ablation as indirect evidence. I agree that this is the least secure part of the central claim. The paper is internally consistent and the ablations are reasonable, but the attribution of the gain to privileged GT-box guidance is not fully established. A random-box control directly tests whether the correctness of the privileged box matters, which is the crux of the 'beyond coordinate labels' claim. The paper also lacks error bars, so a three-seed comparison is important. The proposed test would strengthen or refute the central mechanism. Therefore the reader's CONDITIONAL verdict is appropriate: conditional on the privileged-signal control and distribution-shift diagnostics, the claim would be supported; without them, it remains plausible but not fully verified.","tokens_in":17720,"tokens_out":13563,"duration_ms":158517,"concrete_test":"Train the full 4B/300k/3-epoch pipeline with one change only: the frozen teacher receives a box-marked image in which the green rectangle is a randomly sampled box from another training example, keeping the hint, student input, and all hyperparameters identical. Evaluate Acc@0.7 and mIoU over three seeds and compare against the SFT-only, Original-teacher, and Full IOU-PD rows in Table 7. If the random-box variant is statistically indistinguishable from Full, the improvement does not require the ground-truth box's correctness and the central claim fails. If it falls significantly below Full—toward the Original-teacher or SFT-only level—then the GT-box-specific privileged guidance is load-bearing, and the conditional-mismatch concern is mitigated.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim requires that the frozen teacher's conditional distribution under the privileged input, p_T(·|x_t, ·), is a beneficial and learnable target for p_θ(·|x_s, ·) in Eq. (2) of Section 3.3. Because x_t contains the ground-truth box drawn on the image, the teacher's distribution can be much more peaked at the GT coordinates than anything the student can produce from x_s, which lacks the box. Appendix C explicitly acknowledges this conditional-mismatch risk, but the paper provides no quantitative analysis of how large the mismatch is or whether it decreases during training. The only direct evidence for the privileged component is the Box-teacher vs Original-teacher ablation in Table 7 (rows B vs D). That comparison changes two variables at once: it adds the green box and the hint sentence. It therefore does not isolate visual privileged guidance. No hint-only, box-only, or wrong-box control is run, so the observed gain could in principle come from the extra on-policy distillation signal or from a regularizing peaked target rather than from the ground-truth box's semantic content. This is the weakest link in the argument that GT boxes help beyond coordinate labels.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes IoU-PD, a training-time privileged-distillation method for coordinate-generating multimodal large language models. The student is trained on the original image and referring expression, while a frozen teacher receives a box-marked image plus an appended hint. The total loss is an SFT anchor plus a token-weighted KL distillation over student-generated responses, with weights informed by IoU, coordinate error, digit position, teacher-student agreement, and teacher entropy. At inference the teacher, box overlay, and hint are removed. Experiments on RefCOCO, RefCOCO+, and RefCOCOg with Qwen3-VL-4B report consistent gains over the base model (+4.03 mIoU, +4.25 Acc@0.7 in the main setting), with ablations isolating SFT, original-teacher distillation, box-marked teacher input, and IoU-aware weighting. The paper honestly lists limitations, including the requirement for grounding annotations and the condition-mismatch risk of privileged distillation.","tokens_in":18063,"tokens_out":4910,"duration_ms":49809,"significance":"If the reported gains are robust, the paper makes a practical and conceptually useful contribution: it improves region-level grounding without changing the inference-time interface, adding a teacher branch, or requiring extra inference-time annotations. The ablation structure is a genuine strength: the SFT anchor, the non-privileged self-distillation baseline, and the IoU-aware weighting are separately tested, and the threshold, object-size, and scaling analyses give a more complete picture than a single benchmark number would. The central claim—that ground-truth boxes can serve as privileged guidance beyond coordinate labels—is plausible and largely well supported by the component ablations. However, the key isolation of the box-teacher effect is confounded by the simultaneously introduced text hint, and every result is single-run with no error bars. These issues are fixable within the manuscript's scope but need to be addressed before the central causal claim can be accepted.","major_comments":[{"comment":"The comparison that isolates the privileged box teacher is Variant B ('Original teacher') vs Variant D ('Box teacher'). Moving from B to D changes two variables at once: the green box overlay and the appended hint text 'The answer is located within the green rectangle.' There is no box-only, hint-only, wrong-box, or random-box control. The observed difference (+0.65 Acc@0.7, +0.0051 mIoU) is therefore not uniquely attributable to the semantic content of the ground-truth box. Since this is the central claim of the paper, please add a wrong-box teacher (same overlay and hint but with a non-target box) and a box-overlay-without-hint variant; if a wrong-box teacher produces a similar gain, the effect would be consistent with a peaked/regularizing target rather than with privileged visual guidance about the referred region.","section":"§4.3, Table 7, Appendix E"},{"comment":"All results are reported from single training runs with no error bars or significance tests. The load-bearing component differences are small: B-to-D is 0.65 Acc@0.7, D-to-E is 0.27, E-to-F is 0.08, and F-to-Full is 0.13. These differences are within the range of typical run-to-run variation for LLM fine-tuning, so the finer-grained claims (e.g., that the sample-level IoU factor with tau_r=0.5 helps over tau_r=1.0) are not reliably supported. Please provide multiple seeds with standard deviations, or an equivalent statistical analysis, for at least the Table 7 ablation rows and the main Table 6 comparison.","section":"§4.3, Table 7; §4.2, Table 3"},{"comment":"The method's mechanism requires that the frozen teacher's distribution p_T(·|x_t,·), which is conditioned on the box-marked image, be a learnable and beneficial target for the student's p_θ(·|x_s,·), which does not see the box. Appendix C acknowledges the conditional-mismatch risk but offers only a qualitative argument. No quantitative measurement is provided for how large the mismatch is, how it evolves during training, or whether the KL term functions as genuine guidance rather than a regularizer. Measuring teacher-student KL divergence on held-out prompts before and after training, or reporting teacher accuracy under x_t and student accuracy under x_s, would directly address the load-bearing assumption and materially strengthen the paper.","section":"§3.3, Eq. (2); Appendix C"}],"minor_comments":[{"comment":"Variant B improves mIoU and Acc@0.7 over SFT-only (0.8492 vs 0.8470; 85.63 vs 85.32) but degrades Acc@0.5 (89.80 vs 90.62). The paper does not discuss this cross-metric inconsistency; a brief explanation would help.","section":"Table 2, row 'Original teacher'"},{"comment":"The normalization expression for w̄_t is ambiguous as typeset ('|Y|w_t P_j...'). It should read w̄_t = |Y| w_t / (Σ_j w_j + ε); please fix the notation.","section":"§3.4, Eq. (3)"},{"comment":"The 2B scaling point is only reported at 30k/1 epoch, so the comparison between backbone sizes is incomplete. Adding a 2B/300k/3-epoch run (or explicitly stating why it is omitted) would make the scaling conclusion cleaner.","section":"Table 3 and Figure 4"},{"comment":"The notation 'ŷ ← Decode(p_θ(·|x_s))' is underspecified. Clarify whether this is greedy decoding, sampling, or beam search, and whether the same decoding is used for the teacher's condition.","section":"Algorithm 1, line 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely to be acceptable after additional experiments. The core idea is sound and the overall ablation framework is a strength, but the single ablation that isolates the proposed privileged-box mechanism is confounded, and the magnitudes of several component gains are small enough that single-seed results are not convincing. I would also encourage the editor to weigh whether the Table 1 comparison against general VLMs (some not fine-tuned for grounding) should be complemented by a same-protocol fine-tuned baseline, since the headline 'consistent improvements over strong coordinate-generating baselines' is best supported by the controlled base-model comparison in Table 6/Table 7."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine but incremental training method that improves region-level grounding by 3-4 points over a Qwen3-VL-4B coordinate-generating baseline. The core claim — that a frozen teacher seeing the ground-truth box on its input provides useful guidance beyond plain distillation — is supported by the ablation table. The gain over SFT alone is about one point of Acc@0.7, and the full recipe adds about four points over the base. That is useful progress for REF grounding.\n\nWhat's actually new: the combination of a box-marked privileged teacher with on-policy distillation and a five-factor token weighting for referring-expression grounding. Prior work already used visually enriched teacher guidance for GUI grounding, so the delta is the application to REC and the explicit IoU-aware weights. The ablation design is mostly careful: B vs D separates the box teacher from self-distillation, and C shows the SFT anchor is needed.\n\nSoft spots, in proportion. First, the B-to-D comparison changes two things at once: the teacher gets both the green box and the text hint. A wrong-box control and a hint-only control would make the \"privileged box\" claim much cleaner. The paper argues in App. C that a box overlay is a conservative hint, but that is reasoning, not data. Second, all numbers are single-run; no error bars. The tau_r comparison picks 0.5 as best on what looks like the same test set; a validation split would be more reassuring. Third, no code or checkpoints are released yet. Fourth, the conditional-mismatch risk — teacher's distribution under the box may be far from anything the student can express — is acknowledged in App. C but never measured. The agreement factor likely mitigates it, but the paper doesn't show that.\n\nNone of this breaks the central claim. The ablations are consistent, the gains survive stricter IoU thresholds and object-size splits, and the limitations section is honest about the gains being moderate and requiring annotations. This is a paper that should go to peer review: a serious referee can ask for the missing controls and multi-seed numbers, and the contribution should hold up if they do.","headline":"A modest but real empirical contribution: the privileged-box claim survives the ablations, but the paper needs a wrong-box control, multi-seed runs, and code before the exact size of the effect is fully trustworthy.","tokens_in":18545,"tokens_out":2860,"would_cite":true,"duration_ms":30468,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Ground-truth boxes, shown to a teacher during training, improve visual grounding beyond serving as coordinate labels.","keywords":["visual grounding","privileged distillation","coordinate generation","multimodal large language model","IoU-aware weighting","referring expression","knowledge distillation","on-policy self-distillation"],"falsifier":"Retrain the same pipeline with a teacher whose green rectangle is placed at a random location (same size, same hint text); if the student's mIoU and Acc@0.7 gains stay close to the full IoU-PD numbers, the claim that ground-truth location information drives the improvement is falsified, whereas a drop to the original-teacher baseline would confirm that the privileged location is the active component.","tokens_in":1375,"feed_emoji":"🎯","tokens_out":1769,"duration_ms":65188,"temperature":0.7,"pith_summary":"Visual grounding with multimodal large language models is usually trained to output bounding-box coordinates as text, but scored by geometric overlap between predicted and true boxes. IoU-PD uses ground-truth boxes twice: as the supervised coordinate target, and as a privileged visual hint drawn on the image for a frozen teacher model. The teacher sees the box-marked image plus a short hint, while the student sees only the original image and prompt; the student is trained with a supervised fine-tuning anchor and an IoU-aware distillation loss. At inference the teacher, hint, and box overlay are all removed, so the method changes only training, not the deployed interface. Across standard referring-expression benchmarks, the full setting improves mIoU by 4.03 points and Acc@0.7 by 4.25 points over a strong 4B base model, with ablations attributing the gain to the box-marked teacher and the IoU-aware weighting.","feed_headline":"Ground-truth boxes double as training hints to sharpen visual grounding","feed_subtitle":"A training-only box overlay lifts mean overlap by 4 points and strict-overlap accuracy by 4.25 points.","key_machinery":"The central mechanism is a frozen privileged teacher paired with an on-policy distillation loss. The teacher receives the original image with the ground-truth box overlaid in green, plus the original prompt augmented with the hint 'The answer is located within the green rectangle'; the student receives only the original image and prompt. The student's own decoded box is scored under both branches, and the KL term is token-weighted by a product of factors — exponential in (1 − IoU of the sampled box), softmax over coordinate-level errors, digit-position significance, a sigmoid on teacher–student log-probability agreement, and teacher entropy — so that geometrically important and reliable toke","core_discovery":"The paper claims that ground-truth boxes carry useful training-time guidance beyond their role as output labels. When a frozen teacher conditions on the image with the target box drawn in green and a text hint saying the answer lies inside the green rectangle, its output distribution contains region-aware information that a student, seeing only the original image, can absorb through distillation. The student is anchored by supervised fine-tuning on the true coordinate string and shaped by a token-weighted KL loss whose weights combine a sample-level IoU factor, per-coordinate error, digit-position significance, teacher–student agreement, and teacher confidence. Experiments show consistent re","pith_inferences":["A direct test of the mechanism would be to replace the ground-truth overlay with a randomly placed green box of the same size; if the gain persists, the active ingredient is the visual marker itself rather than the location information.","The same privileged-distillation scheme could transfer to other structured-output tasks with geometric supervision and spatial metrics, such as keypoint detection, dense captioning, or layout generation.","Because the teacher is frozen and sees the base distribution plus the overlay, the method is a form of on-policy self-distillation with a visual hint; it may combine naturally with verifiable rewards that directly optimize IoU.","The paper's own limitation—that the hint is a simple overlay and gains are moderate—points to richer privileged inputs, such as a cropped region or segmentation mask, as a way to reduce the remaining conditional mismatch."],"forward_implications":["Training-time-only privileged visual hints can be layered onto coordinate-generating multimodal LLMs without changing the inference input format or adding a localization head.","The gains increase under stricter overlap thresholds: P@0.9 rises 7.75 points and P@0.95 by 12.84 over the base model, indicating the method pushes many predictions into the high-overlap range.","The method helps small and medium objects more than large ones, suggesting privileged box hints are especially useful when coordinate errors cover a larger fraction of the target region.","Removing the SFT anchor sharply reduces the gain, so privileged distillation complements direct coordinate supervision rather than replacing it.","The approach requires grounding annotations at training time; without ground-truth boxes, the privileged teacher input cannot be constructed."],"fun_headline_variants":["Box hints, not just targets, boost grounding","IoU-PD: ground-truth boxes as training clues","Privileged box overlays sharpen MLLM grounding","Overlay hints lift visual grounding by 4 points"],"cache_read_input_tokens":19840,"weakest_assumption_plain":"The load-bearing premise is that the frozen teacher's distribution, computed on an input that literally contains the ground-truth box, remains a faithful distillation target for a student that will never see that box—that the conditional mismatch between the privileged and standard inputs is small enough that the teacher's logits teach region awareness rather than reliance on the green rectangle.","fun_headline_variants_meta":{"raw":{"variants":["Box hints, not just targets, boost grounding","IoU-PD: ground-truth boxes as training clues","Privileged box overlays sharpen MLLM grounding","Overlay hints lift visual grounding by 4 points"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00014,"raw_usage":{"total_tokens":991,"prompt_tokens":732,"completion_tokens":259,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":205}},"tokens_in":476,"tokens_out":259,"duration_ms":3384,"temperature":1.0,"reasoning_tokens":205,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T04:14:11.287755+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the same pipeline with a teacher whose green rectangle is placed at a random location (same size, same hint text); if the student's mIoU and Acc@0.7 gains stay close to the full IoU-PD numbers, the claim that ground-truth location information drives the improvement is falsified, whereas a drop to the original-teacher baseline would confirm that the privileged location is the active component.","supporting_citations":[],"review_version":2}