{"id":"4e73ab2c-4ee4-4253-a596-ae5704a6fdac","arxiv_id":"2608.10636","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A 524M visual document retriever distilled from both sides of an 8B teacher reaches 86.9% of the teacher's NDCG@5 and beats every reproduced sub-1B baseline.","lead":"DistilVDR is a 524M-parameter document search system that learns to copy the embeddings of a much larger 8B model, without needing relevance labels at student training time. It retrieves document pages at a fraction of the storage and compute cost while keeping most of the large model's search quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing from-scratch contrastive control leaves the central attribution to distillation unproven","rationale":"Both the reader and I converge on a conditional verdict, but for slightly different reasons. The reader's declared weakest_assumption is that the teacher's cosine space sufficiently orders documents and that independent pointwise regression preserves cross-modal calibration. That assumption is partially tested by Table 2 (gap decomposition) and Section 5.1 (contrastive coupling is flat), and the empirical retrieval numbers provide direct evidence that the learned geometry orders documents well, so this is not the most decisive weakness. The more load-bearing gap is the missing from-scratch contrastive control: without it, the paper cannot support the causal statement that distillation, rather than architecture plus data plus compute, yields the reported quality. The paper explicitly acknowledges this in its Limitations section, so the objection is grounded in the manuscript itself rather than an external requirement. The two secondary issues noted by the reader (the code repository link and the 'order of magnitude faster than every multi-vector baseline' phrasing versus Table 3) are real but would not change the verdict: the central comparison in the abstract is against the strongest sub-1B multi-vector baseline, which holds. The proposed concrete test is a single, well-specified experiment with a clear decision rule; if it runs and the gap remains large, the conditional verdict can be upgraded to acceptance, and if the gap collapses, the paper's contribution needs reframing. This supports leaving the reader's CONDITIONAL verdict unchanged.","tokens_in":18321,"tokens_out":6285,"duration_ms":58577,"concrete_test":"Train the exact DistilVDR-HiRes architecture (InternViT-300M + ModernBERT-base document tower, DistilBERT-base query tower, same dynamic tiling, same 1.20M document images and 1.49M queries, same optimizer, batch size, and epoch budget) from scratch using a contrastive objective such as InfoNCE with in-batch hard negatives, without any teacher embeddings, and evaluate on ViDoRe v1+v2+v3 under the identical protocol. If this contrastive-from-scratch model scores within 1-2 NDCG@5 of DistilVDR-HiRes (i.e., roughly 60 or above), distillation is not load-bearing for the headline quality; if it falls 5 or more points short (i.e., at or below roughly 57), the distillation objective is required for the reported result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that pointwise cosine distillation from an 8B teacher produces a 524M single-vector retriever that leads all reproduced sub-1B baselines. The paper never tests whether distillation is actually responsible for this quality. Its own Limitations section states: 'The control that would isolate distillation itself, the same 524M architecture trained contrastively from scratch on the same mixture with hard-negative mining and a comparable compute budget, is the principal experiment we did not run.' Section 5.1 only adds a contrastive term on top of an already-distilled checkpoint, so it cannot separate the contribution of distillation from that of the architecture, data, and compute budget. If the same architecture trained contrastively from scratch reaches comparable NDCG@5, then the proposed 'via Dual-Student Distillation' method is not established as the reason for the reported quality; the improvement over BiModernVBERT would instead reflect the stronger visual backbone, the tiling scheme, and the larger training mixture. This concern is load-bearing because the paper's claimed gap is that prior from-scratch single-vector models 'struggle,' and the paper's own numbers show a 12.3-point gap between BiModernVBERT and HiRes. Without the control, one cannot tell whether the distillation objective or the better recipe closes that gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DistilVDR, a 524M-parameter end-to-end visual document retriever obtained by distilling both the query and document encoders from a frozen 8B vision-language teacher (Qwen3-VL-Embedding-8B) under a pointwise cosine alignment loss. The student is asymmetric: a 70M text-only query encoder and a 454M document encoder that combines InternViT-300M, a dynamic tiling scheme, and a ModernBERT-base backbone. Two release variants differ only in the visual tile budget: HiRes (six tiles plus thumbnail) and Fast (two tiles plus thumbnail). The paper reports NDCG@5 on the 22-dataset ViDoRe v1+v2+v3 suite, with HiRes at 61.74 average (86.9% of the teacher) and Fast at 59.98, both exceeding every reproduced sub-1B baseline and both offering large savings in index footprint and indexing time relative to multi-vector baselines. The paper also includes a gap decomposition (teacher/student combinations), ablations over tile budget, data scale, output dimension, query backbone, and a contrastive-supervision refinement study, and it candidly lists several limitations, including the absence of a from-scratch contrastive control.","tokens_in":18550,"tokens_out":5681,"duration_ms":51707,"significance":"If the central claims hold, DistilVDR is a practically valuable system: it shows that a compact single-vector retriever can come close to an 8B teacher while reducing index storage by roughly 15.6× and indexing substantially faster than multi-vector baselines. The paper's strengths are its unified reproduction of twelve released retrievers under one evaluation and profiling protocol, its detailed gap decomposition, its careful reporting of training-data provenance and teacher precomputation cost, and its unusually honest limitations section. The release of code and the inclusion of a conservative attention-backend reference further support reproducibility. However, the causal attribution of the quality gain to distillation itself is not tested, and one efficiency claim is overstated; these issues require revision before the results can be fully accepted.","major_comments":[{"comment":"The paper's central claim, reflected in its title and framing, is that pointwise cosine distillation is what enables a 524M single-vector retriever to lead all reproduced sub-1B baselines. The Limitations section states: 'The control that would isolate distillation itself, the same 524M architecture trained contrastively from scratch on the same mixture with hard-negative mining and a comparable compute budget, is the principal experiment we did not run.' Section 5.1 starts from an already-distilled checkpoint and adds a contrastive term, so it cannot separate the contribution of the distillation objective from that of the architecture (InternViT-300M, tiling, ModernBERT), the 1.20M-image training mixture, and the compute budget. Given that the paper explicitly contrasts its route with from-scratch small models that 'struggle,' this missing control is load-bearing for the attribution. I recommend adding the from-scratch contrastive training of the same architecture on the same mixture, or, if that is infeasible, re-framing the claims to describe the full recipe rather than the distillation objective alone.","section":"§5.1 and Limitations"},{"comment":"The text claims that DistilVDR-Fast is 'an order of magnitude faster than every multi-vector baseline at any scale.' Table 3 reports SauerkrautLM-ColLFM2-450M, a multi-vector baseline, at 19.02 docs/sec, versus 99.04 docs/sec for DistilVDR-Fast; this is a speedup of approximately 5.2×, which is not an order of magnitude. The comparison is indeed >10× for the other sub-1B multi-vector baselines (colSmol-256M, colSmol-500M, ColModernVBERT), so the claim should be narrowed accordingly or reworded to 'several times faster than the fastest multi-vector baseline and an order of magnitude faster than the slower ones.' This overstatement appears in the abstract, Section 4.3, and the conclusion, so it should be corrected in all three places.","section":"§4.3 and Abstract"}],"minor_comments":[{"comment":"The paper reports single-run NDCG@5 values without error bars or significance tests. Given that some reported differences are small (e.g., Fast versus HiRes on v1, 81.34 versus 82.81), providing multiple seeds or a variance estimate would strengthen the headline comparisons.","section":"§4.2 / Table 1"},{"comment":"The statement that 'HiRes leads every reproduced sub-1B baseline on the high-resolution-sensitive v3 benchmark' is accurate, but the 13.55-point margin over colSmol-500M should perhaps be contextualized by noting that the v3 benchmark is recent and the reproduced baselines may not have been trained or tuned on comparable v3-like professional-domain data.","section":"§4.2"},{"comment":"The conservative attention-backend reference (eager attention) is a good transparency addition, but the main text would benefit from a sentence noting that even with eager attention, DistilVDR-Fast remains faster than every sub-1B multi-vector baseline, as the appendix reports.","section":"Appendix G"},{"comment":"The teacher precomputation cost is reported in H200-GPU-hours, but the §4.1 sentence describing the three sharded batches is dense and would be clearer if the per-part hours were tied to the final deduplicated image counts in the same sentence.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid systems contribution with unusually candid limitations, but the central attribution to distillation is not yet proven because the from-scratch contrastive control is explicitly missing. The efficiency overstatement in the throughput claim is a factual error that needs correction. The missing control is feasible within the paper's scope (the authors already have the data and recipe), so major revision rather than rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I'll cut to the chase: this is a solid systems paper with an unusually honest limitations section, and its value is the measured operating point, not a new theory of distillation. DistilVDR - a 524M asymmetric, encoder-only pair - gets 61.74 NDCG@5 on ViDoRe v1+v2+v3, about 87% of an 8B teacher, with an index 15.6x smaller than the strongest sub-1B multi-vector baseline and much higher indexing throughput. If the numbers hold, it's a genuinely deployable option.\n\nWhat's actually new: NanoVDR distilled only the query side and kept a 2B teacher in the document path. DistilVDR distills both sides from one frozen teacher, discards the teacher, and pairs a heavy document tower with a 70M text-only query tower. The paper also reproduces twelve released retrievers under one evaluation and profiling pipeline, which is real work. The gap decomposition in Table 2 is informative, and Section 5.1's contrastive-refinement ablation is honest: adding contrastive supervision on top of the distilled checkpoint is flat or mildly negative.\n\nThe soft spots, in proportion. First and biggest: there is no from-scratch contrastive control. The authors state in Limitations that 'the control that would isolate distillation itself... is the principal experiment we did not run,' and Section 5.1 starts from an already-distilled checkpoint, so it cannot separate the distillation objective from the stronger encoder, tiling, and data mixture. The title says 'via Dual-Student Distillation,' but that attribution is not established. This is a required follow-up, not a fatal flaw - the practical claim about the resulting system stands regardless, but the paper should not claim to have isolated the mechanism.\n\nSecond, the 'order of magnitude faster' phrasing is loose. Fast does 99 docs/sec versus 2.8 for colSmol-500M, but SauerkrautLM does 19 docs/sec, so 'faster than every multi-vector baseline at any scale' is not strictly 10x. Minor.\n\nThird, the code link points to the NanoVDR repo, not DistilVDR. That has to be fixed, and code plus cached targets should be released.\n\nFourth, no significance testing on the NDCG@5 gaps, which is common for ViDoRe but worth noting.\n\nFor a reader: people working on small VDR models or document-retrieval deployment will get value. I'd send it to review; ask the authors to run or clearly scope the from-scratch control, fix the repo link, and correct the throughput claim. Conditional accept, not reject.","headline":"A well-measured 524M VDR system with an unusually honest limitations section: useful as a deployable operating point, but the title's attribution to distillation is not yet proven because the from-scratch contrastive control was not run.","tokens_in":19079,"tokens_out":3768,"would_cite":true,"duration_ms":32108,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 524M-parameter retriever distills an 8B visual-language teacher into a single vector per page, retaining 86.9% of its ViDoRe NDCG@5 while indexing 15.6 times smaller.","keywords":["visual document retrieval","dual-student distillation","single-vector retriever","cosine alignment loss","asymmetric encoder","ViDoRe benchmark","model compression","embedding distillation"],"falsifier":"Train the identical 524M encoder-only architecture from scratch on the same 1.20M-image mixture with hard-negative mining and contrastive learning under a comparable compute budget; if that model matches or exceeds DistilVDR's 61.74 average NDCG@5, then the distillation objective itself is not the cause of the result. Alternatively, evaluate DistilVDR and the teacher on a corpus where the teacher's cosine ranking is known to be miscalibrated across modalities; if the student does not reproduce that miscalibration, the cosine-space assumption is doing less work than claimed.","tokens_in":18134,"feed_emoji":"🔍","tokens_out":7734,"duration_ms":63689,"temperature":0.7,"pith_summary":"DistilVDR sets out to show that a 524M-parameter, single-vector visual document retriever can inherit most of the retrieval quality of an 8B vision-language model by distilling both the query and document encoders onto that teacher's cached embedding space. The training objective is a pointwise cosine alignment loss, so no relevance labels, negative sampling, or contrastive term is needed at student-training time. On the 22-dataset ViDoRe suite the paper reports 61.74 average NDCG@5 for the high-resolution variant, 86.9% of the teacher's score and ahead of every reproduced sub-1B baseline; the compact variant scores 59.98 with a three-times-smaller visual-token budget. Both variants store one million documents in a 15.6-times-smaller index than the strongest sub-1B multi-vector baseline and index the corpus an order of magnitude faster. If true, the work turns a multi-billion-parameter visual retriever into a deployable 524M system that keeps most of the quality at a fraction of the index, memory, and latency cost.","feed_headline":"Small retriever keeps 87% of 8B teacher's quality","feed_subtitle":"One vector per page indexes 15.6x smaller and an order of magnitude faster than sub-1B multi-vector baselines.","key_machinery":"The load-bearing mechanism is dual-student distillation under a pointwise cosine alignment loss. For a document image d and a query q, the frozen 8B teacher produces cached targets T(d) and T(π◦q); each student minimizes Ld = 1 − ⟨fd(d), T(d)⟩ and Lq = 1 − ⟨fq(π◦q), T(π◦q)⟩ over L2-normalized vectors, with the two students trained independently and never sharing a forward pass. The document tower encodes dynamic tiles (up to six 448×448 tiles plus a thumbnail) with a 300M-parameter vision encoder, re-encodes the projected visual tokens with a 150M-parameter bidirectional text backbone, mean-pools, and projects to 4096 dimensions; the query tower is a 70M text encoder with the teacher's instruction prefix. Retrieval is a single dot product between the student vectors, and the teacher is discarded after training.","core_discovery":"The paper's central claim is that a single-vector dense retriever at 524M parameters, trained by independently regressing each tower onto the frozen teacher's 4096-dimensional L2-normalized vectors, reproduces enough of the teacher's relevance geometry to dominate every reproduced sub-1B visual document retriever and to retain 86.9% of an 8B teacher's average NDCG@5. The authors argue the teacher's embedding space is a complete supervision signal because it was itself trained with relevance supervision, so the student objective can be purely geometric. They further claim the asymmetric encoder-only student, with 454M parameters on the document side and 70M on the query side, matches the text-query/image-document asymmetry of the task and that varying only the visual-tile budget yields two deployment points (HiRes and Fast) that trade 1.76 average NDCG@5 for a three-times-smaller visual-token budget. The gap decomposition shows the residual distance to the teacher is not additive across sides, and that the single-vector format itself is not the bottleneck since the teacher at 71.05 NDCG@5 is level with 4.4B multi-vector models.","pith_inferences":["The flat contrastive-refinement result suggests that once a teacher's geometry is matched pointwise, explicit cross-modal coupling adds little; but the paper does not run the from-scratch contrastive control, so the claim that distillation beats contrastive training for this architecture remains untested.","Since teacher targets are cached once, sweeping teacher scale or family is cheap; a stronger single-vector teacher would likely lift both variants proportionally, and the same recipe could be dropped onto future embedding models without retraining the pipeline.","The fixed per-page tile budget ignores content complexity; a page-complexity estimate that allocates tiles to dense small-text pages could close part of the remaining v3 gap, which is concentrated exactly on such pages.","The 768-d Matryoshka prefix already costs only 3.19 average NDCG@5 while cutting the index 5.3x; combining it with scalar or product quantization, which the paper leaves untested, could make even the compact variant fit in a few GB per million pages."],"forward_implications":["Single-vector visual document retrieval at the 500M scale can retain roughly 85–87% of an 8B teacher's ViDoRe NDCG@5, closing much of the quality gap that previously forced multi-vector late interaction.","Deployments that need one dense vector per page get a 15.6-times-smaller index and roughly two-orders-of-magnitude-lower scoring latency than sub-1B multi-vector baselines, and indexing throughput rises by an order of magnitude.","Because the student objective needs no relevance labels, negatives, or contrastive term, the two towers can be trained independently and in parallel against cached teacher targets, and the teacher runs only once.","The visual-tile budget is a direct quality-versus-cost control: halving the tile budget from six to two tiles costs 1.76 average NDCG@5 while tripling throughput, which gives system builders a single knob for deployment.","The student is bounded by the teacher's own embedding quality: no side of the student can exceed the teacher on the side it replaces, so improving the teacher, or distilling from a multi-vector or cross-encoder teacher, is the path beyond the measured ceiling."],"supporting_citations":[{"why":"It provides the frozen 8B teacher and its 4096-d embedding space, which serve as the distillation target.","marker":"(Li et al., 2026a)"},{"why":"It establishes the pointwise cosine distillation paradigm for VDR and supplies the 1.49M-query training set and base image mixture.","marker":"(Liu et al., 2026)"},{"why":"It defines the ViDoRe v1 benchmark, the ColPali multi-vector baseline, and the ColPali training set used in the mixture.","marker":"(Faysse et al., 2025)"},{"why":"It adds the ViDoRe v2 evaluation datasets with multilingual queries.","marker":"(Macé et al., 2025)"},{"why":"It adds ViDoRe v3, the high-resolution-sensitive benchmark on which DistilVDR-HiRes leads sub-1B models.","marker":"(Loison et al., 2026)"},{"why":"It provides the encoder-only design, the bidirectional text-backbone idea, and the single-vector BiModernVBERT baseline.","marker":"(Teiletche et al., 2025)"},{"why":"It supplies the InternVL-V2 dynamic tiling rule used to cap the visual-token budget.","marker":"(Chen et al., 2025)"},{"why":"It supplies ModernBERT-base, the 150M bidirectional text backbone used over visual tokens.","marker":"(Warner et al., 2025)"}],"fun_headline_variants":["524M retriever keeps 87% of 8B teacher's NDCG","Single-vector VDR: 15.6x smaller index, 10x faster","DistilVDR hits 87% of 8B, beats all sub-1B baselines","Asymmetric student: 70M query, 454M doc, 87% of 8B","Two variants, same encoders: trade 1.76 NDCG for 3x tokens"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The student only learns to reproduce the teacher's per-item embedding directions, so if the teacher's cosine similarities do not order documents by relevance, or if matching each side separately throws away how query and document embeddings calibrate with each other, the student cannot recover teacher-level retrieval quality.","fun_headline_variants_meta":{"raw":{"variants":["524M retriever keeps 87% of 8B teacher's NDCG","Single-vector VDR: 15.6x smaller index, 10x faster","DistilVDR hits 87% of 8B, beats all sub-1B baselines","Asymmetric student: 70M query, 454M doc, 87% of 8B","Two variants, same encoders: trade 1.76 NDCG for 3x tokens"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000298,"raw_usage":{"total_tokens":1811,"prompt_tokens":1120,"completion_tokens":691,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":736,"completion_tokens_details":{"reasoning_tokens":567}},"tokens_in":736,"tokens_out":691,"duration_ms":6917,"temperature":1.0,"reasoning_tokens":567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T20:22:34.529754+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the identical 524M encoder-only architecture from scratch on the same 1.20M-image mixture with hard-negative mining and contrastive learning under a comparable compute budget; if that model matches or exceeds DistilVDR's 61.74 average NDCG@5, then the distillation objective itself is not the cause of the result. Alternatively, evaluate DistilVDR and the teacher on a corpus where the teacher's cosine ranking is known to be miscalibrated across modalities; if the student does not reproduce that miscalibration, the cosine-space assumption is doing less work than claimed.","supporting_citations":[],"review_version":1}