{"id":"0949b43b-0591-47c1-8294-3731087fce45","arxiv_id":"2607.11465","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Score-only distillation with a row-centered all-pairs PairMSE objective lets 0.6B bi-encoders recover up to 50% of the base-to-teacher retrieval gap under matched protocols.","lead":"A compact dense retriever can copy a large teacher's ranking behavior from score vectors alone, without embeddings or hidden states. Under matched retrieval protocols this recovers up to half the quality gap while cutting query and document encoding cost by several times.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"All-8 panel gains may be driven by the three row-source datasets rather than true held-out transfer under the matched protocol.","rationale":"The reader correctly flags external-transfer fragility and scopes the claim to matched protocols; that is the right overall posture and keeps the verdict at CONDITIONAL. The more immediate load-bearing point inside the paper’s own evidence is whether the All-8 number itself already mixes adaptation on the row-source tasks with genuine held-out transfer. Because the manuscript never publishes that split, the strongest quantitative claim rests on an unexamined aggregate. The proposed check is cheap (same embeddings, same evaluator) and would either confirm the panel-wide story or show that even the matched-protocol recovery is narrower than advertised. No stronger internal inconsistency (loss derivation, controls, or cost numbers) appears; the concern is purely about the evidential support for the headline recovery figure. Hence the reader’s CONDITIONAL stands unchanged.","tokens_in":9888,"tokens_out":623,"duration_ms":30750,"concrete_test":"Recompute Table 1 (and the equal-fusion/hard-neg rows of Table 2) as two separate macros: (a) mean NDCG@10 over the three row-source corpora and (b) mean over the five held-out panel corpora, for frozen base vs. each distilled student. If the held-out-5 delta is near zero or negative while row-source-3 accounts for essentially all of the reported All-8 lift, the 25–50% gap-recovery claim on the panel weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (up to 50% base-to-teacher gap recovery on the fixed eight-task panel via score-only distillation) treats All-8 full-corpus macro NDCG@10 as evidence of compression under matched protocols that already separate row-source (SciFact/NFCorpus/FiQA) from held-out tasks (ArguAna/SciDocs/TREC-COVID/Webis-Touche2020/Quora). Tables 1–2 and Figure 2 report only the aggregate; no per-task or row-source vs. eval-only split is given, even though paired-bootstrap machinery for 1,271 row-source vs. 12,505 eval-only queries is mentioned. Section 4.3 already documents regressions on other BEIR-style tasks (HotpotQA/FEVER/Climate-FEVER). If the modest Qwen deltas (+0.019–0.022, fusion CI lower bound +0.002) and larger E5 deltas are concentrated on the three datasets that supplied the C32 rows, the panel-level recovery overstates generalization even inside the authors’ own matched setting. The label-only and MarginMSE controls rule out pure row exposure but do not isolate where the teacher-score signal lands.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper studies black-box score-only distillation for compact dense bi-encoders: a student is trained from teacher query–document score vectors over candidate rows (one ground-truth positive plus pipeline negatives), without teacher embeddings, hidden states, or shared embedding space. The main technical device is a row-centered residual MSE (Eq. 1), shown via a variance identity to be equivalent (up to a constant factor) to uniform all-pairs PairMSE (Eq. 2). On a fixed eight-task BEIR-style panel with full-corpus macro NDCG@10, Qwen3-0.6B and E5-large students improve over their frozen bases when distilled from Qwen3-8B, NV-Embed, or equal fusion targets; label-only CE on the same rows collapses below the base, and positive-negative MarginMSE trails the centered objective. The authors report recovery of roughly 25–50% of the base-to-teacher gap, encoder-side speedups (4.7× query / 9.7× document vs sequential teacher fusion), and mixed external transfer, scoping the claim to compression under matched retrieval protocols.","tokens_in":10265,"tokens_out":1320,"duration_ms":11623,"significance":"If the result holds under the authors’ matched-protocol scope, the work is a useful, carefully scoped contribution to retrieval serving: it shows that black-box embedding teachers can supervise compact bi-encoders via score vectors alone, without representation alignment, and that a memory-linear reformulation of all-pairs PairMSE is sufficient for that signal. Strengths include matched controls (label-only CE and MarginMSE on identical rows), two student families, explicit separation of hard-negative mining and fusion as extensions rather than the main protocol, and honest reporting of mixed external transfer and overlapping multi-seed intervals. The paper does not claim a new loss family or universal adaptation; it claims serving-cost compression of teacher rankings under matched protocols, which is a practically relevant and falsifiable claim for IR systems that already pay for large embedding teachers offline.","major_comments":[{"comment":"Tables 1–2 and Figure 2 report only All-8 full-corpus macro NDCG@10. The manuscript states that SciFact/NFCorpus/FiQA supply training rows while ArguAna/SciDocs/TREC-COVID/Webis-Touche2020/Quora are held out from row construction, and it mentions paired-bootstrap machinery over 1,271 row-source vs 12,505 eval-only queries, yet no row-source vs eval-only (or per-task) split of the panel is shown. Given modest Qwen deltas (+0.019–0.022; fusion CI lower bound +0.002) and Section 4.3’s documented regressions on other BEIR-style tasks, the central “up to 50% gap recovery on the fixed eight-task panel” claim needs that split to establish that gains are not concentrated on the three row-source datasets. Without it, the matched-protocol generalization claim is under-supported even inside the authors’ own design.","section":null},{"comment":"Section 4.1 and Table 1: for the primary Qwen student, multi-seed evidence is reported only for equal fusion (CI [0.454, 0.490]), which overlaps both single-teacher point estimates and leaves the lower bound near the frozen base (+0.002). Most loss controls in Table 2 are single-seed (seed=42). The headline recovery range in Figure 2 and the abstract therefore rests on limited seed coverage for the main student family. Either multi-seed CIs for the single-teacher and centered-MSE main rows, or a clear statement that Qwen gains are seed-sensitive and secondary to the stronger E5-large result, is needed for the load-bearing claim.","section":null}],"minor_comments":[{"comment":"Figure 2’s y-axis (“Recovered teacher gap %”) and the abstract’s “up to 50%” should state explicitly which teacher and which base define the gap for each plotted point; the E5-large vs Qwen points appear to use different teachers.","section":null},{"comment":"Section 3: briefly note that L_center is invariant to additive row offsets but not to multiplicative rescaling of scores; the unit-centered ablation in Table 2 partially addresses this but is not discussed in the text.","section":null},{"comment":"Section 4.4 / Figure 3: clarify that latency is encoder-only microbenchmarks at batch size 8 on MI210, not end-to-end retrieval with ANN/index load, so the 4.7×/9.7× and 10.8× figures are upper bounds on serving benefit.","section":null},{"comment":"Data-generation pipeline for negatives is referenced but not specified (how candidates are sampled, from which index, hardness). A short appendix paragraph would aid reproducibility.","section":null},{"comment":"Typos / polish: “check-ing” line break in §4; “Qwen equal” labels in figures could be expanded once for readers skimming.","section":null}],"recommendation":"major_revision","confidential_remarks":"The contribution is solid systems/IR work with honest scoping, but the missing row-source vs held-out panel split is the one issue that could change how the abstract is read. If the authors can show that eval-only tasks also move, this is closer to minor revision; if gains are almost entirely on SciFact/NFCorpus/FiQA, the claim should be narrowed further. Fit for a solid IR venue is reasonable; novelty is incremental (score-level KD + memory-linear PairMSE) rather than conceptual breakthrough."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is simple: black-box score vectors from large embedding teachers can move compact bi-encoders under a matched protocol, without teacher embeddings or shared spaces, and the authors measure the serving win cleanly (roughly 4.7× query / 9.7× document vs sequential fusion).\n\nWhat is actually new is not PairMSE itself—they correctly treat the centered objective as a memory-linear rewrite of uniform all-pairs PairMSE via the usual variance identity—but the restricted setting: independent embedding teachers, cosine scores only, student remains a bi-encoder. They do the controls that matter. Label-only CE on the same rows collapses below the frozen base (0.304 vs 0.452). Positive-negative MarginMSE trails full centered matching. Gains show up for both Qwen3-0.6B and E5-large. They also report the negative findings on fusion weighting, active acquisition, and external BEIR-style transfer instead of burying them. That honesty is real credit.\n\nThe soft spots are proportionate, not fatal. Novelty is incremental relative to MarginMSE, listwise KD, and RD-Suite. Multi-seed coverage is thin outside the main fusion/hard-neg rows, and code/data are not shipped. The stress-test concern has some bite: All-8 is reported only as macro, even though they mention row-source vs eval-only query counts and already document regressions outside the panel. Without a per-task or source/held-out split, it is hard to know how much of the modest Qwen deltas (+0.019–0.022) live on SciFact/NFCorpus/FiQA. That does not kill the matched-protocol claim—they already scope external transfer as mixed—but a split table would make the evidence sharper.\n\nMath is fine; citation pattern is appropriate; free parameters are ordinary training knobs. This is for people who already run large embedding teachers and care about encoder cost under a fixed retrieval protocol, not for anyone hunting a new ranking-loss family or universal adaptation.\n\nI would send it to peer review. It is careful enough and useful enough to deserve referee time, with a request for the missing split and fuller seeds. Worth engaging if you work on retrieval serving; skip if you only care about representation-alignment distillation.","headline":"Solid, carefully scoped score-only distillation for bi-encoders: real controls, real speedups, modest novelty, and one reporting gap that still needs a split table.","tokens_in":10853,"tokens_out":562,"would_cite":false,"duration_ms":6229,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"A compact retriever can learn a large teacher's ranking behavior from score vectors alone, recovering up to half the quality gap while running several times faster.","keywords":["score-only distillation","dense retrieval","black-box teachers","PairMSE","bi-encoder compression","knowledge distillation","hard-negative mining","BEIR"],"falsifier":"Train the same student on the same score rows and then measure full-corpus NDCG@10 on held-out tasks whose corpus, query type, or relevance convention differ from the row sources; if the distilled student falls to or below the frozen base while the teacher remains stronger, the claimed compression does not transfer.","tokens_in":10794,"feed_emoji":"📦","tokens_out":680,"duration_ms":6064,"temperature":0.7,"pith_summary":"Large embedding models produce better retrieval rankings, but serving them online is costly. This paper asks whether a small bi-encoder can copy a large teacher's ranking behavior when it sees only the teacher's scores over candidate lists—no hidden states, embeddings, or shared vocabulary. The student is trained on fixed-width rows of one ground-truth positive plus pipeline negatives, using a row-centered loss that matches all pairwise score differences efficiently. On a fixed eight-task panel, the protocol lifts both Qwen 0.6B and E5-large students above their frozen bases and recovers up to about half of the gap to the teachers, while the 0.6B model encodes queries and documents several times faster than online teacher fusion. External transfer is mixed, so the result is scoped to matched retrieval protocols rather than universal adaptation. A sympathetic reader cares because the method turns expensive black-box ranking into a cheaper served model without requiring architectural access to the teacher.","feed_headline":"Small retrievers learn big rankings from scores alone","feed_subtitle":"Score-vector distillation recovers up to half the teacher gap and runs several times faster under matched protocols.","key_machinery":"The row-centered score-vector objective: the squared L2 norm of the residual between student and teacher scores after subtracting the row mean. It is a memory-linear reformulation of uniform all-pairs PairMSE, matching all pairwise teacher preferences without building explicit pair tensors.","core_discovery":"Black-box teacher score vectors alone carry enough ranking signal to improve compact bi-encoders under matched retrieval protocols: on a fixed eight-task panel, score-vector distillation recovers up to 50% of the base-to-teacher gap for both Qwen 0.6B and E5-large students, outperforming label-only contrastive training and positive-negative MarginMSE on the same rows, while the distilled 0.6B student is 4.7× faster for queries and 9.7× faster for documents than sequential online teacher fusion.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Compact retrievers learn teacher ranks from score vectors alone","Score-only distillation recovers up to half the teacher gap","Black-box scores train small bi-encoders under matched protocols","0.6B student closes half base-to-teacher gap from scores only","Row-centered PairMSE turns teacher scores into compact retrieval"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That candidate rows built from ground-truth positives and pipeline negatives on a few source datasets, scored by black-box embedding teachers, supply ranking supervision that still improves full-corpus retrieval under the same protocol.","fun_headline_variants_meta":{"raw":{"variants":["Compact retrievers learn teacher ranks from score vectors alone","Score-only distillation recovers up to half the teacher gap","Black-box scores train small bi-encoders under matched protocols","0.6B student closes half base-to-teacher gap from scores only","Row-centered PairMSE turns teacher scores into compact retrieval"]},"model":"grok-4.5","effort":"low","cost_usd":0.003716,"raw_usage":{"total_tokens":1114,"prompt_tokens":742,"num_sources_used":0,"completion_tokens":89,"cost_in_usd_ticks":37160000,"prompt_tokens_details":{"text_tokens":742,"audio_tokens":0,"image_tokens":0,"cached_tokens":0},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":283,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":742,"tokens_out":89,"duration_ms":3637,"temperature":1.0,"reasoning_tokens":283,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T05:24:35.736188+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same student on the same score rows and then measure full-corpus NDCG@10 on held-out tasks whose corpus, query type, or relevance convention differ from the row sources; if the distilled student falls to or below the frozen base while the teacher remains stronger, the claimed compression does not transfer.","supporting_citations":[],"review_version":1}