{"id":"e3b4f1c7-bdc7-4863-b3d8-107af8ba275d","arxiv_id":"2411.15766","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":10,"one_line_summary":"Train dual LLM towers for dense retrieval, then distill the query tower into a small BERT encoder, keeping most of the accuracy gain without the online latency.","lead":"ScalingNote is a two-stage training method that lets a 7B-parameter language model encode documents while keeping the query encoder small enough for real-time search. At Xiaohongshu it improved retrieval relevance and reduced irrelevant results with negligible added query latency.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Section 3.4's theoretical advantage claim rests on an unverified empirical-risk ordering R(teacher) ≤ R(ScalingDoc); without it, Proposition 3.2 does not establish a lower bound, leaving the theoretical support for ScalingNote conditional.","rationale":"The strongest empirical claim—ScalingNote beats ScalingDoc at equal online cost—is supported by Table 4 and the t-test asterisks; the offline numbers are consistent across dataset sizes and the QPS data in Table 6 is plausible. The weakest link is the theoretical framing: the paper lists 'we prove our method's effectiveness theoretically' as a contribution, and Section 3.4's conclusion depends on comparing R(teacher) with R(ScalingDoc). That ordering is not derived, not reported, and not independently established; the cited works address encoder scaling laws, and Figure 1 is the proposed method's own output. A concrete measurement of the two training risks and the distillation distance would settle whether the bound comparison is valid. This does not overturn the empirical system result, so the CONDITIONAL verdict stands.","tokens_in":22468,"tokens_out":8821,"duration_ms":82182,"concrete_test":"On the Large training set, compute the final value of the stage-I loss (Eq. 12) for the trained ScalingDual teacher and for ScalingDoc using identical hyperparameters and data, and compute the mean L2 distance ∥q_stu−q_teacher∥ between the BERT-4L student and the teacher over the training queries after QKD. If R(ScalingDual) > R(ScalingDoc) or the mean distance is not small relative to the risk gap, then the Section 3.4 comparison does not establish the claimed lower bound, and the theoretical claim should be downgraded to an empirical observation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's headline contribution includes a theoretical proof that ScalingNote has a lower generalization upper bound than ScalingDoc (Section 3.4). The proof of Proposition 3.2 is algebraically valid, but the comparative Analysis step requires R(s_Q_R,D_R; S_n) ≤ R(s_Q_D,D_D; S_n). This ordering is asserted via a citation to scaling-law works and Figure 1, which is the paper's own result, not independent evidence. Since ScalingDoc uses a 29M BERT query tower and ScalingDual uses a 7B LLM query tower, the ordering is plausible but not guaranteed: optimization, architecture mismatch (decoder-only vs BERT), and the 1-epoch training budget could invert it. If it fails, the stated inequality between the two upper bounds does not follow, and the theoretical explanation of ScalingNote's advantage collapses, even though the empirical offline results (Table 4) may still stand. The bound comparison also inherits looseness typical of covering-number bounds, so a lower upper bound does not by itself prove lower expected risk; but the empirical-risk ordering is the concrete, checkable weak link.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ScalingNote, a two-stage method for scaling up dense retrieval in an industrial setting (Xiaohongshu). Stage I trains a dual-tower retriever in which both towers are initialized from the same LLM (Qwen2.5-7B), using cross-device contrastive learning and hard-negative margin loss over multiple Matryoshka dimensions. Stage II (QKD) distills the LLM query tower into a small BERT-based online query tower using MSE plus cosine-similarity loss. The authors claim that ScalingNote achieves lower generalization error than ScalingDoc (which scales only the document tower) while preserving online query latency, and they support this with a covering-number generalization bound, scaling-law fits on the validation contrastive entropy, offline retrieval evaluations on three dataset sizes, and a week-long online A/B test.","tokens_in":22722,"tokens_out":3885,"duration_ms":37053,"significance":"If the claims hold, the paper is a useful contribution to industrial dense retrieval: it shows a practical recipe for obtaining large improvements in recall and AUC by scaling only the document tower with an LLM and then distilling the query tower, with a convincing online A/B test on real traffic. The paper's strengths are that the offline experiments are internally consistent, the comparisons control for query/document tower sizes, the datasets are at realistic industrial scale, and the online experiment is a strong piece of evidence. The scaling-law fits are also a valuable descriptive resource for practitioners. However, the theoretical advantage over ScalingDoc is not established by the presented proof, and the scaling laws are in-sample fits rather than predictive laws, so the two most prominent scientific claims are weaker than the empirical results themselves.","major_comments":[{"comment":"The claimed comparison between the generalization bounds of ScalingNote and ScalingDoc hinges entirely on the assertion R(s_Q_R,D_R; S_n) <= R(s_Q_D,D_D; S_n), which is justified only by citations [4,6,37] and by the paper's own Figure 1. Neither source demonstrates this ordering for the specific training setup used here (one epoch, cross-device in-batch negatives, hard margin loss, and MRL projections), and Figure 1 is a result of the same experiments rather than independent evidence. Because the query towers differ not only in size but also in architecture (decoder-only LLM vs. BERT) and optimization budget, the ordering is plausible but unverified. The authors should directly measure the empirical risks of the ScalingDoc tower pair and the ScalingNote stage-I teacher pair on the same training set, or they should reframe the theoretical section as conditional on this ordering. As written, Proposition 3.2 does not establish that ScalingNote has a lower generalization upper bound than ScalingDoc.","section":"Section 3.4, Analysis after Eq. (16)-(17)"},{"comment":"The scaling laws are fitted to validation contrastive entropy using the same Small/Median/Large datasets that are later used both for training the final models and for motivating the choice of 10^8 training pairs. The reported R^2 values are in-sample fit qualities; they do not demonstrate that the fitted laws predict unseen points. To support the claim that the paper 'verifies the scaling law of dense retrieval,' the authors should perform an out-of-sample check (e.g., fit on Small and Median, predict Large) and report prediction error. Additionally, the fitted constants, exponents, and the offset in Eq. (20) are reported without confidence intervals, which is especially problematic for the oddly large inner exponent of 770 in Eq. (20).","section":"Section 4.4, Eqs. (18)-(20) and Figure 5"},{"comment":"The key term that QKD is supposed to make small, namely (2K/n) * sum_i ||q_i^stu - q_i||, is exactly the quantity minimized by the QKD training objective (MSE plus negative cosine similarity). Consequently, the theoretical argument is circular in the sense that it proves the bound is good if the training objective is minimized, rather than providing an independent reason why the distilled student should perform well. Moreover, Table 6 shows a non-monotonic relationship between the training QKD loss and student size: RED BERT-4L has train loss 1.1e-4 while RED BERT-12L has train loss 6.0e-3, yet BERT-12L achieves better retrieval and validation metrics. The paper should explain this inconsistency, and the Lipschitz step (a) in Eq. (32) should state explicitly the boundedness assumption on the dot products q^T d that makes the softplus-lipschitz bound valid.","section":"Section 3.3 and Appendix B, Eq. (31)-(32)"},{"comment":"The abstract and conclusion say that ScalingNote 'outperforms end-to-end models,' but Table 4 shows the opposite: ScalingDual exceeds ScalingNote on every dataset and every recall cutoff (e.g., on Large, R@50 55.15 vs 54.57 and R@1k 90.62 vs 90.02). The accurate claim, which is still strong, is that ScalingNote approaches ScalingDual while maintaining much higher online QPS (33,810 vs 408 in Table 6). This wording should be corrected so that the reported results match the claims.","section":"Abstract and Conclusion vs. Table 4"}],"minor_comments":[{"comment":"The label 'Scaling/glyph1197ote' contains a rendering artifact; it should read 'ScalingNote'.","section":"Figure 1"},{"comment":"The y-axis label 'Crontrastive Entropy' is a typo for 'Contrastive Entropy.'","section":"Figure 5(c)"},{"comment":"There is a duplicated 'and' in 'we denote the batch of all-gathered query representations as Q, and and the batch'.","section":"Section 3.2, paragraph after Eq. (4)"},{"comment":"The paper states that results are averages of three tests but does not report standard deviations or confidence intervals. Adding mean +/- std for the key comparisons would make the reported statistical significance more informative.","section":"Tables 3, 4, and 6"},{"comment":"The mixed scaling law in Eq. (20) has a form L(N,D) = (( (3.15e5/N)^0.12 / 770 + 1.16e2/D )^770 + 0.108, which appears to contain a transcription error in the nested fractions; the intended functional form should be written more clearly and verified against the fitted curve in Figure 5(c).","section":"Section 4.4.4 and Eq. (20)"},{"comment":"The Stage II global batch size is listed as 40,000 with 100 epochs; it would help to clarify whether this is the number of query samples per step or the effective batch size after accumulation, since a batch of 40,000 queries is unusually large for a BERT-4L model.","section":"Appendix A, Table 7"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is genuinely valuable and appears to be from a real industrial deployment, which makes the paper attractive for the journal's audience. My main concern is that the theoretical section is currently oversold: the ordering of empirical risks is not proven, and the scaling laws are curve fits rather than predictions. If the authors can either provide the direct empirical-risk measurement supporting R(s_Q_R,D_R) <= R(s_Q_D,D_D) or substantially soften the theoretical claims, and correct the abstract's contradiction with Table 4, the paper would be a solid candidate for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: ScalingNote is a real system that works, and the offline/online evidence is the strongest part of the paper. The two-stage idea—train a full LLM dual tower, then distill only the query tower to a small BERT—is a sensible way to get most of the LLM-document-tower benefit at low query latency. The consistent wins over ScalingDoc across three data scales, plus the positive online A/B, make a credible case that this is useful in production. Credit where due: the prompts with predicting title/content query terms and then summarizing into [EMB] are a thoughtful response to the 'title-on-display' problem in Xiaohongshu; the MRL multi-dimension training is practical; and the scaling-law fits on industrial data are new measurements even if they are not deep.\n\nSoft spots: First, the theoretical analysis in Section 3.4 is the weakest link. Proposition 3.2's conclusion depends on R(s_Q_R,D_R) ≤ R(s_Q_D,D_D), which is asserted from prior work and their own Figure 1, not established. That is a plausible assumption, but it is exactly the thing being argued—if the 7B query tower trained on the same data does not achieve lower empirical risk than the 29M BERT tower, the bound comparison collapses. Second, the scaling laws are in-sample fits, then used to justify design choices like 10^8 training pairs. That is circular; they are summaries of observed points, not predictive laws. Third, no error bars in the main tables, no code or data, and no comparison to the query-distillation baselines [4,48] they cite. Those are fixable.\n\nIs the central claim still standing? Yes, the empirical part holds. The theorem is not load-bearing for the system result; it is an attempted explanation. Readers should treat the theory as suggestive, not proven.\n\nWho this is for: industry IR folks and anyone working on LLM-based dense retrieval or distillation. It is a solid engineering paper with real data. A serious editor should send it to peer review—I would recommend major revision, asking for the risk ordering to be verified empirically (or the theory reframed as motivation), error bars, and a comparison with the existing query-distillation methods. If those are addressed, it is a useful contribution for the community.","headline":"A substantive industrial scaling study with strong empirical results; the theory section is a loose end, not a fatal flaw.","tokens_in":23316,"tokens_out":2054,"would_cite":true,"duration_ms":19062,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By first training both retrieval towers as 7B LLMs and then distilling only the query tower's embeddings into a small BERT encoder, ScalingNote keeps nearly all of the full-LLM recall gain while holding online query latency at small-model…","keywords":["dense retrieval","dual-tower architecture","large language models","knowledge distillation","query encoder distillation","scaling law","industrial retrieval","contrastive learning"],"falsifier":"Reproduce both strategies from scratch on a public retrieval benchmark such as MS MARCO using the same 7B LLM and the same four-layer BERT query encoder, then check two things: whether the joint 7B/7B teacher's training-set contrastive loss is actually lower than the BERT-4L/7B model's (if not, the theorem's key assumption fails), and whether the QKD-distilled BERT-4L query tower beats the directly trained BERT-4L query tower on held-out recall at equal throughput; a violation of the loss ordering or no recall advantage for the distilled student would settle the claim.","tokens_in":116,"feed_emoji":"🔍","tokens_out":20986,"duration_ms":226517,"temperature":0.7,"pith_summary":"The paper's claim is that industrial dual-tower dense retrieval can have LLM-scale quality without LLM-scale online latency, because the query side — not the document side — is the real bottleneck. Its method, ScalingNote, first trains both towers from the same 7B LLM to unlock the models' retrieval capacity, then freezes the LLM document tower and distills only the query tower into a four-layer BERT student using a loss that combines mean-squared error and cosine similarity. On the platform's largest test set this closes almost the whole gap to the fully scaled dual-LLM system (R@50 54.57 versus 55.15) while beating document-only scaling (47.19) at identical online parameter sizes, with the student running at 33,810 queries per second versus 408 for a 7B query tower. The paper backs the recipe with a generalization-bound argument that the distilled system has a lower expected-risk upper bound than document-only scaling, and with fitted power-law scaling curves for model size and data size on real traffic. If the paper is right, the cost-effective way to deploy LLM-scale retrieval is to scale both towers at training time and then shrink the query side at serving time.","feed_headline":"Get LLM-scale retrieval quality at BERT query speed","feed_subtitle":"Scaling both towers as LLMs, then distilling only the query side, keeps ~99% of the recall gain at 83x query throughput.","key_machinery":"The central mechanism is query-based knowledge distillation (QKD), a loss $$L_{KD} = \\frac{1}{|Q|}\\sum_i \\left(\\|\\boldsymbol{q}_i - \\boldsymbol{q}^{stu}_i\\|$_2^{2}$ - \\$\\lambda$\\,\\mathrm{sim}(\\boldsymbol{q}_i, \\boldsymbol{q}^{stu}_i)\\right)$$ that makes a small student query tower reproduce the LLM teacher's query embeddings in both direction (cosine term) and magnitude (MSE term), keeping the student compatible with the large frozen document tower despite different architecture and vocabulary. The two-stage ordering is what makes this work: Stage I trains both towers from the same 7B LLM with cross-device contrastive learning and a hard-negative margin loss, using a document prompt that predicts title and content query terms separately and summarizes them at an [EMB] token; Stage II freezes the LLM document tower and distills only query embeddings, which is cheap because queries are short and information-light. The theoretical argument rides on a generalization bound (extending [20]) in which the distilled system's expected risk is at most the teacher's empirical risk plus a distillation-error term $\\frac{2K}{n}\\sum_i\\|\\boldsymbol{q}^{stu}_i - \\boldsymbol{q}_i\\|$ plus a complexity term that covers only the query tower, because the document tower is fixed; the bound is tighter than document-only scaling's whenever the larger teacher fits the training data at least as well.","core_discovery":"The central discovery is that in dual-tower retrieval the query tower is the binding constraint, and that constraint can be removed cheaply. Scaling only the document tower with an LLM (ScalingDoc) captures only part of the potential of scaling, because the small query tower limits what the large document tower can express; scaling both towers (ScalingDual) is the quality upper bound but is far too slow online. ScalingNote captures most of that upper bound with none of the latency cost through two stages: it trains both towers from the same 7B LLM with cross-device contrastive learning and a hard-negative margin loss, then freezes the LLM document tower and distills only the query tower's embeddings into a four-layer BERT student using mean-squared error plus cosine similarity. On the largest industrial test set the distilled model reaches R@50 of 54.57 and R@1k of 90.02, versus 55.15 and 90.62 for scaling both towers and 47.19 and 84.37 for scaling only the document tower, at a measured 33,810 queries per second instead of 408. The paper reads this as evidence that the scaling law of dense retrieval holds in real deployments and that a scale-then-distill schedule makes LLM-scale retrieval cost-effective.","pith_inferences":["The same scale-then-distill recipe should transfer to other asymmetric retrieval setups — image-to-text, user-to-item, or cross-modal search — wherever one tower must stay small and fast while the other can be large and offline; the paper only demonstrates it for text query-document pairs.","The proof's crux, that the fully scaled teacher fits training data better than the doc-only-scaled pair, is never measured directly; a sweep with progressively larger teachers would confirm or break the theoretical story, independently of the empirical wins.","The bound suggests a training tweak the authors do not test: weight QKD samples by teacher-student embedding disagreement, so the distillation loss concentrates on exactly the term the theory says matters, potentially closing the small residual gap to ScalingDual.","Because the mixed scaling law is fitted on one platform's traffic, reading it as a budgeting rule for other industrial teams is an extrapolation the paper does not claim; the fitted exponents would need re-estimation on each new corpus."],"forward_implications":["Scaling only the document tower leaves most of the LLM gain on the table: at matched online parameter counts and query speed, ScalingNote beats ScalingDoc by over seven points of R@50 on the large test set, so the small query tower, not the document tower, is the binding constraint.","The latency-accuracy trade-off is a smooth, tunable knob: distilling into 1L, 4L, and 12L BERT students yields R@50 of 49.56, 54.57, and 54.91 against ScalingDual's 55.15, with QPS ranging from 52,205 down to 19,090.","Dense retrieval obeys a power law on industrial data: validation contrastive entropy fits $L(N)=(3.82\\times10^5/N)^{0.14}+0.18$ in model size ($R^2=0.9999$) and an analogous curve in data size, so the returns from adding parameters are predictable.","Deployed online, the method measurably improves retrieval quality: a week-long A/B test on 5% of traffic cut the share of irrelevant documents in the top-4 by 1.165% and raised user satisfaction by 0.135%.","The theoretical bound implies the distilled student inherits the teacher's training-set quality rather than its own architecture's, so the online system's quality floor is set by Stage I, not by the student tower."],"supporting_citations":[{"why":"Supplies the scaling-law functional form and the empirical-risk intuition that larger models fit training data better, on which the theoretical ordering rests.","marker":"[6]"},{"why":"Provides the covering-number generalization-bound framework that Propositions 3.1 and 3.2 extend to the distilled setting.","marker":"[20]"},{"why":"Establishes post-training embedding alignment for asymmetric dual encoders, the direct antecedent of the QKD distillation loss.","marker":"[4]"},{"why":"Shows query-encoder distillation by embedding alignment works as a strong baseline, which QKD builds on and is compared against.","marker":"[48]"},{"why":"Supplies Matryoshka representation learning, the multi-dimension embedding mechanism used in both training stages for downstream flexibility.","marker":"[21]"},{"why":"Defines the TwinBERT small dual-tower baseline whose query-tower size and latency ScalingNote matches.","marker":"[29]"},{"why":"Provides the Qwen2.5 7B backbone used for both towers and for the model-size scaling-law sweep.","marker":"[45]"},{"why":"Cited to support the claim that large dual encoders generalize well, used as evidence for the empirical-risk ordering in the theory.","marker":"[37]"}],"fun_headline_variants":["Distill the query tower: LLM-scale recall at BERT speed","Scale both towers, then shrink the query side for speed","Dual LLM towers distilled to BERT: 83x faster, 99% recall","Most of LLM retrieval quality at 1/83 the query latency","Two-stage scaling: LLM dual-tower, then query distillation"],"cache_read_input_tokens":25344,"weakest_assumption_plain":"The argument only goes through if the 7B query tower trained together with the 7B document tower genuinely fits the training data at least as well as the tiny four-layer BERT query tower paired with that same 7B document tower; the paper assumes this ordering rather than establishing it on its own data, and if it fails, the bound no longer shows that ScalingNote beats ScalingDoc.","fun_headline_variants_meta":{"raw":{"variants":["Distill the query tower: LLM-scale recall at BERT speed","Scale both towers, then shrink the query side for speed","Dual LLM towers distilled to BERT: 83x faster, 99% recall","Most of LLM retrieval quality at 1/83 the query latency","Two-stage scaling: LLM dual-tower, then query distillation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000331,"raw_usage":{"total_tokens":1884,"prompt_tokens":1026,"completion_tokens":858,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":642,"completion_tokens_details":{"reasoning_tokens":761}},"tokens_in":642,"tokens_out":858,"duration_ms":6964,"temperature":1.0,"reasoning_tokens":761,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:55:21.673437+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce both strategies from scratch on a public retrieval benchmark such as MS MARCO using the same 7B LLM and the same four-layer BERT query encoder, then check two things: whether the joint 7B/7B teacher's training-set contrastive loss is actually lower than the BERT-4L/7B model's (if not, the theorem's key assumption fails), and whether the QKD-distilled BERT-4L query tower beats the directly trained BERT-4L query tower on held-out recall at equal throughput; a violation of the loss ordering or no recall advantage for the distilled student would settle the claim.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the scaling-law functional form and the empirical-risk intuition that larger models fit training data better, on which the theoretical ordering rests."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the covering-number generalization-bound framework that Propositions 3.1 and 3.2 extend to the distilled setting."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows query-encoder distillation by embedding alignment works as a strong baseline, which QKD builds on and is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Matryoshka representation learning, the multi-dimension embedding mechanism used in both training stages for downstream flexibility."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the TwinBERT small dual-tower baseline whose query-tower size and latency ScalingNote matches."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Cited to support the claim that large dual encoders generalize well, used as evidence for the empirical-risk ordering in the theory."}],"review_version":1}