Pith. sign in

REVIEW 4 major objections 6 minor 62 references

ScalingNote: Scaling up Retrievers with Large Language Models for Real-World Dense Retrieval

T0 review · 4 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read By first training both retrieval towers as 7B LLMs and then distilling only the query tower's embeddings into a small BERT encoder, ScalingNote keeps nearly all of the full-LLM recall gain while holding online query latency at small-model…

desk verdict A substantive industrial scaling study with strong empirical results; the theory section is a loose end, not a fatal flaw. read the letter →

arxiv 2411.15766 v1 pith:KZIJNJOI submitted 2024-11-24 cs.IR

Suyuan Huang , Chao Zhang , Yuanyuan Wu , Haoxin Zhang , Yuan Wang , Maolin Wang , Shaosheng Cao , Tong Xu
show 7 more authors
Xiangyu Zhao Zengchang Qin Yan Gao Yunhan Bai Jun Fan Yao Hu Enhong Chen
This is my paper · ORCID
classification cs.IR
keywords denseretrievaldual-towerarchitecturelargelanguagemodelsknowledgedistillationqueryencoderscalinglawindustrialcontrastivelearning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's claim is that industrial dual-tower dense retrieval can have LLM-scale quality without LLM-scale online latency, because the query side — not the document side — is the real bottleneck. Its method, ScalingNote, first trains both towers from the same 7B LLM to unlock the models' retrieval capacity, then freezes the LLM document tower and distills only the query tower into a four-layer BERT student using a loss that combines mean-squared error and cosine similarity. On the platform's largest test set this closes almost the whole gap to the fully scaled dual-LLM system (R@50 54.57 versus 55.15) while beating document-only scaling (47.19) at identical online parameter sizes, with the student running at 33,810 queries per second versus 408 for a 7B query tower. The paper backs the recipe with a generalization-bound argument that the distilled system has a lower expected-risk upper bound than document-only scaling, and with fitted power-law scaling curves for model size and data size on real traffic. If the paper is right, the cost-effective way to deploy LLM-scale retrieval is to scale both towers at training time and then shrink the query side at serving time.

What carries the argument

The central mechanism is query-based knowledge distillation (QKD), a loss $$L_{KD} = \frac{1}{|Q|}\sum_i \left(\|\boldsymbol{q}_i - \boldsymbol{q}^{stu}_i\|$_2^{2}$ - \$\lambda$\,\mathrm{sim}(\boldsymbol{q}_i, \boldsymbol{q}^{stu}_i)\right)$$ that makes a small student query tower reproduce the LLM teacher's query embeddings in both direction (cosine term) and magnitude (MSE term), keeping the student compatible with the large frozen document tower despite different architecture and vocabulary. The two-stage ordering is what makes this work: Stage I trains both towers from the same 7B LLM with cross-device contrastive learning and a hard-negative margin loss, using a document prompt that predicts title and content query terms separately and summarizes them at an [EMB] token; Stage II freezes the LLM document tower and distills only query embeddings, which is cheap because queries are short and information-light. The theoretical argument rides on a generalization bound (extending [20]) in which the distilled system's expected risk is at most the teacher's empirical risk plus a distillation-error term $\frac{2K}{n}\sum_i\|\boldsymbol{q}^{stu}_i - \boldsymbol{q}_i\|$ plus a complexity term that covers only the query tower, because the document tower is fixed; the bound is tighter than document-only scaling's whenever the larger teacher fits the training data at least as well.

What would settle it

Reproduce both strategies from scratch on a public retrieval benchmark such as MS MARCO using the same 7B LLM and the same four-layer BERT query encoder, then check two things: whether the joint 7B/7B teacher's training-set contrastive loss is actually lower than the BERT-4L/7B model's (if not, the theorem's key assumption fails), and whether the QKD-distilled BERT-4L query tower beats the directly trained BERT-4L query tower on held-out recall at equal throughput; a violation of the loss ordering or no recall advantage for the distilled student would settle the claim.

Watch

Extended reading notes

Core claim

The central discovery is that in dual-tower retrieval the query tower is the binding constraint, and that constraint can be removed cheaply. Scaling only the document tower with an LLM (ScalingDoc) captures only part of the potential of scaling, because the small query tower limits what the large document tower can express; scaling both towers (ScalingDual) is the quality upper bound but is far too slow online. ScalingNote captures most of that upper bound with none of the latency cost through two stages: it trains both towers from the same 7B LLM with cross-device contrastive learning and a hard-negative margin loss, then freezes the LLM document tower and distills only the query tower's embeddings into a four-layer BERT student using mean-squared error plus cosine similarity. On the largest industrial test set the distilled model reaches R@50 of 54.57 and R@1k of 90.02, versus 55.15 and 90.62 for scaling both towers and 47.19 and 84.37 for scaling only the document tower, at a measured 33,810 queries per second instead of 408. The paper reads this as evidence that the scaling law of dense retrieval holds in real deployments and that a scale-then-distill schedule makes LLM-scale retrieval cost-effective.

Load-bearing premise

The argument only goes through if the 7B query tower trained together with the 7B document tower genuinely fits the training data at least as well as the tiny four-layer BERT query tower paired with that same 7B document tower; the paper assumes this ordering rather than establishing it on its own data, and if it fails, the bound no longer shows that ScalingNote beats ScalingDoc.

Editorial extensions

If this is right

  • Scaling only the document tower leaves most of the LLM gain on the table: at matched online parameter counts and query speed, ScalingNote beats ScalingDoc by over seven points of R@50 on the large test set, so the small query tower, not the document tower, is the binding constraint.
  • The latency-accuracy trade-off is a smooth, tunable knob: distilling into 1L, 4L, and 12L BERT students yields R@50 of 49.56, 54.57, and 54.91 against ScalingDual's 55.15, with QPS ranging from 52,205 down to 19,090.
  • Dense retrieval obeys a power law on industrial data: validation contrastive entropy fits $L(N)=(3.82\times10^5/N)^{0.14}+0.18$ in model size ($R^2=0.9999$) and an analogous curve in data size, so the returns from adding parameters are predictable.
  • Deployed online, the method measurably improves retrieval quality: a week-long A/B test on 5% of traffic cut the share of irrelevant documents in the top-4 by 1.165% and raised user satisfaction by 0.135%.
  • The theoretical bound implies the distilled student inherits the teacher's training-set quality rather than its own architecture's, so the online system's quality floor is set by Stage I, not by the student tower.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same scale-then-distill recipe should transfer to other asymmetric retrieval setups — image-to-text, user-to-item, or cross-modal search — wherever one tower must stay small and fast while the other can be large and offline; the paper only demonstrates it for text query-document pairs.
  • The proof's crux, that the fully scaled teacher fits training data better than the doc-only-scaled pair, is never measured directly; a sweep with progressively larger teachers would confirm or break the theoretical story, independently of the empirical wins.
  • The bound suggests a training tweak the authors do not test: weight QKD samples by teacher-student embedding disagreement, so the distillation loss concentrates on exactly the term the theory says matters, potentially closing the small residual gap to ScalingDual.
  • Because the mixed scaling law is fitted on one platform's traffic, reading it as a budgeting rule for other industrial teams is an extrapolation the paper does not claim; the fitted exponents would need re-estimation on each new corpus.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ScalingNote, a two-stage method for scaling up dense retrieval in an industrial setting (Xiaohongshu). Stage I trains a dual-tower retriever in which both towers are initialized from the same LLM (Qwen2.5-7B), using cross-device contrastive learning and hard-negative margin loss over multiple Matryoshka dimensions. Stage II (QKD) distills the LLM query tower into a small BERT-based online query tower using MSE plus cosine-similarity loss. The authors claim that ScalingNote achieves lower generalization error than ScalingDoc (which scales only the document tower) while preserving online query latency, and they support this with a covering-number generalization bound, scaling-law fits on the validation contrastive entropy, offline retrieval evaluations on three dataset sizes, and a week-long online A/B test.

Significance. If the claims hold, the paper is a useful contribution to industrial dense retrieval: it shows a practical recipe for obtaining large improvements in recall and AUC by scaling only the document tower with an LLM and then distilling the query tower, with a convincing online A/B test on real traffic. The paper's strengths are that the offline experiments are internally consistent, the comparisons control for query/document tower sizes, the datasets are at realistic industrial scale, and the online experiment is a strong piece of evidence. The scaling-law fits are also a valuable descriptive resource for practitioners. However, the theoretical advantage over ScalingDoc is not established by the presented proof, and the scaling laws are in-sample fits rather than predictive laws, so the two most prominent scientific claims are weaker than the empirical results themselves.

major comments (4)
  1. [Section 3.4, Analysis after Eq. (16)-(17)] The claimed comparison between the generalization bounds of ScalingNote and ScalingDoc hinges entirely on the assertion R(s_Q_R,D_R; S_n) <= R(s_Q_D,D_D; S_n), which is justified only by citations [4,6,37] and by the paper's own Figure 1. Neither source demonstrates this ordering for the specific training setup used here (one epoch, cross-device in-batch negatives, hard margin loss, and MRL projections), and Figure 1 is a result of the same experiments rather than independent evidence. Because the query towers differ not only in size but also in architecture (decoder-only LLM vs. BERT) and optimization budget, the ordering is plausible but unverified. The authors should directly measure the empirical risks of the ScalingDoc tower pair and the ScalingNote stage-I teacher pair on the same training set, or they should reframe the theoretical section as conditional on this ordering. As written, Proposition 3.2 does not establish that ScalingNote has a lower generalization upper bound than ScalingDoc.
  2. [Section 4.4, Eqs. (18)-(20) and Figure 5] The scaling laws are fitted to validation contrastive entropy using the same Small/Median/Large datasets that are later used both for training the final models and for motivating the choice of 10^8 training pairs. The reported R^2 values are in-sample fit qualities; they do not demonstrate that the fitted laws predict unseen points. To support the claim that the paper 'verifies the scaling law of dense retrieval,' the authors should perform an out-of-sample check (e.g., fit on Small and Median, predict Large) and report prediction error. Additionally, the fitted constants, exponents, and the offset in Eq. (20) are reported without confidence intervals, which is especially problematic for the oddly large inner exponent of 770 in Eq. (20).
  3. [Section 3.3 and Appendix B, Eq. (31)-(32)] The key term that QKD is supposed to make small, namely (2K/n) * sum_i ||q_i^stu - q_i||, is exactly the quantity minimized by the QKD training objective (MSE plus negative cosine similarity). Consequently, the theoretical argument is circular in the sense that it proves the bound is good if the training objective is minimized, rather than providing an independent reason why the distilled student should perform well. Moreover, Table 6 shows a non-monotonic relationship between the training QKD loss and student size: RED BERT-4L has train loss 1.1e-4 while RED BERT-12L has train loss 6.0e-3, yet BERT-12L achieves better retrieval and validation metrics. The paper should explain this inconsistency, and the Lipschitz step (a) in Eq. (32) should state explicitly the boundedness assumption on the dot products q^T d that makes the softplus-lipschitz bound valid.
  4. [Abstract and Conclusion vs. Table 4] The abstract and conclusion say that ScalingNote 'outperforms end-to-end models,' but Table 4 shows the opposite: ScalingDual exceeds ScalingNote on every dataset and every recall cutoff (e.g., on Large, R@50 55.15 vs 54.57 and R@1k 90.62 vs 90.02). The accurate claim, which is still strong, is that ScalingNote approaches ScalingDual while maintaining much higher online QPS (33,810 vs 408 in Table 6). This wording should be corrected so that the reported results match the claims.
minor comments (6)
  1. [Figure 1] The label 'Scaling/glyph1197ote' contains a rendering artifact; it should read 'ScalingNote'.
  2. [Figure 5(c)] The y-axis label 'Crontrastive Entropy' is a typo for 'Contrastive Entropy.'
  3. [Section 3.2, paragraph after Eq. (4)] There is a duplicated 'and' in 'we denote the batch of all-gathered query representations as Q, and and the batch'.
  4. [Tables 3, 4, and 6] The paper states that results are averages of three tests but does not report standard deviations or confidence intervals. Adding mean +/- std for the key comparisons would make the reported statistical significance more informative.
  5. [Section 4.4.4 and Eq. (20)] The mixed scaling law in Eq. (20) has a form L(N,D) = (( (3.15e5/N)^0.12 / 770 + 1.16e2/D )^770 + 0.108, which appears to contain a transcription error in the nested fractions; the intended functional form should be written more clearly and verified against the fitted curve in Figure 5(c).
  6. [Appendix A, Table 7] The Stage II global batch size is listed as 40,000 with 100 epochs; it would help to clarify whether this is the number of query samples per step or the effective batch size after accumulation, since a batch of 40,000 queries is unusually large for a BERT-4L model.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the theoretical bound is algebraically derived from an external uniform-deviation bound, the scaling laws are explicit fits rather than disguised predictions, and no load-bearing self-citation chain is present.

full rationale

No circular step identified. The main theoretical claim (Proposition 3.2) is an algebraic consequence of the uniform-deviation bound in Theorem B.1 plus Lipschitz and Cauchy-Schwarz steps; the distillation term 2K/n * sum ||q_stu_i - q_i|| is derived as an upper bound on the teacher-student empirical-risk gap, and the QKD loss minimizes exactly that distance, so the proof does not assume its own conclusion as an input. The comparison with ScalingDoc does rely on the premise R(s_Q_R,D_R) <= R(s_Q_D,D_D), which the paper supports by citing [4,6,37] and its own Figure 1; that premise is an empirical assumption rather than an equation-level self-reference, so it is a correctness/rigor concern, not circularity. The scaling-law formulas (18)-(20) are explicit fits to validation contrastive entropy with reported R^2 values; they are extrapolated to motivate the 10^8-pair training scale, not used to 'predict' the same fitted points as if they were independent held-out results. No load-bearing self-citations appear: the references to NoteLLM works are related-work context only. The paper's central empirical comparison (Table 4) is an external benchmark against non-circular baselines, so the offline and online results stand independently of any self-referential justification.

Assumptions & free parameters 10 free parameters · 5 assumptions · 0 invented entities

The central method itself introduces no new physical entities. Its claims rest on fitted scaling constants, a set of hand-chosen hyperparameters, and several unproven modeling assumptions, most notably that larger models have lower empirical risk and that the point-wise loss preserves the contrastive learning analysis.

free parameters (10)
  • Scaling law constant C_model = 3.82e5
    Fitted in Eq. 18 to minimize validation contrastive entropy across Qwen2.5 model sizes; R2 = 0.9999.
  • Scaling law exponent alpha_model = 0.14
    Fitted exponent in Eq. 18.
  • Scaling law offset delta_model = 0.18
    Fitted offset in Eq. 18.
  • Scaling law constant C_data = 2.60e3
    Fitted in Eq. 19 on data-size sweep.
  • Scaling law offset delta_data = 1.4e-7
    Fitted in Eq. 19; near zero.
  • Mixed scaling law coefficients = 3.15e5, 0.12, 770, 1.16e2, 0.108
    Fitted in Eq. 20 across model and data sizes.
  • Hard negative margin = 0.2
    Chosen by hand; used in margin loss Eq. 9.
  • Hard negative loss weight = 0.5
    Chosen by hand; alpha in Eq. 12.
  • Distillation lambda = 1
    Chosen by hand; weight on cosine term in Eq. 15.
  • Softmax temperature tau = not reported
    Appears in contrastive losses Eqs. 5-6 but is not specified in implementation details.
assumptions (5)
  • domain assumption The large jointly trained teacher tower has lower empirical risk than a small-query tower on the same data.
    Used in Section 3.4 Analysis to conclude R(s_Q_R,D_R) <= R(s_Q_D,D_D); not proven in this paper, only supported by citation and Figure 1.
  • domain assumption The list-wise softmax contrastive loss can be replaced by a point-wise BCE loss for generalization analysis.
    Appendix B Eq. 22-23; this changes the training objective and is a simplification, not an equivalence.
  • standard math Cosine similarity can be treated as a scaled dot product in the proof.
    Appendix B; standard scaling argument, reasonable for analysis.
  • standard math The uniform deviation bound from USTAD (Theorem B.1) applies.
    Cited as [20]; used to derive Propositions 3.1 and 3.2.
  • domain assumption The scaling law functional form L(x) = (C/x)^alpha + delta describes retrieval loss.
    Taken from [6,10,16] and fit to data in Section 4.4; the functional form is assumed, not derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ScalingNote: Scaling up Retrievers with Large Language Models for Real-World Dense Retrieval." pith.science (2026). https://pith.science/paper/KZIJNJOI

@misc{pith2026241115766,
  author       = {Pith},
  title        = {Pith review of: ScalingNote: Scaling up Retrievers with Large Language Models for Real-World Dense Retrieval},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/KZIJNJOI}},
  note         = {Machine review of arXiv:2411.15766}
}
read the original abstract

Dense retrieval in most industries employs dual-tower architectures to retrieve query-relevant documents. Due to online deployment requirements, existing real-world dense retrieval systems mainly enhance performance by designing negative sampling strategies, overlooking the advantages of scaling up. Recently, Large Language Models (LLMs) have exhibited superior performance that can be leveraged for scaling up dense retrieval. However, scaling up retrieval models significantly increases online query latency. To address this challenge, we propose ScalingNote, a two-stage method to exploit the scaling potential of LLMs for retrieval while maintaining online query latency. The first stage is training dual towers, both initialized from the same LLM, to unlock the potential of LLMs for dense retrieval. Then, we distill only the query tower using mean squared error loss and cosine similarity to reduce online costs. Through theoretical analysis and comprehensive offline and online experiments, we show the effectiveness and efficiency of ScalingNote. Our two-stage scaling method outperforms end-to-end models and verifies the scaling law of dense retrieval with LLMs in industrial scenarios, enabling cost-effective scaling of dense retrieval systems. Our online method incorporating ScalingNote significantly enhances the relevance between retrieved documents and queries.

Figures

Figures reproduced from arXiv: 2411.15766 by the authors.

Figure 1
Figure 1. Performance and queries per second (QPS) compar [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The framework of ScalingNote. The first stage is fully scaling the dual-tower using scaled training data, which learns [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The online framework includes two stages: offline [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Key information about data collection includes: (a) two sources of query-document pairs: user click behaviors and [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: The scaling laws of the LLM-based dual-tower architecture for real-world dense retrieval on Xiaohongshu. The dots [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

62 extracted references · 39 canonical work pages

  1. [1]

    Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023. Task-aware Retrieval with Instructions. In Findings of ACL. 3650–3675

    work page 2023

  2. [2]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. arXiv preprint arXiv:2404.05961 (2024)

    arXiv 2024

  3. [3]

    Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. InNeurIPS. 1877–1901

    work page 2020

  4. [4]

    Daniel Campos, Alessandro Magnani, and ChengXiang Zhai. 2023. Quick dense retrievers consume kale: Post training kullback leibler alignment of embeddings for asymmetrical dual encoders. arXiv preprint arXiv:2304.01016 (2023)

    arXiv 2023

  5. [5]

    Fairstein Yaron Cohen, Nachshon and Guy Kushilevitz. 2024. Extremely efficient online query encoding for dense retrieval. In Findings of NAACL. 43–50

    work page 2024

  6. [6]

    Yan Fang, Jingtao Zhan, Qingyao Ai, Jiaxin Mao, Weihang Su, Jia Chen, and Yiqun Liu. 2024. Scaling laws for dense retrieval. In SIGIR. 1339–1349

    work page 2024

  7. [7]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997 (2023)

    arXiv 2023

  8. [8]

    Yunzhong He, Yuxin Tian, Mengjiao Wang, Feier Chen, Licheng Yu, Mao- long Tang, Congcong Chen, Ning Zhang, Bin Kuang, and Arul Prakash. 2023. Que2engage: Embedding-based retrieval for relevant and engaging products at facebook marketplace. In WWW. 386–390

    work page 2023

Show all 62 references
  1. [9]

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the Knowledge in a Neural Network. arXiv:1503.02531

    2015 arXiv

  2. [10]

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. 2022. Training compute-optimal large language models. In NeurIPS. 30016–30030

    2022

  3. [11]

    Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In SIGIR. 113–122

    2021

  4. [12]

    Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020. Embedding- based retrieval in facebook search. In KDD. 2553–2561

    2020

  5. [13]

    Gautier Izacard and Edouard Grave. 2021. Distilling Knowledge from Reader to Retriever for Question Answering. In ICLR

    2021

  6. [14]

    Herve Jegou, Matthijs Douze, and Cordelia Schmid. 2010. Product quantization for nearest neighbor search. TPAMI 33, 1 (2010), 117–128

    2010

  7. [15]

    Ting Jiang, Shaohan Huang, Zhongzhi Luan, Deqing Wang, and Fuzhen Zhuang

  8. [16]

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 (2020)

    2020 arXiv

  9. [17]

    Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen Tau Yih. 2020. Dense passage retrieval for open- domain question answering. In EMNLP. 6769–6781

    2020

  10. [18]

    Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, Vol. 1. 2

    2019

  11. [19]

    Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In SIGIR. 39–48

    2020

  12. [20]

    Seungyeon Kim, Ankit Singh Rawat, Manzil Zaheer, Wittawat Jitkrittum, Veer- anjaneyulu Sadhanala, Sadeep Jayasumana, Aditya Krishna Menon, Rob Fergus, and Sanjiv Kumar. [n. d.]. USTAD: Unified Single-model Training Achieving Diverse Scores for Information Retrieval. In ICML

  13. [21]

    Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, et al. 2022. Matryoshka representation learning. NeurIPS 35 (2022), 30233–30249

    2022

  14. [22]

    Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi, Bryan Catanzaro, and Wei Ping. 2024. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv preprint arXiv:2405.17428 (2024)

    2024 arXiv

  15. [23]

    Chaofan Li, Zheng Liu, Shitao Xiao, Yingxia Shao, and Defu Lian. 2024. Llama2Vec: Unsupervised Adaptation of Large Language Models for Dense Retrieval. In ACL. 3490–3500

    2024

  16. [24]

    Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. 2021. Embedding-based product retrieval in taobao search. In KDD. 3181–3189

    2021

  17. [25]

    Zihan Liao, Hang Yu, Jianguo Li, Jun Wang, and Wei Zhang. 2024. D2LLM: Decomposed and Distilled Large Language Models for Semantic Search. In ACL. 14798–14814

    2024

  18. [26]

    Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP- 2021). 163–173

    2021

  19. [27]

    Yiding Liu, Weixue Lu, Suqi Cheng, Daiting Shi, Shuaiqiang Wang, Zhicong Cheng, and Dawei Yin. 2021. Pre-trained language model for web-scale retrieval in baidu search. In KDD. 3365–3375

    2021

  20. [28]

    Yiqun Liu, Kaushik Rangadurai, Yunzhong He, Siddarth Malreddy, Xunlong Gui, Xiaoyi Liu, and Fedor Borisyuk. 2021. Que2search: fast and accurate query and document understanding for search at facebook. In KDD. 3376–3384

    2021

  21. [29]

    Wenhao Lu, Jian Jiao, and Ruofei Zhang. 2020. Twinbert: Distilling knowledge to twin-structured compressed bert models for large-scale retrieval. In CIKM. 2645–2652

    2020

  22. [30]

    Zepu Lu, Jin Chen, Defu Lian, Zaixi Zhang, Yong Ge, and Enhong Chen. 2024. Knowledge distillation for high dimensional search index. NeurIPS 36 (2024)

    2024

  23. [31]

    Yuanjie Lyu, Zhiyu Li, Simin Niu, Feiyu Xiong, Bo Tang, Wenjin Wang, Hao Wu, Huanyong Liu, Tong Xu, and Enhong Chen. 2024. Crud-rag: A comprehensive chinese benchmark for retrieval-augmented generation of large language models. arXiv preprint arXiv:2401.17043 (2024)

    2024 arXiv

  24. [32]

    Yuanjie Lyu, Zihan Niu, Zheyong Xie, Chao Zhang, Tong Xu, Yang Wang, and Enhong Chen. 2024. Retrieve-Plan-Generation: An Iterative Planning and An- swering Framework for Knowledge-Intensive LLM Generation. arXiv preprint arXiv:2406.14979 (2024)

    2024 arXiv

  25. [33]

    Guangyuan Ma, Yongliang Ma, Xing Wu, Zhenpeng Su, Ming Zhou, and Songlin Hu. 2024. Task-level Distributionally Robust Optimization for Large Language Model-based Dense Retrieval. arXiv preprint arXiv:2408.10613 (2024)

    2024 arXiv

  26. [34]

    Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine- tuning llama for multi-stage text retrieval. In SIGIR. 2421–2425

    2024

  27. [35]

    J MacQueen. 1967. Some methods for classification and analysis of multivariate observations. In Proceedings of 5-th Berkeley Symposium on Mathematical Statistics and Probability/University of California Press

    1967

  28. [36]

    Alessandro Magnani, Feng Liu, Suthee Chaidaroon, Sachin Yadav, Praveen Reddy Suram, Ajit Puthenputhussery, Sijie Chen, Min Xie, Anirudh Kashi, Tony Lee, et al. 2022. Semantic retrieval at walmart. In KDD. 3495–3503

    2022

  29. [37]

    Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, et al . 2022. Large Dual Encoders Are Generalizable Retrievers. In EMNLP. 9844–9855

    2022

  30. [38]

    Wenjun Peng, Guiyang Li, Yue Jiang, Zilong Wang, Dan Ou, Xiaoyi Zeng, Derong Xu, Tong Xu, and Enhong Chen. 2024. Large language model based long-tail query rewriting in taobao search. In WWW. 20–28

    2024

  31. [39]

    Jacob Pfau, William Merrill, and Samuel R Bowman. 2024. Let’s Think Dot by Dot: Hidden Computation in Transformer Language Models. arXiv preprint arXiv:2404.15758 (2024)

    2024 arXiv

  32. [40]

    Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxi- ang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In NAACL. 5835–5847

    2021

  33. [41]

    Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deep- speed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD. 3505–3506

    2020

  34. [42]

    2021.{Zero-offload}: Democratizing{billion-scale} model training

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhang, Dong Li, and Yuxiong He. 2021.{Zero-offload}: Democratizing{billion-scale} model training. In USENIX Annual Technical Con- ference. 551–564

    2021

  35. [43]

    Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A Joint Training Method for Dense Passage Retrieval and Passage Re-ranking. In EMNLP. 2825–2835

    2021

  36. [44]

    Yixuan Tang and Yi Yang. 2024. Pooling And Attention: What Are Effective Designs For LLm-Based Embedding Models? arXiv preprint arXiv:2409.02727 (2024)

    2024 arXiv

  37. [45]

    Qwen Team. 2024. Qwen2.5: A Party of Foundation Models. https://qwenlm. github.io/blog/qwen2.5/

    2024

  38. [46]

    Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2023. Improving text embeddings with large language models. arXiv preprint arXiv:2401.00368 (2023)

    2023 arXiv

  39. [47]

    Liang Wang, Nan Yang, and Furu Wei. [n. d.]. Query2doc: Query Expansion with Large Language Models. In EMNLP

  40. [48]

    Yuxuan Wang and Lyu Hong. 2023. Query Encoder Distillation via Embedding Alignment is a Strong Baseline Method to Boost Dense Retriever Online Efficiency. In Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP). 290–298

    2023

  41. [49]

    Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N Bennett, Junaid Ahmed, and Arnold Overwijk. [n. d.]. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In ICLR

  42. [50]

    Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, and Enhong Chen. 2023. Large language models for generative information extraction: A survey. arXiv preprint arXiv:2312.17617 (2023)

    2023 arXiv

  43. [51]

    Lanling Xu, Jianxun Lian, Wayne Xin Zhao, Ming Gong, Linjun Shou, Daxin Jiang, Xing Xie, and Ji-Rong Wen. 2022. Negative sampling for contrastive ScalingNote: Scaling up Retrievers with Large Language Models for Real-World Dense Retrieval Conference acronym ’XX, June 03–05, 20...

    2022 arXiv

  44. [52]

    Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Jiafeng Guo, Min Zhang, and Shaoping Ma

  45. [53]

    Bowen Zhang, Kehua Chang, and Chunping Li. 2024. Simple techniques for enhancing sentence embeddings in generative language models. In International Conference on Intelligent Computing . Springer, 52–64

    2024

  46. [54]

    Chao Zhang, Shiwei Wu, Haoxin Zhang, Tong Xu, Yan Gao, Yao Hu, and En- hong Chen. 2024. NoteLLM: A Retrievable Large Language Model for Note Recommendation. In WWW. 170–179

    2024

  47. [55]

    Chao Zhang, Haoxin Zhang, Shiwei Wu, Di Wu, Tong Xu, Yan Gao, Yao Hu, and Enhong Chen. 2024. NoteLLM-2: Multimodal Large Representation Models for Recommendation. arXiv preprint arXiv:2405.16789 (2024)

    2024 arXiv

  48. [56]

    Jianjin Zhang, Zheng Liu, Weihao Han, Shitao Xiao, Ruicheng Zheng, Yingxia Shao, Hao Sun, Hanqing Zhu, Premkumar Srinivasan, Weiwei Deng, et al. 2022. Uni-retriever: Towards learning the unified embedding based retriever in bing sponsored search. In KDD. 4493–4501

    2022

  49. [57]

    Wayne Xin Zhao, Jing Liu, Ruiyang Ren, and Ji-Rong Wen. 2024. Dense text retrieval based on pretrained language models: A survey. ACM TOIS 42, 4 (2024), 1–60

    2024

  50. [58]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  51. [59]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zhicheng Dou, and Ji-Rong Wen. 2023. Large language models for information retrieval: A survey.arXiv preprint arXiv:2308.07107 (2023). APPENDIX A TRAINING DETAILS Table 7: Training hyp...

    2023

  52. [62]

    Peony" despite the

    Let ℓ be the loss function, which is𝐿ℓ -Lipschitz in its non-target variable. Assume that the embedding functions in Q andD output embeddings with 𝐿2 norms at most 𝐾. Define the uniform deviation as follows: E𝑛(Q,D) = sup 𝑄∈Q,𝐷∈D 1 𝑛 ∑︁ 𝑖∈[𝑛] ℓ(𝑠𝑄,𝐷 𝑞𝑖,𝑑𝑖 ,𝑦𝑖)− E𝑞,𝑑ℓ(𝑠𝑄,𝐷 𝑞,𝑑 ...

    2018

  53. [2021]

    In SIGIR

    Optimizing dense retrieval model training with hard negatives. In SIGIR. 1503–1512

  54. [2023]

    arXiv preprint arXiv:2307.16645 (2023)

    Scaling sentence embeddings with large language models. arXiv preprint arXiv:2307.16645 (2023)

    2023 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.