Pith. sign in

REVIEW 4 major objections 5 minor 23 references

NEAR$^2$: A Nested Embedding Approach to Efficient Product Retrieval and Ranking

T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Pith's one-line read A 64-dimension embedding beats the full 768-dimension model in product search.

desk verdict A claimed 12x embedding-size win is undermined by an inverted MNRL loss, a non-monotonic memory table, and a missing control for extra fine-tuning epochs. read the letter →

arxiv 2506.19743 v1 pith:VBXBG2E4 submitted 2025-06-24 cs.IR cs.CL

classification cs.IRcs.CL
keywords productretrievalrankingnestedembeddingsMatryoshkarepresentationlearningembeddingefficiencymultiplenegativelossonlinecontrastivee-commercesearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that a nested training objective lets e-commerce retrieval models answer queries with 64-dimensional embeddings instead of 768 while matching or beating the full-size model on accuracy. The authors continue fine-tuning encoder-based Transformers on ranked product-title data, optimizing multiple prefix lengths of the same embedding at once with ranking losses. On four test sets spanning short, implicit, common-string, and alphanumeric queries, the 64-dimension version of models such as eBERT and eBERT-siam outperforms the same models at full dimension. If true, search systems could serve large product catalogs with roughly twelve times less embedding storage and over a hundred times less memory, at no added inference cost. The paper also reports that the effect holds across multiple negative ranking loss and online contrastive loss.

What carries the argument

The load-bearing mechanism is the nested embedding objective: a weighted sum of task losses applied to the first $m$ dimensions of the embedding for $m \in \{64,128,256,512,768\}$, with the multiple negative ranking loss or online contrastive loss as the task loss. During training, each prefix head is scored by its own ranking loss, so the model learns to make every prefix itself a usable retrieval embedding. At inference the system simply truncates the embedding to 64 dimensions and computes cosine similarity, which is what delivers the claimed 12× size reduction and 100× memory reduction. The ablations show the choice of loss ordering matters: applying the nested objective on top of a model already fine-tuned on MNRL or OCL improves results, while applying an MRL loss first and then MNRL+OCL slightly degrades them.

What would settle it

Train a control model with identical hyperparameters, data, and two-epoch schedule, using the same loss but no nested heads, and compare its full-dimension metrics with the NEAR2 model at 64 dimensions. If the control's full-dimension metrics match or exceed the nested 64-dimension result, the nested objective is not the source of the improvement. A second check is to evaluate nested sizes the model never saw in training, such as 32, to test whether the coarse-to-fine property generalises beyond the trained prefix sizes.

Watch

Extended reading notes

Core claim

NEAR2 applies a Matryoshka-style nested representation objective to product retrieval and ranking: while fine-tuning a 768-dimension encoder, it jointly optimises the 64-, 128-, 256-, 512-, and 768-dimension prefixes of the same embedding as separate output heads, summing their ranking losses. After training, only the small prefix needs to be stored and compared at inference. On the CQ, CQ-balanced, CQ-common-str, and CQ-alphanum test sets, the 64-dimension embeddings improve precision, recall, NDCG, and MRR over the same model fine-tuned without the nested objective at full dimension; for example, eBERT-siam's NDCG@5 rises by about 11% on the CQ set with the small vectors. The authors interpret this as evidence that nested supervision packs the information needed for retrieval into a compact prefix, making similarity scores more discriminative than those of the baseline models.

Load-bearing premise

The reported gains are attributed to the nested objective, but no control model was fine-tuned for the same extra epochs on the same data without the nested loss, so part or all of the improvement could come from continued fine-tuning alone.

Editorial extensions

If this is right

  • Retrieval latency and storage scale down: a 64-dimension embedding needs 12× less space than 768, and the paper reports over 100× smaller memory usage at inference, so larger product catalogs can be served with the same infrastructure.
  • The gain is not tied to one loss function: NEAR2 improves models fine-tuned with multiple negative ranking loss and with online contrastive loss, and works on BERT, eBERT, and eBERT-siam backbones.
  • The method handles hard query types: accuracy improves on test sets built for short, implicit, common-string, and alphanumeric queries, where exact-match and semantic methods typically struggle.
  • Deployment can be adaptive: the same trained model can be truncated to different embedding sizes for different latency budgets without retraining.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because no equal-duration control without the nested objective was run, the cleanest attribution of the gains to nesting itself is not yet established; a matched fine-tuning control could change the size of the reported effect.
  • The qualitative score-distribution shift—nested models spread similarity scores out while baselines cluster near one value—suggests the objective acts partly as a calibration or sharpening mechanism, which could explain gains on hard queries beyond mere compression.
  • A testable extension is to evaluate prefix lengths outside the trained set, such as 32, to check whether the coarse-to-fine property generalises beyond the sizes seen in training.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes NEAR^2, a nested embedding approach based on Matryoshka representation learning (MRL) combined with a multiple negative ranking loss (MNRL) for e-commerce product retrieval and ranking. The method is applied to several encoder-based Transformers (BERT, eBERT, eBERT-siam, and their UCO variants) by continuing training on eBay's internal graded relevance data for two epochs. The paper claims up to 12x smaller embedding dimensions (768→64) with improved retrieval/ranking metrics on four test sets, as well as more than 100x smaller memory usage at inference, while introducing no extra training cost. The evaluation reports percentage improvements over baselines from Saadany et al. [3] and includes an ablation study over loss combinations.

Significance. If the central claims held, this work would offer a practical way to reduce embedding storage and inference cost in large-scale e-commerce search without sacrificing accuracy, which is directly relevant to a serious IR venue. The combination of MRL with a ranking loss for product retrieval is a sensible idea and the experimental scope covers several models and challenging query types. However, the paper currently suffers from fundamental issues in the description of the loss function, a missing matched-duration control experiment that leaves the performance gains un-attributed, and a clearly erroneous memory-usage table. These issues are load-bearing for the main conclusions.

major comments (4)
  1. [§3.1, Eq. (2)] The MNRL loss as written is inverted. For cosine similarity f, the standard ranking loss should penalize cases where f(q,p) is not greater than f(q,n)+margin, i.e., max(0, f(q,n) − f(q,p) + margin). Equation (2) instead uses max(0, f(q,p) − f(q,n) + margin), which penalizes exactly the desired ordering. The accompanying text makes the same error: it states the goal is to 'minimize the similarity between (q,p_i)'. If the implementation actually used the standard MNRL, the paper's description must be corrected to make the method reproducible; if Eq. (2) was actually used, the reported improvements are inexplicable. This is a central, correctness-relevant defect in the methodology section.
  2. [§4.2, Tables 2, 4, and 5] The paper's central attribution is untested because there is no matched-duration control. NEAR2 models are produced by continuing training for 2 epochs on the IGR data with the nested objective, but the baseline numbers in Tables 2 and 4 are taken from Saadany et al. [3] for models that did not receive these extra epochs. Any additional in-domain fine-tuning on graded-relevance data could improve retrieval and ranking metrics, independent of the nested structure. The ablation in Table 5 makes this confound visible: adding NEAR2 after OCL-only fine-tuning gives +32.09% NDCG@5, while after MNRL+OCL it is only +3.34%. A control model trained for the same number of epochs on the same data with the MRL term disabled (e.g., standard MNRL only) is necessary to isolate the effect of the nesting loss. Without it, the headline claim of improved accuracy at smaller dimensions is not established.
  3. [Table 3] The memory-usage figures are non-monotonic in embedding size: 768D→398.03 MB, 512D→2.77 MB, 256D→4.09 MB, 128D→0.55 MB, 64D→1.56 MB. This is impossible for any storage that scales with embedding dimensionality, and it directly undermines the abstract's claim of '100× smaller memory usage'. The values appear to suffer from a units or arithmetic error; the table must be corrected or the memory-related claim removed.
  4. [Abstract and §4.2] The abstract and conclusion claim that NEAR2 introduces 'no extra cost in training', but §4.2 explicitly states that the authors continued training the PTLMs and UCO models for 2 epochs on the IGR data. If the baselines from [3] did not receive these epochs, then NEAR2 incurs additional training cost; if the baselines also received the extra epochs, that must be described. As written, the claim is contradicted by the experimental setup and must be clarified or corrected.
minor comments (5)
  1. [§5, footnote and §3.2] Footnote 2 states that BERT and eBERT 'were not fine-tuned on e-commerce data', but §3.2 says eBERT was 'pre-trained on a corpus of approximately three billion product titles'. Please clarify the intended distinction between fine-tuning and pre-training to avoid apparent contradiction.
  2. [§3.1, Eq. (1)] The importance weights c_m in the MRL loss are introduced but never specified (no values or tuning procedure are reported). State how these weights were set.
  3. [Throughout] The name 'NEAR2' is rendered inconsistently as 'NEAR2', 'NEAR 2', and 'NEAR^2'; use a single notation consistently.
  4. [Tables 2 and A.1] The deltas are reported relative to baselines from a prior paper; it would be useful to also provide the absolute baseline metric values in an appendix to make the comparisons interpretable.
  5. [§4.2] The sentence 'excluding pairs with a score of 3' relative to a 1–5 relevance scale is ambiguous: specify whether score 3 ('Good') is treated as neutral/removed and why.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: NEAR2 is an empirical application of the external Matryoshka objective; the missing training-time control is a validity concern, not a circular derivation.

full rationale

The paper's derivation chain is empirical rather than deductive: NEAR2 applies the MRL loss (Eq. 1) from Kusupati et al. [13] with MNRL (Eq. 2) as the task loss, continues training on eBay IGR data, and evaluates retrieval/ranking metrics on the four CQ test sets from Saadany et al. [3]. The headline efficiency claim (12x smaller embeddings, 100x smaller memory) is an observed outcome of evaluating at dimension 64, not a quantity that was fitted to the test data and then reported as a prediction. The comparisons in Tables 2, 4, A.1, and A.2 are deltas against the prior results in Saadany et al. [3]; those baselines are external empirical numbers, not defined in terms of NEAR2's output. Self-citations to [3] and [16] supply the UCO models and test sets, but the central NEAR2 result is not derived from those citations; it is a new fine-tuning procedure followed by evaluation. There is no self-definitional step, no fitted input renamed as a prediction, no imported uniqueness theorem, and no ansatz smuggled through a self-citation. The absence of a matched-duration control without the MRL loss is a genuine threat to attribution of the observed gains, because the extra two epochs of continued fine-tuning described in Section 4.2 could explain part of the improvement; however, that is a methodological confound, not a circularity of the kind where a claimed prediction is equivalent to its inputs by construction. Similarly, the apparent sign issue in Eq. 2 and the non-monotonic memory figures in Table 3 are correctness/reproducibility concerns, not circular reasoning. Therefore the paper's core claim has independent empirical content and the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim relies on MRL as a prior technique, on proprietary eBay data and models, and on several hand-set hyperparameters. No new entities are introduced.

free parameters (4)
  • MRL importance weights c_m
    In Eq. 1, each nested dimension size m has a weight c_m; the paper does not state these values or whether they were tuned, which affects the loss balance.
  • MNRL margin = 0.75
    Set by hand for all experiments; the ranking loss depends on this margin.
  • Nested dimension set M = {64, 128, 256, 512, 768}
    The set of embedding sizes is chosen by hand; the 12x efficiency claim relies on the smallest size (64) versus the full size (768).
  • Continued training epochs = 2
    The number of epochs of additional fine-tuning on IGR data is a manual choice that affects all reported gains.
assumptions (3)
  • domain assumption Matryoshka Representation Learning produces nested embeddings whose prefixes match the accuracy of independently trained embeddings.
    The paper inherits this claim from Kusupati et al. [13] and does not re-derive or verify it for the product retrieval task.
  • domain assumption The IGR graded relevance annotations provide a reliable training signal for query-title relevance.
    All fine-tuning uses these internal eBay annotations, assuming they reflect true user relevance.
  • domain assumption The CQ test sets are representative of e-commerce retrieval challenges such as short, implicit, and alphanumeric queries.
    The test sets are defined in the authors' prior work [3], and the paper assumes they are valid benchmarks.

how reviews work

0 comments
Cite this review

Pith. "Pith review of NEAR$^2$: A Nested Embedding Approach to Efficient Product Retrieval and Ranking." pith.science (2026). https://pith.science/paper/VBXBG2E4

@misc{pith2026250619743,
  author       = {Pith},
  title        = {Pith review of: NEAR$^2$: A Nested Embedding Approach to Efficient Product Retrieval and Ranking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VBXBG2E4}},
  note         = {Machine review of arXiv:2506.19743}
}
abstract

E-commerce information retrieval (IR) systems struggle to simultaneously achieve high accuracy in interpreting complex user queries and maintain efficient processing of vast product catalogs. The dual challenge lies in precisely matching user intent with relevant products while managing the computational demands of real-time search across massive inventories. In this paper, we propose a Nested Embedding Approach to product Retrieval and Ranking, called NEAR$^2$, which can achieve up to $12$ times efficiency in embedding size at inference time while introducing no extra cost in training and improving performance in accuracy for various encoder-based Transformer models. We validate our approach using different loss functions for the retrieval and ranking task, including multiple negative ranking loss and online contrastive loss, on four different test sets with various IR challenges such as short and implicit queries. Our approach achieves an improved performance over a smaller embedding dimension, compared to any existing models.

Figures

Figures reproduced from arXiv: 2506.19743 by the authors.

Figure 1
Figure 1. NEAR2 combines UCO optimisation [3] with MRL [13] to learn multiple nested embedding representations of different sizes (𝑧 ∈ R 𝑑 as the full embedding representation) with multi-task learning, which are adaptive to different downstream tasks such as retrieval or ranking during inference. The initial 𝑚−dimensions of the Matryoshka representation, where 𝑚 ∈ 𝑀, the set of nested repre￾sentation sizes, form a compact an… view at source ↗
Figure 2
Figure 2. Examples of query-title pairs from the CQ-common-str test set. The search queries can be very short and ambiguous, but the retrieved products can be very different as shown in (a), or their titles can be quite close in semantic relation as shown in (b) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 8 canonical work pages

  1. [3]

    Saadany, S

    H. Saadany, S. Bhosale, S. Agrawal, D. Kanojia, C. Orasan, Z. Wu, Centrality-aware product retrieval and ranking, in: F. Dernoncourt, D. Preo¸ tiuc-Pietro, A. Shimorina (Eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, Association for Computational Linguistics, Miami, Florida, US, 2024, pp. 215...

  2. [1]

    S. Li, F. Lv, T. Jin, G. Lin, K. Yang, X. Zeng, X.-M. Wu, Q. Ma, Embedding-based product retrieval in taobao search, in: Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, KDD ’21, Association for Computing Machinery, New York, NY , USA, 2021, p. 3181–3189. URL: https://doi.org/10.1145/3447548.3467101. doi:10.1145/3447548.3467101

  3. [2]

    Keyvan, J

    K. Keyvan, J. X. Huang, How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges, ACM Comput. Surv. 55 (2022). URL: https: //doi.org/10.1145/3534965. doi:10.1145/3534965

  4. [4]

    D. N. Mhawi, H. W. Oleiwi, N. H. Saeed, H. L. Al-Taie, An efficient information retrieval system using evolutionary algorithms, Network 2 (2022) 583–605. URL: https://www.mdpi.com/ 2673-8732/2/4/34. doi:10.3390/network2040034

  5. [5]

    K. A. Hambarde, H. Proença, Information retrieval: Recent advances and beyond, IEEE Access 11 (2023) 76581–76604. doi:10.1109/ACCESS.2023.3295776

  6. [6]

    Y . Zhu, H. Yuan, S. Wang, J. Liu, W. Liu, C. Deng, H. Chen, Z. Liu, Z. Dou, J.-R. Wen, Large language models for information retrieval: A survey, arXiv preprint (2023).arXiv:2308.07107

  7. [7]

    Vaswani, N

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, I. Polosukhin, Attention is all you need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, Curran Associates Inc., Red Hook, NY , USA, 2017, p. 6000–6010

  8. [8]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transform- ers for language understanding, in: J. Burstein, C. Doran, T. Solorio (Eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, V olume 1 (Long and Short Papers), ...

Show all 23 references
  1. [9]

    J. Guo, Y . Fan, Q. Ai, W. B. Croft, A deep relevance matching model for ad-hoc retrieval, in: Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, CIKM ’16, Association for Computing Machinery, New York, NY , USA, 2016, p. 55–64. UR...

  2. [10]

    Mitra, F

    B. Mitra, F. Diaz, N. Craswell, Learning to match using local and distributed representations of text for web search, in: Proceedings of the 26th International Conference on World Wide Web, WWW ’17, International World Wide Web Conferences Steering Committee, Republic and Cant...

  3. [11]

    S. Kim, A. S. Rawat, M. Zaheer, S. Jayasumana, V . Sadhanala, W. Jitkrittum, A. K. Menon, R. Fergus, S. Kumar, Embeddistill: A geometric knowledge distillation for information retrieval,

  4. [12]

    H. Wan, S. S. Patel, J. W. Murdock, S. Potdar, S. Joshi, Fast and light-weight answer text retrieval in dialogue systems, in: A. Loukina, R. Gangadharaiah, B. Min (Eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Ling...

  5. [13]

    Kusupati, G

    A. Kusupati, G. Bhatt, A. Rege, M. Wallingford, A. Sinha, V . Ramanujan, W. Howard-Snyder, K. Chen, S. Kakade, P. Jain, et al., Matryoshka representation learning, in: Advances in Neural Information Processing Systems, 2022

  6. [14]

    X. Li, Z. Li, J. Li, H. Xie, Q. Li, ESE: Espresso sentence embeddings, arXiv preprint (2024). arXiv:2402.14776

  7. [15]

    Henderson, R

    M. Henderson, R. Al-Rfou, B. Strope, Y .-H. Sung, L. Lukács, R. Guo, S. Kumar, B. Miklos, R. Kurzweil, Efficient natural language response suggestion for smart reply, arXiv preprint arXiv:1705.00652 (2017)

  8. [16]

    Saadany, S

    H. Saadany, S. Bhosale, S. Agrawal, Z. Wu, C. Or˘asan, D. Kanojia, Product retrieval and ranking for alphanumeric queries, in: Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM ’24, Association for Computing Machinery, New York,...

  9. [17]

    Carlsson, A

    F. Carlsson, A. C. Gyllensten, E. Gogoulou, E. Y . Hellqvist, M. Sahlgren, Semantic re-tuning with contrastive tension, in: International Conference on Learning Representations, 2021. URL: https://openreview.net/forum?id=Ov_sMNau-PF

  10. [18]

    Jiang, Y

    Y . Jiang, Y . Shang, R. Li, W.-Y . Yang, G. Tang, C. Ma, Y . Xiao, E. Zhao, A unified neural network approach to e-commerce relevance learning, in: Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data, DLP-KDD ’19, Associati...

  11. [19]

    D. Kang, W. Jang, Y . Park, Evaluation of e-commerce websites using fuzzy hierarchical topsis based on e-s-qual, Applied Soft Computing 42 (2016) 53–65. URL: https://www.sciencedirect.com/ science/article/pii/S1568494616300047. doi:https://doi.org/10.1016/j.asoc.2016.01.017

  12. [20]

    Loshchilov, F

    I. Loshchilov, F. Hutter, Decoupled weight decay regularization, in: International Conference on Learning Representations, 2019. URL: https://openreview.net/forum?id=Bkg6RiCqY7

  13. [21]

    Järvelin, J

    K. Järvelin, J. Kekäläinen, Cumulated gain-based evaluation of ir techniques, ACM Trans. Inf. Syst. 20 (2002) 422–446. URL: https://doi.org/10.1145/582415.582418. doi:10.1145/582415.582418

  14. [22]

    plants” with eBERT. Based on the gold label, the expected product title should include “potted plants

    C. Lee, R. Roy, M. Xu, J. Raiman, M. Shoeybi, B. Catanzaro, W. Ping, Nv-embed: Improved techniques for training llms as generalist embedding models, 2024. URL: https://arxiv.org/abs/2405. 17428.arXiv:2405.17428. A. Additional Figures and Tables Model Precision@𝑘Recall@𝑘NDCG@𝑘M...

  15. [2023]

    URL: https://openreview.net/forum?id=BT03V9Re9a

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.