Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Improving Ad matching via Cluster-Adaptive Keyword Expansion and Relevance tuning

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read eBay shows document-side semantic keyword expansion can widen ad reach without wrecking relevance.

desk verdict Useful industrial pattern, but the A/B table contradicts the abstract's 'improves both relevance and CTR'—treat the headline result as unsupported. read the letter →

arxiv 2505.18897 v1 pith:CCMSJFOA submitted 2025-05-24 cs.IR cs.AI

Dipanwita Saha , Anis Zaman , Hua Zou , Ning Chen , Xinxin Shu , Nadia Vase , Abraham Bagherjeiran This is my paper · ORCID
classification cs.IRcs.AI
keywords semantickeywordexpansiondenseembeddingscluster-adaptivethresholdingadrelevancegradientboosteddecisiontreesbroadmatche-commercesearchnearestneighbor
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a production ad-matching system at eBay that expands seller-supplied keywords on the document side, so a query like "Apple phone cover" can match an ad keyword "iPhone case" without rewriting the query. The authors claim that a pre-trained siamese embedding model, nearest-neighbor search, and cluster-specific similarity thresholds can add semantically related keyword variants while keeping precision, and that a lightweight stacked tree model fixes the relevance score of the newly matched items. They run a two-week A/B test showing progressive gains in impressions, bought items per click, and ad revenue as each component is added, with slot-normalized CTR staying neutral. If this holds, token-based ad retrieval can be upgraded to semantic matching without replacing the existing infrastructure.

What carries the argument

The pipeline rests on three objects. (1) A pre-trained siamese embedding model (micro-BERT encoder fine-tuned on eBay titles, trained with contrastive learning) that maps ad keywords to dense vectors; it supplies the semantic space in which cosine distance approximates intent similarity. (2) A k-means clustering of that space into 1,000 clusters per market, where each cluster's threshold is the $p$-th quantile of intra-cluster centroid distances; a single global quantile $p = 99.9999$ governs the precision-recall tradeoff, and the threshold adapts to local semantic density. (3) A stacked relevance model $f_{adj}(x) = f_{GBDT}(x) + \sum_{i=1}^{t_r} g_i(x)$ with at most two shallow trees ($t_r < 2$, depth $\le 5$) trained on human-judged relevance scores for expansion inventory, which corrects the baseline GBDT's bias on expanded matches. The FAISS flat index performs the nearest-neighbor search, and gender/numeric post-processing filters remove attribute-inconsistent expansions. Together these pieces let the system run daily refreshes and produce final query-item matches through a market-specific relevance threshold.

What would settle it

On a held-out set of human-judged query-item pairs drawn from keyword-expansion inventory, compute the base GBDT's RMSE and the stacked model's RMSE; if the stacked model does not achieve a lower RMSE (or at least the >4% reduction on Bad/Fair items claimed), the central relevance claim is unsupported. A second check: compare a global similarity threshold against the cluster-adaptive thresholds on the same TPR/impression-lift curve; if the global threshold matches or beats the cluster-adaptive one in offline TPR at matched impression lift, the adaptive-threshold contribution collapses.

Watch

Extended reading notes

Core claim

The central claim is that broad-match ad retrieval can be extended on the document side: instead of expanding the user query, the system expands the advertiser's keyword into semantically related variants using dense embeddings and nearest-neighbor search, then uses two guards to keep quality. First, a k-means partition of the keyword embedding space sets a per-cluster similarity threshold at a global quantile of intra-cluster distances, so dense, specific clusters get stricter cutoffs and broad, ambiguous clusters get looser ones. Second, because an expanded keyword may retrieve items only partially aligned with the original intent, the production GBDT relevance model is augmented by up to two shallow trees trained on human-judged labels for expansion inventory, and a market-specific threshold is applied to final matches. The A/B results show that embedding-only expansion raises impressions but hurts CTR, cluster thresholding recovers most of the CTR loss, and relevance model refinement brings CTR nearly back to baseline while increasing bought items per click and ad revenue.

Load-bearing premise

The whole relevance-tuning stage works only if the baseline GBDT's residual errors on expanded-match items are structured enough for one or two shallow trees to correct; if those residuals are mostly noise, the stacked trees add no signal and the system's final quality guard collapses.

Editorial extensions

If this is right

  • Token-based ad retrieval systems can gain semantic coverage by expanding advertiser keywords offline, leaving query processing and existing retrieval infrastructure unchanged.
  • Cluster-adaptive thresholds let one global hyperparameter (the quantile $p$) control precision per region of embedding space, so broad terms get recall and specific terms get precision.
  • Incremental stacking of one or two shallow trees on human labels lets the relevance model adapt to new inventory without a full retrain, reducing risk of overfitting when new data is limited.
  • Two-week A/B evidence indicates that precision tuning and relevance filtering are the components that turn raw impression gains into revenue and bought-items gains.
  • The containment matching rule (expanded keyword must be a token subset of the query) keeps expansion interpretable and limits brand conquesting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The residual-correction design implies the base GBDT must already be strong; the method is a bias-correction patch, not a replacement, so its benefit is bounded by how much of the expansion error is systematic rather than random.
  • The same cluster-adaptive thresholding recipe could transfer to other marketplaces or to verticals like jobs or real estate, provided per-market clusters are retrained as inventory shifts.
  • Because thresholds are derived from a static embedding space, drift in buyer vocabulary or catalog mix would require periodic re-clustering and threshold recomputation; the paper flags this as future work.
  • A natural extension is to let the quantile $p$ or the thresholds react to query-side click feedback, which would make the expansion adapt to user behavior rather than only to human-labeled relevance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a document-side semantic keyword expansion system for sponsored search at eBay. It uses a pre-trained siamese embedding model to find semantically related ad keywords, k-means clustering with per-cluster similarity quantile thresholds to control expansion precision, and an incrementally trained shallow decision-tree ensemble added to a baseline GBDT relevance model. The authors report offline TPR results for threshold selection, offline RMSE improvements for the stacked relevance model, and a two-week A/B test comparing three variants (embeddings only, plus cluster thresholds, plus relevance model) against a token-based expansion baseline. The abstract claims the system 'improves both relevance and click-through rate (CTR)' and the conclusion repeats that the framework enhances recall and precision.

Significance. If the central claims were supported, the paper would offer a practical way to extend token-based ad retrieval with semantic expansions while controlling precision via density-aware thresholds and a lightweight relevance adjustment, a combination that could transfer to other e-commerce marketplaces. The paper is also candid about its assumptions, particularly in Section 3.3, and it reports a real A/B experiment rather than only offline metrics. However, the paper's own online results contradict the headline claim: Table 3 shows negative CTR, negative bought-items-per-click, and negative average relevance for all three variants relative to the token-based baseline. The reported positive impression lift is credible, but the claimed relevance and engagement improvements are not established by the submitted evidence.

major comments (4)
  1. [Abstract and Section 4.2, Table 3] The abstract states that the system 'improves both relevance and click-through rate (CTR)', but Table 3 reports the final variant (Emb+Cluster threshold+Relevance model) with CTR lift -0.62%, BI/Click lift -1.96%, Slot Neutral CTR lift -0.13%, and Avg Relevance lift -0.18% relative to the token-based baseline. Section 4.2 claims that 'subsequent refinements significantly recovered CTR and yielded net gains in BI/click and revenue,' yet the table shows BI/Click still negative and contains no revenue metric at all. This is an internal inconsistency in the manuscript's central evidentiary claim, not merely a disagreement with an external benchmark.
  2. [Section 4.1.1 and Table 3] The offline RMSE improvements for the stacked model (greater than 4% for Fair/Bad items and greater than 1% for Excellent items) are reported without sample sizes, standard errors, or confidence intervals, and Table 2 shows only five selected examples. More importantly, no calibration, ablation, or analysis connects these held-out RMSE improvements to the online metrics in Table 3; the final variant's average relevance online is negative, so the offline gains do not rescue the online relevance claim.
  3. [Section 4.1, Figure 2a] The production quantile p=99.9999 is selected on the basis of True Positive Rate computed with ChatGPT-3.5-assisted labeling of expansions, and the same TPR metric is then used to justify the post-processing filters and the cluster-thresholding quality. No validation of the ChatGPT-3.5 labels against human relevance judgments is reported, and no inter-annotator agreement is given. This makes the threshold-selection evidence circular with respect to the paper's own definition of expansion quality, and the validity of the entire precision-control mechanism depends on that one proxy metric.
  4. [Section 3.3 and Eq. (2)] The paper itself identifies the load-bearing risk: the stacked trees are beneficial only if residual errors of the baseline GBDT are structured enough for one or two shallow trees to correct. The online results in Table 3, where the relevance model refinement improves CTR from -1.26% to -0.62% but leaves Avg Relevance at -0.18% and BI/Click at -1.96%, suggest that the residual-structure assumption did not hold strongly in production. This should be treated as a central limitation, and the conclusion should not claim relevance improvement until this is reconciled.
minor comments (5)
  1. [Section 3.2 and Algorithm 1] There is an inconsistency in the similarity threshold direction: Section 3.2 says variants v are kept when sim(k,v) ≤ τ_m, while Algorithm 1 line 8 keeps neighbors when sim(e_k,e_n) ≥ τ_m. The text also calls τ_m a distance threshold but defines it via a quantile of distances to the cluster centroid, while Figure 2b labels it a quantile threshold; please clarify whether τ_m is a cosine distance or cosine similarity cutoff.
  2. [Section 3.3, Eq. (2)] The notation 't_r =< 2' is ambiguous; I assume it means t_r ≤ 2. Also, 'refereed' should be 'referred'. In addition, calling this a 'stacked model' is unconventional, since the additional trees are added directly to the baseline prediction rather than meta-learned on baseline predictions.
  3. [Section 3.3, paragraph 2] The base relevance model is described both as a 'Gaussian regression' and as 'Gradient Boosted Decision Tree (GBDT)' trained with more than 100 trees; please clarify whether the base model is a GBDT with a Gaussian regression loss or a different model, since this affects the interpretation of the additive-tree adjustment.
  4. [Section 4.1, first paragraph] The description of k-fold resampling for clustering stability is incomplete: no definition of assignment consistency, no number of folds, and no quantitative stability result is given. The elbow-method choice of M=1000 is also reported without showing the WCSS curve or sensitivity.
  5. [Table 2] The columns 'Old Rel.', 'Adj. Rel', and 'True Rel' mix different scales: the first two are model regression scores on a 0–5 scale, while 'True Rel' is a categorical label (Bad, Fair, Good). Please use a consistent display, such as showing human label values with model scores, and state the number of evaluation examples used.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the pipeline is component-based and the final A/B test is an external benchmark.

full rationale

The paper's derivation chain is not circular. The embedding model is taken from external prior work (Mandal et al. [4]) with no author overlap and no dependence on the present results. Cluster thresholds are defined as quantiles of intra-cluster distance distributions, and the global quantile p is selected offline using TPR/impression-lift experiments; TPR is then reported as an evaluation outcome, not as a prediction forced by the fitted parameter. The stacked relevance model is trained on human-labeled examples for the expanded inventory and evaluated on a held-out set from the same distribution, which is standard supervised model adjustment rather than a circular validation. The final A/B test compares against a token-based baseline and is an independent, externally grounded benchmark; it is not constructed from the paper's own fitted values. The abstract's claim that the system improves both relevance and CTR is not supported by the negative numbers in Table 3, but this is an internal empirical inconsistency or correctness issue, not a circularity. No load-bearing self-citation, uniqueness import, or definitional equivalence appears in the paper.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The central claim inherits several unproven premises: transferability of the pre-trained query model to ad keywords, semantic validity of k-means cluster regions, structured residuals for the stacked relevance model, and reliability of a ChatGPT-assisted TPR metric. Four hyperparameters are fitted or tuned. No invented entities, forces, or quantities are introduced.

free parameters (4)
  • Global quantile p for cluster thresholds = 99.9999%
    Chosen as production baseline by evaluating TPR and impression lift across quantiles in Section 4.1. Equation (1) defines tau_m = Quantile(D_m, p), so p directly sets all cluster similarity cutoffs.
  • Number of k-means clusters M = 1000
    Selected via the elbow method on average within-cluster sum of squares in Section 4.1. This determines the semantic granularity on which adaptive thresholds are computed.
  • Market-specific relevance threshold t_rel = Not disclosed
    Tuned per market via offline relevance filter simulations on a holdout set with human judgments, described in Section 3.3.
  • Number of additional stacked trees t_r = <= 2
    Set to a small number with depth <= 5 to limit overfitting in Section 3.3, Equation (2).
assumptions (6)
  • domain assumption The pre-trained two-tower siamese model of Mandal et al. [4], trained on eBay query-title pairs, yields dense embeddings in which cosine distance reflects semantic relevance for ad keyword fragments.
    Section 3.1.1 reuses the model without fine-tuning on ad keywords, assuming partial query semantics transfer to keyword subsets.
  • domain assumption k-means with M=1000 clusters produces semantically coherent regions whose intra-cluster distance distributions are a valid basis for adaptive similarity cutoffs.
    Section 3.2 and Section 4.1 assert cluster homogeneity through elbow and k-fold consistency, but no cluster-quality metrics such as silhouette scores are reported.
  • domain assumption Residual errors of the baseline GBDT relevance model are structured enough for t_r <= 2 shallow trees to improve relevance predictions.
    Stated in Section 3.3: 'it assumes residual errors are structured enough for meaningful adjustments.' If residuals are noise, the stacked model adds no signal.
  • domain assumption Human judgments on the five-point relevance scale are a reliable, consistent ground truth for training and evaluating the relevance model.
    Human labels are the training target for the base and stacked models and define the TPR and offline RMSE metrics in Sections 3.3 and 4.1.1.
  • ad hoc to paper The TPR metric computed with ChatGPT-3.5-assisted labeling is a valid measure of semantic expansion quality.
    Section 4.1 introduces assistant-based TPR without prompt details, sample sizes, or human agreement statistics; this metric is used to select p and to justify the thresholding approach.
  • standard math FAISS flat index gives exact nearest neighbors, so batch search does not bias the neighbor set.
    Section 3.1.1 uses a FAISS flat index for accuracy; exhaustive search is a standard exact method, so this assumption is routine.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Ad matching via Cluster-Adaptive Keyword Expansion and Relevance tuning." pith.science (2026). https://pith.science/paper/CCMSJFOA

@misc{pith2026250518897,
  author       = {Pith},
  title        = {Pith review of: Improving Ad matching via Cluster-Adaptive Keyword Expansion and Relevance tuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CCMSJFOA}},
  note         = {Machine review of arXiv:2505.18897}
}
read the original abstract

In search advertising, keyword matching connects user queries with relevant ads. While token-based matching increases ad coverage, it can reduce relevance due to overly permissive semantic expansion. This work extends keyword reach through document-side semantic keyword expansion, using a language model to broaden token-level matching without altering queries. We propose a solution using a pre-trained siamese model to generate dense vector representations of ad keywords and identify semantically related variants through nearest neighbor search. To maintain precision, we introduce a cluster-based thresholding mechanism that adjusts similarity cutoffs based on local semantic density. Each expanded keyword maps to a group of seller-listed items, which may only partially align with the original intent. To ensure relevance, we enhance the downstream relevance model by adapting it to the expanded keyword space using an incremental learning strategy with a lightweight decision tree ensemble. This system improves both relevance and click-through rate (CTR), offering a scalable, low-latency solution adaptable to evolving query behavior and advertising inventory.

Figures

Figures reproduced from arXiv: 2505.18897 by the authors.

Figure 1
Figure 1. Flowchart of the offline and online process where 𝑔𝑖(𝑥) are the additional trees trained on a small dataset of new sponsored listings. The training objective is to minimize the error between model predictions and human-judged relevance scores. Given a dataset 𝐷 = {(𝑥𝑗 , 𝑦𝑗 )} 𝑁 𝑗=1, where 𝑦𝑗 represents human-judged relevance scores, the optimization is performed by minimizing the loss ℒ = ∑︀𝑁 𝑗=1 ℓ (︀ 𝑓adj(𝑥𝑗 ), 𝑦𝑗 … view at source ↗
Figure 2
Figure 2. (a) TPR trends across quantile thresholds, and (b) distribution of quantile thresholds across clusters. 95%–99.999%), we observed a substantial number of semantically unrelated or mismatched expansions. These included incorrect gender substitutions (e.g., “men’s shoes” expanded to “women’s sandals”) and numerical inconsistencies (e.g., “iPhone 13 case” matching “iPhone 12 accessories”), which are particularly proble… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LESER: Learning to Expand via Search Engine-feedback Reinforcement in e-Commerce

    cs.IR 2025-09 conditional novelty 6.0 of 10

    Fine-tuning a LLaMA model with GRPO on live search-engine feedback yields query expansions that retrieve more relevant and diverse product results in e-commerce search.

Reference graph

Works this paper leans on

9 extracted references · 7 canonical work pages · cited by 1 Pith paper

  1. [1]

    Azimi, A

    J. Azimi, A. Alam, R. Zhang, Ads keyword rewriting using search engine results, Association for Computing Machinery, New York, NY, USA, 2015. URL: https://doi.org/10.1145/2740908.2742739. doi:10.1145/2740908.2742739

    arXiv 2015

  2. [2]

    Mandal, I

    A. Mandal, I. K. Khan, P. S. Kumar, Query rewriting using automatic synonym extraction for e-commerce search, in: eCOM@SIGIR, 2019. URL: https://api.semanticscholar.org/CorpusID: 198120240

    work page 2019

  3. [3]

    Grbovic, N

    M. Grbovic, N. Djuric, V. Radosavljevic, F. Silvestri, R. Baeza-Yates, A. Feng, E. Ordentlich, L. Yang, G. Owens, Scalable semantic matching of queries to ads in sponsored search advertising, in: Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’16, Association for Computing Machinery, ...

    arXiv 2016

  4. [4]

    Semantic Equivalence of e-Commerce Queries

    A. Mandal, D. Tunkelang, Z. Wu, Semantic equivalence of e-commerce queries, arXiv preprint arXiv:2308.03869 (2023)

    work page Pith review arXiv 2023

  5. [5]

    S. Li, F. Lv, T. Jin, G. Li, Y. Zheng, Z. Tao, Q. Liu, X. Zeng, J. T.-Y. Kwok, Q. Ma, Query rewriting in taobao search, Proceedings of the 31st ACM International Conference on Information & Knowledge Management (2022). URL: https://api.semanticscholar.org/CorpusID:252904991

    work page 2022

  6. [6]

    W. Wu, G. Liu, H. Ye, C. Zhang, T. Wu, D. Xiao, W. Lin, X. Zhu, EENMF: an end-to-end neural matching framework for e-commerce sponsored search, CoRR abs/1812.01190 (2018). URL: http: //arxiv.org/abs/1812.01190.arXiv:1812.01190

    work page Pith review arXiv 2018

  7. [7]

    H. Shi, Z. Rao, Y. Wu, Z. Zhang, C. Wang, Keyword augmentation via generative methods, in: S. Malmasi, S. Kallumadi, N. Ueffing, O. Rokhlenko, E. Agichtein, I. Guy (Eds.), Proceedings of the 4th Workshop on e-Commerce and NLP, Association for Computational Linguistics, Online, 2021, pp. 33–37. URL: https://aclanthology.org/2021.ecnlp-1.5/. doi:10.18653/v1...

    work page doi:10.18653/v1/2021.ecnlp-1.5 2021

  8. [8]

    Bischl, M

    B. Bischl, M. Lang, L. Kotthoff, J. Schiffner, J. Richter, E. Studerus, G. Casalicchio, Z. M. Jones, mlr: Machine learning in r, Journal of Machine Learning Research 17 (2016) 1–5. URL: http: //jmlr.org/papers/v17/15-066.html

    work page 2016

Show all 9 references
  1. [9]

    Johnson, M

    J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with GPUs, IEEE Transactions on Big Data 7 (2019) 535–547

    2019

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.