{"id":"3d63e39f-88e4-4820-946b-04c42885df42","arxiv_id":"2505.18897","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A production ad-matching system expands seller keywords with embeddings, cluster-adaptive thresholds, and a stacked relevance model, yet its A/B data do not show the claimed CTR and relevance gains.","lead":"This report reviews an eBay ad-matching system that expands seller keywords with dense embeddings, cluster-specific similarity thresholds, and a stacked relevance model. The reported two-week A/B test shows impression gains but not the headline CTR and relevance improvements.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own A/B table contradicts the abstract: all three variants show negative CTR, negative BI/click, and negative average relevance, so the central claim that the system improves both relevance and CTR is unsupported.","rationale":"I read the paper as an industrial system description whose main scientific claim is the abstract's statement that the proposed system 'improves both relevance and click-through rate (CTR).' For that claim to hold, the online A/B experiment would need to show positive, statistically reliable CTR and relevance improvements. The reported Table 3 shows the opposite: every variant has negative CTR, negative BI/Click, negative slot-neutral CTR, and negative average relevance, with only impressions and CPC increasing. The text itself acknowledges a 'slight dip in CTR' but then asserts net gains in BI/click and revenue despite the table showing negative BI/Click and no revenue column. This is a direct internal contradiction, not a matter of differing from an external consensus. The reader's rationale already identified this table contradiction, which is why I partially agree, but the reader's formal 'weakest_assumption' focused instead on the residual-structure assumption of the stacked relevance model. That assumption is real and is even self-identified in Section 3.3, but it is secondary: even if the stacked model worked perfectly, the online metrics still would not support the central claim. I also checked whether the negative metrics could be definitionally inverted (e.g., lower values better); the paper's own language ('slight dip in CTR,' 'neutral' for slot-normalized CTR, 'higher ASP' as compensation) indicates that higher is better and these are genuine degradations. The offline RMSE improvements in Section 4.1.1 are suggestive but not connected to the online relevance metric, which also moves negatively. The system has plausible engineering components—document-side expansion, cluster-adaptive thresholds, and incremental GBDT stacking—and the paper does not overclaim mechanistic novelty, but the central quantitative evidence is self-contradictory. No formal verification or released code is available to independently check the pipeline. Since the central claim is unsupported by the paper's own data, I maintain the reader's REJECT verdict.","tokens_in":7085,"tokens_out":3627,"duration_ms":34339,"concrete_test":"Re-analyze the raw two-week A/B logs, or at minimum obtain the confidence intervals and metric definitions behind Table 3, and compute 95% bootstrap or delta-method intervals for CTR, BI/Click, Slot Neutral CTR, and Avg Relevance for the final variant (Emb+Cluster threshold+Relevance model) versus the shared control. If the interval for any of these metrics excludes zero in the positive direction, the claim survives; if the intervals include zero or are negative, the abstract must be revised and the rejection stands.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires that the deployed system improves both relevance and CTR, yet Table 3, the only online A/B evidence, reports negative values for every relevance and engagement metric across all three variants: CTR lifts are -3.1%, -1.26%, and -0.62%; BI/Click lifts are -3.12%, -1.53%, and -1.96%; Slot Neutral CTR lifts are -1.23%, -0.65%, and -0.13%; and Avg Relevance lifts are -0.47%, -0.25%, and -0.18%. Only Impressions (7.48%, 3.74%, 3%) and CPC (4.72%, 3.52%, 3.05%) are positive, and the paper interprets the CPC increase as rising competition rather than a quality improvement. The Section 4.2 narrative states that 'subsequent refinements significantly recovered CTR and yielded net gains in BI/click and revenue,' but the table shows BI/Click still negative and no revenue metric is reported. The offline RMSE improvements in Section 4.1.1 are not connected to the online metrics by any calibration or ablation, so they do not rescue the online relevance claim. The abstract's phrase 'improves both relevance and click-through rate (CTR)' is directly contradicted by the paper's own numbers; this is an internal inconsistency, not merely a disagreement with external consensus. The stacked relevance model's residual-structure assumption flagged in Section 3.3 is a secondary risk, but even if that model worked perfectly offline, the reported online metrics would not support the headline claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a document-side semantic keyword expansion system for sponsored search at eBay. It uses a pre-trained siamese embedding model to find semantically related ad keywords, k-means clustering with per-cluster similarity quantile thresholds to control expansion precision, and an incrementally trained shallow decision-tree ensemble added to a baseline GBDT relevance model. The authors report offline TPR results for threshold selection, offline RMSE improvements for the stacked relevance model, and a two-week A/B test comparing three variants (embeddings only, plus cluster thresholds, plus relevance model) against a token-based expansion baseline. The abstract claims the system 'improves both relevance and click-through rate (CTR)' and the conclusion repeats that the framework enhances recall and precision.","tokens_in":7480,"tokens_out":2912,"duration_ms":27308,"significance":"If the central claims were supported, the paper would offer a practical way to extend token-based ad retrieval with semantic expansions while controlling precision via density-aware thresholds and a lightweight relevance adjustment, a combination that could transfer to other e-commerce marketplaces. The paper is also candid about its assumptions, particularly in Section 3.3, and it reports a real A/B experiment rather than only offline metrics. However, the paper's own online results contradict the headline claim: Table 3 shows negative CTR, negative bought-items-per-click, and negative average relevance for all three variants relative to the token-based baseline. The reported positive impression lift is credible, but the claimed relevance and engagement improvements are not established by the submitted evidence.","major_comments":[{"comment":"The abstract states that the system 'improves both relevance and click-through rate (CTR)', but Table 3 reports the final variant (Emb+Cluster threshold+Relevance model) with CTR lift -0.62%, BI/Click lift -1.96%, Slot Neutral CTR lift -0.13%, and Avg Relevance lift -0.18% relative to the token-based baseline. Section 4.2 claims that 'subsequent refinements significantly recovered CTR and yielded net gains in BI/click and revenue,' yet the table shows BI/Click still negative and contains no revenue metric at all. This is an internal inconsistency in the manuscript's central evidentiary claim, not merely a disagreement with an external benchmark.","section":"Abstract and Section 4.2, Table 3"},{"comment":"The offline RMSE improvements for the stacked model (greater than 4% for Fair/Bad items and greater than 1% for Excellent items) are reported without sample sizes, standard errors, or confidence intervals, and Table 2 shows only five selected examples. More importantly, no calibration, ablation, or analysis connects these held-out RMSE improvements to the online metrics in Table 3; the final variant's average relevance online is negative, so the offline gains do not rescue the online relevance claim.","section":"Section 4.1.1 and Table 3"},{"comment":"The production quantile p=99.9999 is selected on the basis of True Positive Rate computed with ChatGPT-3.5-assisted labeling of expansions, and the same TPR metric is then used to justify the post-processing filters and the cluster-thresholding quality. No validation of the ChatGPT-3.5 labels against human relevance judgments is reported, and no inter-annotator agreement is given. This makes the threshold-selection evidence circular with respect to the paper's own definition of expansion quality, and the validity of the entire precision-control mechanism depends on that one proxy metric.","section":"Section 4.1, Figure 2a"},{"comment":"The paper itself identifies the load-bearing risk: the stacked trees are beneficial only if residual errors of the baseline GBDT are structured enough for one or two shallow trees to correct. The online results in Table 3, where the relevance model refinement improves CTR from -1.26% to -0.62% but leaves Avg Relevance at -0.18% and BI/Click at -1.96%, suggest that the residual-structure assumption did not hold strongly in production. This should be treated as a central limitation, and the conclusion should not claim relevance improvement until this is reconciled.","section":"Section 3.3 and Eq. (2)"}],"minor_comments":[{"comment":"There is an inconsistency in the similarity threshold direction: Section 3.2 says variants v are kept when sim(k,v) ≤ τ_m, while Algorithm 1 line 8 keeps neighbors when sim(e_k,e_n) ≥ τ_m. The text also calls τ_m a distance threshold but defines it via a quantile of distances to the cluster centroid, while Figure 2b labels it a quantile threshold; please clarify whether τ_m is a cosine distance or cosine similarity cutoff.","section":"Section 3.2 and Algorithm 1"},{"comment":"The notation 't_r =< 2' is ambiguous; I assume it means t_r ≤ 2. Also, 'refereed' should be 'referred'. In addition, calling this a 'stacked model' is unconventional, since the additional trees are added directly to the baseline prediction rather than meta-learned on baseline predictions.","section":"Section 3.3, Eq. (2)"},{"comment":"The base relevance model is described both as a 'Gaussian regression' and as 'Gradient Boosted Decision Tree (GBDT)' trained with more than 100 trees; please clarify whether the base model is a GBDT with a Gaussian regression loss or a different model, since this affects the interpretation of the additive-tree adjustment.","section":"Section 3.3, paragraph 2"},{"comment":"The description of k-fold resampling for clustering stability is incomplete: no definition of assignment consistency, no number of folds, and no quantitative stability result is given. The elbow-method choice of M=1000 is also reported without showing the WCSS curve or sensitivity.","section":"Section 4.1, first paragraph"},{"comment":"The columns 'Old Rel.', 'Adj. Rel', and 'True Rel' mix different scales: the first two are model regression scores on a 0–5 scale, while 'True Rel' is a categorical label (Bad, Fair, Good). Please use a consistent display, such as showing human label values with model scores, and state the number of evaluation examples used.","section":"Table 2"}],"recommendation":"reject","confidential_remarks":"The paper has a genuinely useful engineering component: the document-side expansion with cluster-adaptive thresholds is plausible and the impression lift in Table 3 is positive. However, the central advertised outcome—simultaneously improved relevance and CTR—is contradicted by the paper's own A/B table, and the offline evidence is too weak to substitute. If the authors were to reframe the contribution as a coverage/impressions optimization with neutral-to-slightly-negative engagement and provide a revenue metric, the work might be salvageable as a negative-result or systems paper; as submitted, the headline claim is unsupported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2505.18897. The system idea is solid: document-side semantic expansion, cluster-adaptive quantile thresholds, and a small stacked GBDT for relevance adjustment form a coherent, deployable pipeline. The paper is honest about the residual-structure assumption in Section 3.3 and gives a concrete implementation with FAISS and Airflow. That part is worth reading.\n\nThe big problem is the evaluation. Table 3 shows all three variants with negative CTR (-3.1% to -0.62%), negative BI/click (-3.12% to -1.96%), and negative average relevance (-0.47% to -0.18%) relative to the token-based baseline. Only impressions and CPC go up, and the paper interprets the CPC rise as competition, not quality. Then Section 4.2 says 'subsequent refinements significantly recovered CTR and yielded net gains in BI/click and revenue,' but the table shows BI/click still negative and no revenue metric is reported. That is an internal contradiction, not a matter of interpretation. The abstract's 'improves both relevance and click-through rate (CTR)' is directly contradicted by the paper's own numbers.\n\nThe offline RMSE improvements in Section 4.1.1 are a minor positive, but they come without sample sizes or error bars and are never calibrated to the online metrics, so they don't rescue the online claim. There is also mild circularity: the global quantile p is chosen using the same TPR metric that later justifies the thresholding.\n\nThat said, the paper is not worthless. The integration of known components is genuinely new at the system level, and the authors clearly understand the trade-offs; they explicitly flag the residual-structure risk. If they can either correct the abstract to match the data or provide the missing revenue and BI numbers that support their narrative, the paper becomes a useful industrial case study. As it stands, it's a good design document with an unsupported headline.\n\nFor a reviewer: I'd send it out rather than desk-reject, because it's a real production system with real A/B data and the flaws are fixable. But I'd ask for major revision and a rewritten abstract. For a reading group, it's a maybe—good for a discussion of how easy it is to overclaim from system metrics.\n\nBottom line: cite it if you're building a similar pipeline, but don't cite the headline result.","headline":"Useful industrial pattern, but the A/B table contradicts the abstract's 'improves both relevance and CTR'—treat the headline result as unsupported.","tokens_in":7954,"tokens_out":2240,"would_cite":false,"duration_ms":19461,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"eBay shows document-side semantic keyword expansion can widen ad reach without wrecking relevance.","keywords":["semantic keyword expansion","dense embeddings","cluster-adaptive thresholding","ad relevance","gradient boosted decision trees","broad match","e-commerce search","nearest neighbor search"],"falsifier":"On a held-out set of human-judged query-item pairs drawn from keyword-expansion inventory, compute the base GBDT's RMSE and the stacked model's RMSE; if the stacked model does not achieve a lower RMSE (or at least the >4% reduction on Bad/Fair items claimed), the central relevance claim is unsupported. A second check: compare a global similarity threshold against the cluster-adaptive thresholds on the same TPR/impression-lift curve; if the global threshold matches or beats the cluster-adaptive one in offline TPR at matched impression lift, the adaptive-threshold contribution collapses.","tokens_in":6930,"feed_emoji":"🎯","tokens_out":4498,"duration_ms":37830,"temperature":0.7,"pith_summary":"This paper reports a production ad-matching system at eBay that expands seller-supplied keywords on the document side, so a query like \"Apple phone cover\" can match an ad keyword \"iPhone case\" without rewriting the query. The authors claim that a pre-trained siamese embedding model, nearest-neighbor search, and cluster-specific similarity thresholds can add semantically related keyword variants while keeping precision, and that a lightweight stacked tree model fixes the relevance score of the newly matched items. They run a two-week A/B test showing progressive gains in impressions, bought items per click, and ad revenue as each component is added, with slot-normalized CTR staying neutral. If this holds, token-based ad retrieval can be upgraded to semantic matching without replacing the existing infrastructure.","feed_headline":"Semantic keyword expansion widens ad reach, protects relevance","feed_subtitle":"Cluster-tuned thresholds and a stacked relevance model let eBay broaden matches without sacrificing precision.","key_machinery":"The pipeline rests on three objects. (1) A pre-trained siamese embedding model (micro-BERT encoder fine-tuned on eBay titles, trained with contrastive learning) that maps ad keywords to dense vectors; it supplies the semantic space in which cosine distance approximates intent similarity. (2) A k-means clustering of that space into 1,000 clusters per market, where each cluster's threshold is the $p$-th quantile of intra-cluster centroid distances; a single global quantile $p = 99.9999$ governs the precision-recall tradeoff, and the threshold adapts to local semantic density. (3) A stacked relevance model $f_{adj}(x) = f_{GBDT}(x) + \\sum_{i=1}^{t_r} g_i(x)$ with at most two shallow trees ($t_r < 2$, depth $\\le 5$) trained on human-judged relevance scores for expansion inventory, which corrects the baseline GBDT's bias on expanded matches. The FAISS flat index performs the nearest-neighbor search, and gender/numeric post-processing filters remove attribute-inconsistent expansions. Together these pieces let the system run daily refreshes and produce final query-item matches through a market-specific relevance threshold.","core_discovery":"The central claim is that broad-match ad retrieval can be extended on the document side: instead of expanding the user query, the system expands the advertiser's keyword into semantically related variants using dense embeddings and nearest-neighbor search, then uses two guards to keep quality. First, a k-means partition of the keyword embedding space sets a per-cluster similarity threshold at a global quantile of intra-cluster distances, so dense, specific clusters get stricter cutoffs and broad, ambiguous clusters get looser ones. Second, because an expanded keyword may retrieve items only partially aligned with the original intent, the production GBDT relevance model is augmented by up to two shallow trees trained on human-judged labels for expansion inventory, and a market-specific threshold is applied to final matches. The A/B results show that embedding-only expansion raises impressions but hurts CTR, cluster thresholding recovers most of the CTR loss, and relevance model refinement brings CTR nearly back to baseline while increasing bought items per click and ad revenue.","pith_inferences":["The residual-correction design implies the base GBDT must already be strong; the method is a bias-correction patch, not a replacement, so its benefit is bounded by how much of the expansion error is systematic rather than random.","The same cluster-adaptive thresholding recipe could transfer to other marketplaces or to verticals like jobs or real estate, provided per-market clusters are retrained as inventory shifts.","Because thresholds are derived from a static embedding space, drift in buyer vocabulary or catalog mix would require periodic re-clustering and threshold recomputation; the paper flags this as future work.","A natural extension is to let the quantile $p$ or the thresholds react to query-side click feedback, which would make the expansion adapt to user behavior rather than only to human-labeled relevance."],"forward_implications":["Token-based ad retrieval systems can gain semantic coverage by expanding advertiser keywords offline, leaving query processing and existing retrieval infrastructure unchanged.","Cluster-adaptive thresholds let one global hyperparameter (the quantile $p$) control precision per region of embedding space, so broad terms get recall and specific terms get precision.","Incremental stacking of one or two shallow trees on human labels lets the relevance model adapt to new inventory without a full retrain, reducing risk of overfitting when new data is limited.","Two-week A/B evidence indicates that precision tuning and relevance filtering are the components that turn raw impression gains into revenue and bought-items gains.","The containment matching rule (expanded keyword must be a token subset of the query) keeps expansion interpretable and limits brand conquesting."],"supporting_citations":[{"why":"Supplies the pre-trained siamese model that generates dense keyword embeddings for semantic equivalence.","marker":"[4]"},{"why":"Provides the FAISS library with a flat index used for nearest neighbor search over keyword embeddings.","marker":"[9]"},{"why":"Underlies the GBM implementation used for the incremental stacked relevance trees.","marker":"[8]"},{"why":"Demonstrates embedding-based semantic matching of queries to ads, the paradigm this system extends.","marker":"[3]"},{"why":"Represents the earlier keyword-rewriting approach this work contrasts with.","marker":"[1]"},{"why":"Shows a large-scale query rewriting system in e-commerce that motivates the coverage goal.","marker":"[5]"}],"fun_headline_variants":["Cluster-tuned keyword expansion lifts ad revenue without hurting CTR","Semantic keyword expansion, cluster-filtered for precision","Advertiser-side expansion that boosts revenue and keeps relevance","Smart thresholds make keyword expansion pay off","Expanding ad keywords, not queries, improves matching"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole relevance-tuning stage works only if the baseline GBDT's residual errors on expanded-match items are structured enough for one or two shallow trees to correct; if those residuals are mostly noise, the stacked trees add no signal and the system's final quality guard collapses.","fun_headline_variants_meta":{"raw":{"variants":["Cluster-tuned keyword expansion lifts ad revenue without hurting CTR","Semantic keyword expansion, cluster-filtered for precision","Advertiser-side expansion that boosts revenue and keeps relevance","Smart thresholds make keyword expansion pay off","Expanding ad keywords, not queries, improves matching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000481,"raw_usage":{"total_tokens":2356,"prompt_tokens":901,"completion_tokens":1455,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":1380}},"tokens_in":517,"tokens_out":1455,"duration_ms":10865,"temperature":1.0,"reasoning_tokens":1380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:24:06.170279+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out set of human-judged query-item pairs drawn from keyword-expansion inventory, compute the base GBDT's RMSE and the stacked model's RMSE; if the stacked model does not achieve a lower RMSE (or at least the >4% reduction on Bad/Fair items claimed), the central relevance claim is unsupported. A second check: compare a global similarity threshold against the cluster-adaptive thresholds on the same TPR/impression-lift curve; if the global threshold matches or beats the cluster-adaptive one in offline TPR at matched impression lift, the adaptive-threshold contribution collapses.","supporting_citations":[{"cited_title":"Semantic Equivalence of e-Commerce Queries","cited_arxiv_id":"2308.03869","evidence_quote":"Supplies the pre-trained siamese model that generates dense keyword embeddings for semantic equivalence."},{"cited_title":"Johnson, M","cited_arxiv_id":null,"evidence_quote":"Provides the FAISS library with a flat index used for nearest neighbor search over keyword embeddings."},{"cited_title":"Bischl, M","cited_arxiv_id":null,"evidence_quote":"Underlies the GBM implementation used for the incremental stacked relevance trees."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows a large-scale query rewriting system in e-commerce that motivates the coverage goal."}],"review_version":1}