{"id":"b435c6df-a29b-45cd-8df9-c643dd89dc64","arxiv_id":"2502.00131","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Aligning advertiser keyphrase relevance filters with eBay Search relevance judgments, instead of biased click data, improved relevance filtering and online sales metrics.","lead":"eBay Advertising researchers identify 'middleman bias': click-based training for keyphrase relevance only sees keyphrases that eBay Search already accepted, so relevance filters miss what Search would reject. They train a small cross-encoder on Search relevance judgments and report double-digit percentage gains in impressions, clicks, and purchases in an A/B test.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified auction-gate premise: the claim that click logs only contain keyphrases that pass Search's relevance filter (Section 1) is load-bearing but unsupported; if false, the middleman bias mechanism is invalid.","rationale":"The paper's central claim has two components: a diagnosis (middleman bias from sample selection in click logs) and a remedy (align with Search relevance). The online A/B test provides good evidence for the remedy against the Jaccard baseline, but the diagnosis is supported only by the asserted auction-gate mechanism in Section 1. Because this mechanism is unverified and specific to eBay's production systems, it is the weakest link in the causal story. If the mechanism is false, the paper's conceptual contribution collapses, even though the practical method might still work. A log audit would settle this directly. The reader flagged the same premise as a secondary concern in the weakest_assumption, so there is partial agreement; the reader's primary weakest assumption (Search label quality) is partially mitigated by the successful online test, whereas the auction-gate premise remains unaddressed by any empirical evidence in the paper.","tokens_in":7738,"tokens_out":10502,"duration_ms":103212,"concrete_test":"Audit 30 days of production logs: for every item-keyphrase pair that received at least one impression or click in the advertising system, determine whether that pair is also present in the set that passed Search's relevance filter. If any clicked pair is absent from the passed-filter set, quantify the fraction; even a small fraction (e.g., >0.5%) would indicate that the auction gate is not the exclusive path to logged clicks and that the middleman bias mechanism is incomplete or false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the premise in Section 1 that the auction mechanism ensures that click data logged will only contain keyphrases that pass Search's relevance filter. This premise is asserted, not demonstrated with logs. It is load-bearing because it is the sole mechanism for the novel 'middleman bias': if the click training distribution actually includes item-keyphrase pairs that fail Search's relevance filter (e.g., via other ad surfaces, post-filter logging, or filter bypass), then the described sample-selection bias does not exist as stated, and the justification for aligning Advertising with Search is weakened. The paper provides no empirical verification of this mechanism, and the claim is specific to eBay's internal systems. A false premise would not necessarily invalidate the practical method (training on Search labels), but it would invalidate the theoretical framing and the paper's explanation for why the method works. No code, data, or log documentation is provided to support the premise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This industrial paper, authored by eBay researchers, introduces the notion of 'middleman bias' in advertiser keyphrase relevance: because the auction process allegedly logs only keyphrases that pass eBay Search's relevance filter, click/sales-based training data lack examples of Search-irrelevant keyphrases, and models trained on such data inherit a sample-selection bias. To mitigate this, the authors curate 24 million/3 million item-keyphrase pairs labeled by Search's relevance judgment, train several bi-encoders and small cross-encoders, and select a bert-tiny cross-encoder based on offline F1 and inference latency. The deployed model is reported to increase impressions, clicks, bought items, and CVR by 20.17%, 13.01%, 26.04%, and 11.6%, respectively, over the Jaccard production baseline, while CTR remained neutral.","tokens_in":120,"tokens_out":6952,"duration_ms":109192,"significance":"The paper's main contribution is a large-scale industrial case study connecting sample-selection bias in ad relevance training to the 'middleman' role of search, together with a deployed remedy. The online A/B test against the production Jaccard baseline is the strongest evidence: it is a behavior-based, independent signal with reported p-values, and it supports the practical value of the proposed model. The paper also offers a useful latency-versus-quality comparison of bi-encoders and tiny cross-encoders at production scale. However, the middleman-bias mechanism is asserted rather than verified with logs, the Search labels used for both training and evaluation are not described, and offline metrics are reported without uncertainty quantification; these gaps currently prevent the paper from fully supporting its conceptual framing and its stronger comparative claims.","major_comments":[{"comment":"The assertion that 'the auction mechanism also ensures that the click data logged will only contain keyphrases which pass Search's relevance filter' is load-bearing for the middleman-bias mechanism, but no log-based evidence is provided. If click logs contain item-keyphrase pairs that did not pass Search's filter, for instance through other ad surfaces, post-filter logging, or filter bypass, the stated sample-selection mechanism is inaccurate. Please report the overlap between logged click/impression pairs and Search's filter decisions, and describe any exceptions.","section":"Section 1"},{"comment":"The provenance of the 24 million/3 million Search relevance labels is not described. The paper does not state how Search's relevance judgment is generated, its error rate, or whether the training and evaluation labels come from the same source. Because the models are trained and evaluated on the same label source, the offline 'alignment' F1 partly measures fit to that oracle rather than alignment with an independently validated notion of relevance. Please describe the label-generation process, quality-control measures, and add an independent evaluation set such as a human-judged sample or temporally held-out Search judgments.","section":"Section 3"},{"comment":"Precision, recall, and F1 depend on a decision threshold, but the threshold used for each neural model is not given. Without this, the model comparison in Table 1 is under-specified. In addition, the F1 differences among bert-mini (0.81), bert-tiny (0.80), and the contrastive bi-encoder (0.79) are small, and the text calls one of these a 'significant margin' without confidence intervals or significance tests. Please report the threshold-selection procedure and provide bootstrap confidence intervals or significance tests for the offline metrics.","section":"Section 4, Table 1"},{"comment":"The click-based IRNS dataset is curated with fixed thresholds of at least 30 impressions and a CTR lower bound of 0.05. These choices are presented without justification or sensitivity analysis. Because these thresholds determine which item-keyphrase pairs enter the in-batch negatives, they can materially affect the trained model. Please provide a sensitivity analysis or empirical justification for these cutoffs, and note whether the results are stable across reasonable variations.","section":"Section 3, IRNS data curation"},{"comment":"The online A/B test description is incomplete: there is no traffic split, duration, number of items/sellers, or specification of the statistical test used for the reported p-values. The text states that the model 'delivered on all fronts' while also reporting that CTR was neutral (p=0.17); please reconcile this wording and provide the A/B setup and metric definitions so that the headline claims can be properly interpreted.","section":"Section 4, online A/B test"}],"minor_comments":[{"comment":"The phrase 'with a lower lower bound of 0.05 CTR' contains a typo; it should be 'with a lower bound of 0.05 CTR'.","section":"Section 3"},{"comment":"The text says 'the bi-encoders of IRNS and Constrative'; 'Constrative' should be 'Contrastive'.","section":"Section 4"},{"comment":"The caption contains the typo 'kephrases'; it should be 'keyphrases'.","section":"Figure 2 caption"},{"comment":"The note 'Darker colors represent more favorable scores' is not meaningful in a plain-text table; either add actual visual highlighting or remove the note.","section":"Table 1"},{"comment":"The column header 'Diff Full' is ambiguous; please make it explicit, for example 'Diff / Full latency (hours)', and state the benchmark configuration in the caption or a footnote.","section":"Table 1"},{"comment":"The sentence claiming the data 'guarantees proper representation of all categories while still maintaining preferential bias of models towards high performing categories' is self-contradictory; please rephrase to state the intended trade-off.","section":"Section 3"},{"comment":"Please define the online metrics exactly (e.g., CVR = bought items / clicks) and specify the statistical tests used to obtain the p-values, as the current description is insufficient for reproducibility.","section":"Section 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript fits the industrial/companion track of WWW and the online A/B result is compelling. The central risk is that the middleman-bias framing is asserted rather than demonstrated: the auction-gate premise and the nature of the Search labels are both unverified, and the offline evaluation is partly circular. These issues are addressable with additional analyses rather than being fatal, so I recommend major revision rather than rejection. The paper would also benefit from a short discussion of whether the proposed alignment approach might inherit biases from Search's own relevance judgments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid applied IR paper from eBay with a real production A/B test. The 'middleman bias' framing is new packaging for sample selection bias, and the authors admit that. The actual contribution is showing that training an ad-keyphrase relevance model on Search's own relevance judgments beats a Jaccard baseline in production, with double-digit lifts in impressions, clicks, and bought items. That result is worth taking seriously.\n\nWhat's good: the paper is honest about its own previous failures—a human-judgment BERT model that had to be pulled, an xgboost CTR model that underperformed Jaccard offline. The cross-encoder versus bi-encoder comparison, including latency, is useful: bert-tiny, a 2-layer cross-encoder, beats more complex bi-encoders and matches their latency. The A/B test is a real behavior-based signal, independent of the offline labels.\n\nWhere it's soft: first, the 'middleman bias' mechanism rests on the claim that the auction ensures click logs only contain keyphrases that passed Search's relevance filter. That's asserted in Section 1 but not backed with logs. It's plausible for eBay's internal system, but it is load-bearing for the framing: if the gate isn't perfect, the described bias doesn't hold, even though the practical fix (train on Search labels) could still work. Second, the 24M/3M Search relevance labels are never described—how they're generated, error rate, whether it's a model or human judgments. Since the offline evaluation uses the same label source the model trained on, those F1 numbers are partly tautological. The A/B test is the real evidence, and it's good. Third, offline scores have no error bars or significance tests. Minor: no code or data, no sensitivity analysis for thresholds. These are all expected in a 5-page industry paper.\n\nBottom line: the central claim—deploying this model improved key business metrics in production—is supported by the A/B test. The theoretical framing deserves scrutiny but doesn't undermine the empirical result. This is a useful paper for anyone working on ad relevance, sample selection bias, or production IR. It deserves peer review; a decent referee will ask for the label-generation details and log evidence for the gate claim.","headline":"Solid industrial paper: the middleman-bias framing is new but unverified at the gate; the production A/B test is the real contribution.","tokens_in":8452,"tokens_out":3223,"would_cite":true,"duration_ms":27749,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that auction-log clicks hide the negatives an ad-keyphrase model must learn, so it retrains on Search's own relevance labels and lifts bought items by 26 percent.","keywords":["middleman bias","advertiser keyphrase recommendation","sample selection bias","search relevance alignment","cross-encoders","bi-encoders","online advertising","e-commerce"],"falsifier":"A direct check is to inspect the auction logs and verify that every logged click or sale indeed came from an item-keyphrase pair that passed Search's relevance filter; if any logged outcome bypassed the filter, the asserted data-generation mechanism fails. A second check is to take a random sample of the 24 million training labels, have independent human annotators re-judge the same item-keyphrase pairs, and measure agreement with Search's pass/fail labels; low agreement would mean the alignment target is too noisy to serve as ground truth.","tokens_in":7561,"feed_emoji":"📈","tokens_out":8601,"duration_ms":79889,"temperature":0.7,"pith_summary":"This paper argues that eBay's advertiser keyphrase recommendations were trained on the wrong signal. Because only keyphrases that pass Search's relevance filter ever win auction impressions, the click and sales logs used to train the relevance model contain no examples of the failures it exists to catch; the authors call this 'middleman bias.' Their proposed fix is to relabel the training data with Search's own relevance judgments, using 24 million item-keyphrase pairs judged pass/fail by Search, and to model that alignment directly. A small two-layer cross-encoder trained this way scored offline F1 of 0.80 against the production Jaccard model's 0.70, and in an online A/B test it increased bought items by 26 percent while surfacing 7 percent fewer keyphrases to sellers. If the claim holds, marketplace advertising systems can debias keyphrase recommendations by treating the gate's relevance output as ground truth rather than downstream outcomes.","feed_headline":"Aligning ad-keyphrase filters to Search lifts bought items 26%","feed_subtitle":"eBay replaces Jaccard token overlap with a two-layer BERT trained on Search's own relevance labels, boosting conversions.","key_machinery":"The central object is 'middleman bias,' a named form of sample-selection bias: the auction sits between Advertising and Search, and because the auction only logs keyphrases that passed Search's relevance filter, the click-based training set is missing all the negatives Advertising must catch. The fix is to realign Advertising's relevance filter with Search's, using 24 million item-keyphrase pairs labeled by Search's own pass/fail judgment as ground truth. The models that carry the argument are bi-encoders and cross-encoders; the deployed model is bert-tiny, a two-layer transformer cross-encoder with hidden size 128, chosen because its 1.5-hour differential batch inference latency matches the bi-encoders while its offline F1 of 0.80 beats the Jaccard baseline's 0.70.","core_discovery":"The paper's central claim is that advertiser keyphrase relevance is not a property of the item-keyphrase pair alone, but of a two-system interaction in which Search acts as a middleman: keyphrases Search deems irrelevant never reach the auction, so click data only ever records pairs that passed Search's filter. Training on such data teaches the Advertising model to under-filter, since it never observes the negatives it should block. The paper's discovery is that relabelling the training data with Search's own pass/fail relevance judgments removes this bias, and that even a tiny cross-encoder trained on those labels agrees with Search far better than the token-overlap Jaccard baseline and improves real auction outcomes. Specifically, the deployed bert-tiny model raised impressions by 20.17 percent, clicks by 13.01 percent, bought items by 26.04 percent, and conversion rate by 11.6 percent, with click-through rate unchanged.","pith_inferences":["The same gate-label alignment should transfer to any marketplace where an intermediate filter truncates logged outcomes before training; a direct test would compare gate-label-trained filters with click-trained filters on held-out human judgments.","The paper's note that the new model restored short head keyphrases suggests a testable mechanism for the bought-item gain: releasing single-token keyphrases penalized by Jaccard may drive most of the improvement, so the benefit could be concentrated in head terms rather than spread across the tail.","Since Search's relevance filter is itself a learned system, alignment labels will drift as Search updates; a rolling agreement metric between Advertising and Search would tell when the 24-million-pair training set needs refreshing.","Independent human relevance judgments could separate two readings of the result: alignment with Search improves true relevance, or it merely conforms Advertising to a filter that Search itself should correct."],"forward_implications":["Relevance filters in advertising platforms can be debiased by relabeling with the gate's own relevance decisions instead of downstream clicks, eliminating the missing-negative problem.","A two-layer cross-encoder can deliver this alignment at production scale: 1.5 hours for the daily differential batch and near-real-time serving for new or revised items.","Aligning Advertising with Search improves marketplace outcomes: in the reported A/B test, bought items rose 26.04 percent and impressions 20.17 percent while the number of keyphrases shown to sellers fell 7 percent.","Cross-encoders, even very small ones, are the better model family for this agreement task, beating all bi-encoder training objectives on offline F1."],"supporting_citations":[{"why":"Frames the sample-selection-bias problem and the debiasing-by-extra-data approach that justifies using Search's labels instead of clicks.","marker":"[7]"},{"why":"Provides the contrastive loss objective that produced the best bi-encoder results.","marker":"[8]"},{"why":"Supplies the in-batch random negative sampling used for the click-based training variant.","marker":"[10]"},{"why":"Defines the Jaccard index that is the production baseline and the main comparison point.","marker":"[11]"},{"why":"Models cascaded ranking systems as flows, the conceptual basis for aligning Advertising and Search as dependent systems.","marker":"[14]"},{"why":"Gives the cross-encoder versus bi-encoder comparison and the concatenation/difference feature construction used by the softmax bi-encoder.","marker":"[15]"},{"why":"Supports in-batch negative sampling as a technique that often yields better results.","marker":"[16]"},{"why":"Provides the tiny and mini model architectures whose low inference latency makes the semantic filter deployable.","marker":"[17]"}],"fun_headline_variants":["Search's relevance labels lift ad-bought items 26%","Middleman bias curbed: align keyphrases to Search, buys up 26%","Train ad filters on Search labels: 26% more bought items","Search alignment boosts ad relevance: 26% more buys","Middleman bias: keyphrase relevance realigned to Search for 26% lift"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach depends on treating Search's pass/fail relevance filter as correct ground truth for advertiser keyphrase relevance, and on the claim that auction logs only record keyphrases that passed that filter; if either premise fails, the debiasing story collapses.","fun_headline_variants_meta":{"raw":{"variants":["Search's relevance labels lift ad-bought items 26%","Middleman bias curbed: align keyphrases to Search, buys up 26%","Train ad filters on Search labels: 26% more bought items","Search alignment boosts ad relevance: 26% more buys","Middleman bias: keyphrase relevance realigned to Search for 26% lift"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":2985,"prompt_tokens":879,"completion_tokens":2106,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":2010}},"tokens_in":495,"tokens_out":2106,"duration_ms":14490,"temperature":1.0,"reasoning_tokens":2010,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T20:04:36.615013+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is to inspect the auction logs and verify that every logged click or sale indeed came from an item-keyphrase pair that passed Search's relevance filter; if any logged outcome bypassed the filter, the asserted data-generation mechanism fails. A second check is to take a random sample of the 24 million training labels, have independent human annotators re-judge the same item-keyphrase pairs, and measure agreement with Search's pass/fail labels; low agreement would mean the alignment target is too noisy to serve as ground truth.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames the sample-selection-bias problem and the debiasing-by-extra-data approach that justifies using Search's labels instead of clicks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Models cascaded ranking systems as flows, the conceptual basis for aligning Advertising and Search as dependent systems."}],"review_version":1}