{"id":"61a3d2a7-2dcb-47d0-9150-b7d99036637b","arxiv_id":"2607.23121","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A gated hybrid retrieval system shows rule-based keywords beat LLM queries for retargeting but lose for prospecting, and routing 10% of users to LLM semantic search raises ad conversions by 27.6%.","lead":"SMART mixes cheap rule-based keyword search with LLM-generated queries, using a learned quality gate to call the expensive LLM for only about 10% of users. In a two-week ad-serving test, it raised conversion rate by 27.6% over an embedding-only baseline while cutting LLM cost by 90%.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Online A/B confound: +27.6% CR lift is not attributable to gated LLM routing because the treatment arm changes four components at once; no online ablation isolates the gate/LLM path.","rationale":"The reader's weakest assumption identifies the same concern I find most load-bearing: the online A/B test does not isolate the contribution of the gated LLM path. The paper's headline result is a whole-system comparison, while its narrative and cost claim depend on the gate and LLM path being the key drivers. I considered whether a more fundamental issue exists—e.g., the LLM-as-a-judge circularity in the Relevance Score, or threshold overfitting in choosing the 10% operating point. These are real but secondary: the online conversion metric is a genuine business outcome, and the offline decomposition is supported by several robustness checks (delayed-label sensitivity in Figure 3, temporal gate generalization in Table 9, run-to-run stability in Table 8). The confounding of components is the most direct threat to the central causal claim. I would not reject the paper—the engineering system may well work—but the evidence as presented warrants a conditional verdict, exactly as the reader concluded. No change to the reader's verdict is needed.","tokens_in":12810,"tokens_out":4621,"duration_ms":47048,"concrete_test":"Run a follow-up online A/B on the same traffic with at least three randomized arms for 1–2 weeks: (A) production U2I baseline; (B) SMART with gate disabled / LLM path removed, i.e., keyword/BM25 + LambdaMART + MMR only; (C) full SMART. If CR(B)/CR(A) is not statistically different from CR(C)/CR(A) (with pre-registered equivalence bounds), then the gated LLM path contributes little and the central attribution fails; if C clearly beats B, the gate/LLM path is supported. A secondary arm replacing the learned gate with a random 10% selector would separately test the coverage-gap routing signal. Report CR with confidence intervals and pre-registered significance thresholds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that SMART's +27.6% conversion-rate lift over a strong U2I embedding baseline validates the adaptive gating design: route ~10% of users to LLM semantic prospecting and cut LLM cost by 90% (Abstract; §5.4 Table 10). The load-bearing condition is that the online lift can be attributed to the gated LLM path. That condition is not met by the reported experiment. The treatment differs from the baseline in four simultaneous changes: (1) the rule-generated keyword/BM25 path (Algorithm 1), (2) the LambdaMART reranker, (3) the MMR diversity layer, and (4) the GBDT gate + LLM/ANN path (Figure 1, §4.1). §5.4 itself notes that Sections 3 and 5.1–5.3 hold indices and rerankers fixed to isolate query-generation effects, but the production A/B does no such isolation: full SMART is compared head-to-head against the two-tower U2I ANN model. No online arm runs, e.g., keyword/BM25 + reranker without the LLM path, or a random-gate 10% arm. The offline results (Tables 1–2, Figure 2) show keyword+BM25 already dominates on retargeting, so a large part of the online CR gain could plausibly come from adding the exact-match keyword path or the reranker, independent of the LLM gate. Thus the specific attribution of +27.6% to adaptive LLM routing, and the associated cost-efficiency claim, is not established by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SMART proposes a hybrid retrieval architecture for Dynamic Product Ads (DPA) that separates two retrieval objectives: a zero-cost rule-generated keyword path on a BM25 index for retargeting, and a gated LLM-query path on a dense ANN index for semantic prospecting. A GBDT quality gate audits the keyword pass output and routes only about 10% of users to the LLM path. The paper claims, based on offline studies (Tables 1–6, Figure 2) and a 2-week online A/B test (Table 10), that this gated pairing captures most of the semantic-prospecting gains at a 90% reduction in LLM cost, and improves ad conversion rate by +27.6% over a two-tower embedding baseline. The offline evaluation includes a temporal holdout, a delayed-label sensitivity check, and run-to-run stability analysis.","tokens_in":13269,"tokens_out":4794,"duration_ms":51530,"significance":"If the results hold, SMART is a practically significant contribution: it provides production-scale evidence that rule-generated keywords and LLM-generated queries serve complementary retrieval objectives, and that output-auditing gates can be an effective cost-control mechanism for LLM-augmented retrieval. The paper's main strengths are its use of a strict temporal holdout (Table 9), a delayed-label robustness check with bootstrap intervals (Figure 3), run-to-run stability verification (Table 8), and a live-traffic A/B test. The decomposition claim is actionable for any industrial retrieval system with fast heuristic and expensive neural components. However, the load-bearing online attribution is undermined by a multi-component treatment arm, and the offline gating evaluation contains a fit-then-evaluate loop on the same metric used for training and threshold selection.","major_comments":[{"comment":"The online A/B test compares full SMART against a two-tower U2I ANN baseline, but the treatment differs from the baseline in at least four components at once: the rule-generated keyword/BM25 path (Algorithm 1), the LambdaMART reranker, the MMR diversity layer, and the GBDT gate plus LLM/ANN path. The abstract and conclusion attribute the +27.6% conversion-rate lift to adaptive LLM routing and the associated 90% cost reduction, but no online arm isolates the gate or even the LLM path. Tables 1–2 show that the keyword/BM25 path alone is strong on retargeting, so a substantial portion of the lift could plausibly come from components independent of the LLM gate. The paper should either add an online ablation (e.g., keyword+BM25+reranker without the LLM path, or a fixed/random-gate 10% arm) or reframe the claim as a full-system effect rather than evidence for the adaptive gating design.","section":"§5.4, Table 10"},{"comment":"The routing gate is trained on labels defined as 'SMART beats keywords on CatRecall@200' across two generation cycles, and the same CatRecall@200 metric is then used both to sweep the gate threshold in Figure 2 and to report the Gated-10% row in Table 6. This is a fit-then-evaluate loop on the same quantity: the threshold is selected to maximize the very metric on which the gate was trained, which can inflate the reported +1.1% CatRecall@200 gain and the associated Relevance Score trajectory. Table 9's temporal validation is encouraging but only reports AUC and CatRecall@5 lift; it does not validate the headline CatRecall@200 and Relevance Score at the 10% operating point on held-out days. Please report held-out-day gated performance (with confidence intervals) for the same metrics used to select the operating point.","section":"§4.2, §5.1, Table 6"},{"comment":"The core decomposition result (Table 2) and the gated operating point (Table 6) are presented as point estimates without uncertainty. The cross-metric inversion — CatRecall@200 and Relevance Score ranking the same systems in opposite directions — is load-bearing for the architecture, yet no bootstrap intervals, permutation tests, or significance statements accompany these orderings. Furthermore, Relevance Score is an LLM-as-a-judge metric using Gemini 3 Flash, the same model family that powers the generative path (Gemini 2.5 Flash Lite). The paper states the outputs are anchored against real conversions, but no such anchoring evidence is shown for the offline tables. Please provide intervals for the relevant orderings and an independent judge (e.g., human-rated relevancy or a conversion-anchored proxy) for the prospecting metric.","section":"§3.1, Tables 1–2, Table 6"}],"minor_comments":[{"comment":"Equation (2) introduces Rθ(·) but this notation is not defined in the text; Algorithm 2 uses R_kw and R_smart inconsistently with Eq. (1). Please align notation.","section":"§4.2, Eq. (2)"},{"comment":"The feature names in Table 5 (e.g., 'Product set density', 'Result diversity') are not formally defined. A one-sentence definition for each would make the gate's behavior reproducible.","section":"Table 5"},{"comment":"The x-axis label 'T +1 +2 +3 +4 +5 +6 +7' is visually ambiguous; it should read 'Label window: purchases on days T..T+k' with explicit tick values.","section":"Figure 3"},{"comment":"The abstract claims 'controlled experiments on millions of users', but the offline diagnostic in §3.1 reports 20K unique users. Clarify whether 'millions' refers to the online A/B or a different population, to avoid an apparent inconsistency.","section":"Abstract, §3.1"},{"comment":"The online A/B table reports CR and CTR relative only, with no absolute values, user counts, or confidence intervals. Reporting at least the sample size and a bootstrap/credible interval for the CR lift would materially strengthen the production claim.","section":"Table 10"},{"comment":"The limitations section is honest about temporal horizon and label censoring, but should explicitly acknowledge that the online A/B does not decompose the effect by component; adding this caveat would align the conclusions with the evidence.","section":"§6.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a strong industrial case study, and the offline temporal and delayed-label checks are above typical for this venue. The main risk is that the headline 'adaptive gating drives +27.6% CR' is not supported by the reported A/B because the treatment conflates four components. This is fixable in a revision by adding an online ablation or by rewriting the central claim as a full-system effect with the gate contribution shown offline only. I also flag the offline fit-then-evaluate loop on CatRecall@200 as a correctness risk that needs a held-out-day validation of the chosen operating point."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper's real contribution is a documented empirical split: rule keywords + BM25 win on retargeting (CatRecall), LLM + ANN win on prospecting (Relevance Score), and a learned gate that routes only ~10% of users with coverage gaps to the LLM path. The offline evaluation is thoughtful: temporal holdout (day T to T+1), delayed-label robustness check, run-to-run stability, and a candidate-slot analysis. That is more rigor than most industrial papers. The gate design — auditing the cheap pass's output rather than routing on input alone — is genuinely a new twist on cost-aware LLM routing.\n\nThe soft spot the reader flagged is real: the online A/B compares full SMART against the U2I embedding baseline, swapping in four things at once (keyword/BM25 path, LambdaMART reranker, MMR, gate+LLM). Nothing isolates the gate or even the LLM path. So the +27.6% CR lift is a system-level result, not a validation of adaptive LLM routing. I'd add a second issue: the gate is trained on CatRecall@200, and the same metric is used to pick the 10% threshold and report offline lift — a mild fit-then-evaluate loop. The held-out-day result mitigates it but doesn't eliminate it. Also, the Relevance Score is an LLM judge from the same vendor, and it's not validated against human labels; they acknowledge this in the limitations. Error bars are missing on Tables 1/2/6.\n\nThat said, I don't think these are fatal. The paper is honest about the attribution limit (Section 6.2 says conversion attribution by path is future work), and the core decomposition claim is supported by the controlled offline experiments. The online ablation would be the natural fix, and it's addressable.\n\nThis paper belongs in a venue that values industrial case studies. It deserves a serious referee — the methodological caveats are manageable and the empirical base is unusually thorough. I'd take it to reading group.","headline":"Solid industrial systems paper with a genuine decomposition and gate design, but the headline online lift is a system-level effect, not proof of the gating story.","tokens_in":13783,"tokens_out":2348,"would_cite":true,"duration_ms":23591,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that keyword search and LLM search solve opposite problems—retargeting versus prospecting—and that a learned gate routing only ~10% of users to the LLM captures most of the benefit at 90% lower cost.","keywords":["Dynamic Product Ads","LLM query generation","hybrid retrieval","BM25","dense ANN","adaptive routing","quality gate","conversion rate"],"falsifier":"Run an online A/B test with at least three arms: (1) keyword/BM25 plus reranker only, (2) same plus an ungated random 10% LLM routing, and (3) full SMART with the learned gate. If arm 1 matches the 27.6% conversion lift, or if arm 2 matches arm 3, then the coverage-gap gate is not the source of the gains. A cheaper offline check: compare gate-selected 10% users against a random 10% user sample on Relevance Score; the paper's own numbers (oracle +6.9pp vs GBDT +1.1pp CatRecall@5) already suggest much of the possible lift is left on the table.","tokens_in":12713,"feed_emoji":"🎯","tokens_out":5808,"duration_ms":51028,"temperature":0.7,"pith_summary":"This paper tries to establish that in dynamic product ads, rule-generated keyword queries and LLM-generated queries are not interchangeable: keyword queries on a lexical BM25 index win at retargeting (re-surfacing known brands and categories), while LLM queries on a dense vector index win at prospecting (discovering new categories). On that basis the authors build SMART, a hybrid retrieval system that serves every user with the zero-cost keyword pass and uses a lightweight learned gate to audit the keyword results for coverage gaps, sending only about 10% of users to the LLM path. Offline, the gated system matches nearly all of the LLM's semantic-prospecting gains while cutting LLM cost by 90%. In a two-week live A/B test, SMART raised the ad conversion rate by 27.6% over a production embedding-based retriever, with click-through rate flat. The paper's central practical claim is that adaptive, output-auditing routing—not always-on LLM retrieval—is the cost-effective way to get LLM value into ad retrieval.","feed_headline":"Gated 10% LLM routing lifts ad conversion rate by 27.6%","feed_subtitle":"Cheap keywords serve everyone; a learned gate sends ~10% of users to an LLM for semantic discovery.","key_machinery":"The load-bearing mechanism is the learned quality gate, a GBDT classifier over 54 features that audits the output of the cheap keyword pass before deciding whether to invoke the LLM. Its defining signal is the coverage gap: the fraction of a user's historical purchase categories missing from the top-200 BM25 results. The second supporting mechanism is the cross-metric inversion documented in Table 2, which shows the same systems ranked in opposite order by CatRecall@200 (retargeting) and Relevance Score (prospecting); this justifies the explicit assignment of keyword queries to BM25 and LLM queries to ANN. The gate's threshold is tuned to admit ~10% of traffic, the band where CatRecall peaks","core_discovery":"The central discovery is a decomposition plus a routing scheme. Controlled experiments on millions of users show that the two query strategies rank oppositely on the two metrics: keyword+BM25 leads on CatRecall@200 by 15.9–24.1% over LLM-on-BM25, while LLM+ANN leads on Relevance Score by 22.6% over keyword+BM25. Because 73% of users see no metric change when LLM candidates are appended, the benefit is concentrated in a ~10% minority. SMART's quality gate predicts that minority from a coverage-gap feature, gap(u)=1-|cat(Rkw)∩cat(H)|/|cat(H)|, and routes only those users to the LLM path, capturing a Relevance Score of 2.36 (vs 2.39 for always-on LLM+ANN) at 10% of the cost, and lifting online","pith_inferences":["The output-auditing gate is a general pattern beyond ads: any high-throughput retrieval stack with a cheap first stage and an expensive neural second stage can condition the expensive stage on the cheap stage's actual output rather than on input difficulty alone; this is a testable extension in web search and e-commerce.","Because the online test swaps multiple components at once (keyword path, reranker, diversity layer, gate), the paper leaves open how much of the 27.6% lift is attributable to the LLM path versus the keyword/BM25 path and reranker; an ablation isolating the gate would be the decisive follow-up.","The coverage-gap features derive from a category taxonomy, so the method transfers most directly to domains with structured product metadata; in unstructured domains, one would need a learned surrogate for 'coverage' over the first-stage results.","The inverse-U cost curve suggests an operating-point methodology: sweep the routing fraction and pick the knee where marginal quality per additional LLM dollar collapses; production systems can re-tune this continuously."],"forward_implications":["Any dynamic-product-ads system with a structured catalog can adopt the template: zero-cost lexical pass for all users, output-auditing gate, gated dense/LLM pass for the ~10% with coverage deficits, and expect most of the semantic-prospecting lift at a fraction of the cost.","The retargeting/prospecting inversion gives a diagnostic for when generative query expansion helps or hurts: it helps on dense indices for open-ended intent, and hurts on lexical indices for exact-token retargeting.","A 10% routing fraction suffices to reach near-peak quality; routing more users through the LLM actually degrades category recall because noise displaces finite candidate slots (CatRecall falls from 56.6% at 10% routing to 54.5% at 100%).","The +27.6% conversion lift with flat CTR implies the hybrid surfaces high-intent discovery items rather than clickbait, so the gains should translate to revenue, not just engagement."],"fun_headline_variants":["LLM gate: 10% of users, 90% less cost, 27.6% more conversions","Adaptive routing: keywords for retarget, LLM for discovery","90% cheaper: gate sends only 10% to LLM, lifts ads 27.6%","The 10% that matters: selective LLM routing boosts conversion","Smart gate: 10% LLM calls, 90% cost cut, 27.6% conversion"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The online A/B test attributes the +27.6% conversion lift to the full SMART system, but the treatment differs from the embedding baseline in several components at once—the keyword/BM25 path, the reranker, the diversity layer, and the gated LLM path—so the paper does not isolate the marginal contribution of the gate or the LLM itself.","fun_headline_variants_meta":{"raw":{"variants":["LLM gate: 10% of users, 90% less cost, 27.6% more conversions","Adaptive routing: keywords for retarget, LLM for discovery","90% cheaper: gate sends only 10% to LLM, lifts ads 27.6%","The 10% that matters: selective LLM routing boosts conversion","Smart gate: 10% LLM calls, 90% cost cut, 27.6% conversion"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00058,"raw_usage":{"total_tokens":2595,"prompt_tokens":799,"completion_tokens":1796,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":543,"completion_tokens_details":{"reasoning_tokens":1678}},"tokens_in":543,"tokens_out":1796,"duration_ms":12541,"temperature":1.0,"reasoning_tokens":1678,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T03:32:08.280621+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an online A/B test with at least three arms: (1) keyword/BM25 plus reranker only, (2) same plus an ungated random 10% LLM routing, and (3) full SMART with the learned gate. If arm 1 matches the 27.6% conversion lift, or if arm 2 matches arm 3, then the coverage-gap gate is not the source of the gains. A cheaper offline check: compare gate-selected 10% users against a random 10% user sample on Relevance Score; the paper's own numbers (oracle +6.9pp vs GBDT +1.1pp CatRecall@5) already suggest much of the possible lift is left on the table.","supporting_citations":[],"review_version":1}