{"id":"c9e86722-fb1f-4d18-9b51-bf1b6931bbb8","arxiv_id":"2607.03880","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Inventory-aware RAG plus LoRA-tuned Llama-2 rewrites zero-fill sponsored-search queries to 68% fill rate and higher NDCG@8 than GPT-4 on 10k historical Walmart queries.","lead":"Walmart's InvAwr-RAG rewrites zero-ad search queries by blending inventory-aware LLM generation with historical successful queries, lifting offline fill rate from 0% to 68%. If it holds live, it could convert a large share of empty sponsored-search slots into revenue and better product discovery.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"Offline fill-rate gains rest on an unablated hybrid of LLM rewrites and historical popular queries, so the inventory-aware RAG contribution itself is not isolated.","rationale":"The reader correctly flags that offline metrics may not transfer to live revenue/CTR under bidding and inventory churn; that remains a real limitation and justifies CONDITIONAL. However, an equally load-bearing and more immediate concern is internal to the offline result itself: the 68% figure is produced by an unablated hybrid pipeline. Without isolating the contribution of inventory-aware rewriting from the simple injection of historically successful queries, the strongest claim cannot be attributed to the method the paper advertises. The proposed three-way ablation on the identical 10k set would settle the attribution question with existing data and would either strengthen or further qualify the offline evidence before any live A/B is run. Because the engineering direction is still plausible and no mathematical inconsistency exists, the verdict stays CONDITIONAL rather than REJECT; the confidence and required controls simply become stricter.","tokens_in":7519,"tokens_out":564,"duration_ms":5447,"concrete_test":"Re-evaluate the same 10k zero-fill set under three controlled conditions that reuse the identical retrieval index and cross-encoder threshold: (A) popular-query merge only, (B) inventory-retrieved LLM rewrites only (no popular merge), (C) full InvAwr-RAG. If fill rate under (A) already exceeds ~50% or the gap (C)–(B) is <10 points, the inventory-aware generation claim is substantially weakened.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (Table 1 / §3.2) attributes the 0%→68% fill-rate lift (and NDCG@8 0.6847) on 10k historically zero-fill queries to InvAwr-RAG, i.e., inventory-aware retrieval + LLM rewrite + merge with popular historical queries (Steps 2–6 of §2.3). The only comparator is GPT-4 (53%/0.6458), which is not described as receiving the same inventory retrieval, budget filter, or historical-query merge. Because the pipeline always merges LLM outputs with popular queries that already produce ads (Step 5–6), it is possible that a large fraction of the recovered fill comes from simply injecting historically successful queries rather than from the inventory-conditioned generation. No ablation (LLM-only, popular-only, inventory-retrieval-only, or GPT-4 + same retrieval/merge) is reported, so the load-bearing causal link between the novel inventory-aware RAG component and the headline 68% number is not secured.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.5","summary":"The paper proposes InvAwr-RAG, an inventory-aware RAG pipeline for sponsored-search query rewriting at Walmart. A rule-based classifier routes low-performing queries; a two-tower BERT retrieves top-N inventory items under budget constraints; an LoRA-fine-tuned Llama2-7B rewrites the query conditioned on those items; the rewrites are merged with historically popular queries; and a cross-encoder filters final ads. On a static set of 10 000 historically zero-fill queries the system reports a fill-rate lift from 0 % to 68 % and NDCG@8 of 0.6847 (versus GPT-4 at 53 % / 0.6458). The authors claim this hybrid approach can convert previously unfilled searches into ad impressions and project substantial future revenue impact, while deferring live A/B validation of CTR, conversion and revenue.","tokens_in":7787,"tokens_out":937,"duration_ms":7190,"significance":"If the offline gains survive live traffic and can be shown to stem from the inventory-conditioned generation rather than from simply injecting historically successful queries, the work would be a practically valuable contribution to industrial sponsored search. The problem of zero-fill queries (stated as ~13 % of traffic) is real and revenue-relevant; the hybrid RAG design is a sensible engineering response. The manuscript already ships a concrete end-to-end system description, a non-trivial offline evaluation on a hard zero-fill set, and a clear plan for A/B testing. Those strengths make the paper of interest to the e-commerce IR community even if the causal attribution of the 68 % figure remains incomplete.","major_comments":[{"comment":"Table 1 / §3.2: the headline 0 %→68 % fill-rate (and NDCG@8 0.6847) is attributed to InvAwr-RAG, yet the pipeline always merges LLM rewrites with historically popular queries that already produce ads (Steps 5–6 of §2.3). The sole comparator is GPT-4, which is not stated to receive the same inventory retrieval, budget filter or historical-query merge. Without ablations (LLM-only, popular-only, inventory-retrieval-only, or GPT-4 + identical retrieval/merge) the causal contribution of the novel inventory-aware RAG component cannot be isolated; a large fraction of the recovered fill may simply come from re-injecting known successful queries.","section":null},{"comment":"§3.1: N=20 and K=5 are justified only by informal user-experience and compute arguments. No sensitivity analysis, grid search or ablation over these free parameters is reported, so it is unclear whether the reported fill-rate and NDCG numbers are robust or are tuned to a single operating point.","section":null},{"comment":"§3.2 and §5: all quantitative claims rest on a static 10 k zero-fill log; live CTR, conversion and revenue impact under real-time bidding and inventory churn are explicitly deferred to future A/B tests. The central business claim (potential $1 B revenue) therefore remains an untested extrapolation.","section":null}],"minor_comments":[{"comment":"Abstract and Introduction contain grammatical slips (“an opportunity for to earn”, “a significant 68 % increase in fill rate” when the baseline is 0 %).","section":null},{"comment":"Figures 1–4 are referenced but their captions and visual content are only partially described; higher-resolution or more self-contained captions would help.","section":null},{"comment":"The LoRA rank, learning-rate schedule and exact prompt template used for the Llama2-7B fine-tune are not specified, limiting reproducibility.","section":null},{"comment":"Related-work coverage of recent industrial query-rewriting systems (beyond CLOVER and ProphetNet-Ads) is thin.","section":null}],"recommendation":"major_revision","confidential_remarks":"The core engineering idea is sound and the offline numbers are interesting, but the missing ablations make the novelty claim overstated for a top-tier IR venue. If the authors can supply even a modest ablation table (LLM-only vs. popular-only vs. full hybrid) the paper becomes much stronger; without it I would lean toward workshop rather than main-track acceptance."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a practical Walmart systems paper for the real problem of ~13% sponsored queries that return zero ads. They assemble two-tower BERT inventory retrieval, LoRA-tuned Llama-2 rewrite conditioned on the retrieved products and budgets, then blend the LLM outputs with historically successful popular queries. On a fixed 10k historical zero-fill set they report 68% fill rate and NDCG@8 of 0.68, beating a plain GPT-4 rewrite at 53%/0.65. That measured system outcome on their traffic is new; the individual components are not.\n\nWhat they do well is the end-to-end pipeline description (classifier, retrieval under budget, prompt construction, hybrid merge, cross-encoder filter) and the honest framing that this is about filling empty slots while keeping relevance. The hybrid idea is pragmatic: pure generation can drift, pure history can be stale. Citations to dense retrieval and generative rewrite work are appropriate and not padded.\n\nThe soft spots are real but proportionate. There is no ablation of the merge step, so we cannot tell how much of the 68% comes from simply injecting popular queries that already produce ads versus the inventory-conditioned generation. The GPT-4 baseline is not given the same retrieval or merge, so the comparison is not clean. N=20 and K=5 are chosen by informal UX/compute arguments. Everything is offline; live A/B, CTR, conversion and revenue are deferred. The $1B projection is speculative. No public data or code, as expected for this setting. None of these break the central offline signal, but they keep the causal credit for the “inventory-aware RAG” piece unsecured.\n\nThis is for industrial IR and ads practitioners who need a concrete reference for zero-fill recovery. Theory readers can skip. It is coherent, engages the literature honestly, and the offline numbers are positive enough that a serious editor should send it to referees (especially for an eCom workshop) rather than desk-reject. I would engage if the topic is live for me; otherwise a quick skim of the pipeline and Table 1 is enough.","headline":"Solid industrial systems paper that turns zero-fill ad queries into 68% filled ones offline via inventory-aware RAG + historical merge, but the evaluation never isolates the new pieces.","tokens_in":8412,"tokens_out":531,"would_cite":false,"duration_ms":12980,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Inventory-aware RAG rewriting lifts fill rate on zero-ad e-commerce queries from 0% to 68%.","keywords":["Dynamic Query Rewriting","Generative AI in Advertising","Sponsored Search","E-commerce Advertising","RAG","Inventory-Aware Retrieval","Query Rewrite LLM"],"falsifier":"A live A/B test that routes the same under-performing queries through InvAwr-RAG versus the production baseline and measures fill rate, CTR, conversion rate and revenue; a null or negative lift on those online metrics would falsify the central claim.","tokens_in":8389,"feed_emoji":"🛒","tokens_out":591,"duration_ms":4570,"temperature":0.7,"pith_summary":"About 13% of searches on a large e-commerce platform return no sponsored ads, leaving revenue and engagement on the table. This paper claims that rewriting those queries with an inventory-aware retrieval-augmented generation system, then blending the new queries with historically successful ones, can recover ads for most of them without sacrificing relevance. On a held-out set of 10,000 historically empty queries the system reaches a 68% fill rate and a higher NDCG@8 than a strong general-purpose language model. The practical stake is straightforward: every previously blank slot that now shows a relevant, budgeted ad becomes a new revenue and discovery opportunity for both the platform and its advertisers.","feed_headline":"Zero-ad queries jump to 68% fill with inventory-aware rewrites","feed_subtitle":"RAG plus live stock and past hits turns empty e-commerce searches into revenue slots","key_machinery":"Inventory-Aware RAG (InvAwr-RAG): real-time cosine retrieval of N=20 budgeted inventory items, prompt construction from those items plus the original query, LoRA fine-tuned Llama-2 generation of K=5 rewrites, and fusion with historically successful popular queries before cross-encoder ranking.","core_discovery":"The InvAwr-RAG pipeline—two-tower BERT retrieval of live inventory items, LoRA-tuned Llama-2 rewriting conditioned on those items, and merge with popular historical queries—raises fill rate from 0% to 68% on 10,000 previously empty queries while improving NDCG@8 over both the zero baseline and GPT-4 rewrites.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["InvAwr-RAG lifts zero-ad e-com queries to 68% fill","Inventory-aware Llama rewrites fill 68% empty ad queries","Two-tower RAG plus history turns 0% fill into 68%","Live stock conditioned rewrites raise fill from 0 to 68%","BERT inventory RAG improves NDCG while hitting 68% fill"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That the offline fill-rate and human-judged relevance gains measured on a static log of empty queries will still appear as live revenue, click-through and conversion lifts once real-time bidding, inventory churn and actual shoppers are in the loop.","fun_headline_variants_meta":{"raw":{"variants":["InvAwr-RAG lifts zero-ad e-com queries to 68% fill","Inventory-aware Llama rewrites fill 68% empty ad queries","Two-tower RAG plus history turns 0% fill into 68%","Live stock conditioned rewrites raise fill from 0 to 68%","BERT inventory RAG improves NDCG while hitting 68% fill"]},"model":"grok-4.5","effort":"low","cost_usd":0.004912,"raw_usage":{"total_tokens":1388,"prompt_tokens":758,"num_sources_used":0,"completion_tokens":99,"cost_in_usd_ticks":49120000,"prompt_tokens_details":{"text_tokens":758,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":531,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":758,"tokens_out":99,"duration_ms":4061,"temperature":1.0,"reasoning_tokens":531,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T23:16:34.781909+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"A live A/B test that routes the same under-performing queries through InvAwr-RAG versus the production baseline and measures fill rate, CTR, conversion rate and revenue; a null or negative lift on those online metrics would falsify the central claim.","supporting_citations":[],"review_version":1}