{"id":"fa9893af-9814-4b4c-8173-2cb46eb68b83","arxiv_id":"2506.17863","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"MarketingFM generates e-commerce ad copy with RAG and an LLM; AutoEval uses LLM-as-a-Judge plus rule checks and self-refines its prompts, with online tests showing significant clicks and impressions lifts but no significant CTR lift.","lead":"This paper describes MarketingFM, a system that uses retrieval-augmented large language models to generate keyword-specific ad copy for e-commerce search ads, plus automated evaluation tools called AutoEval. Online A/B tests showed significant gains in clicks and impressions over template ads, although the abstract's headline claim of higher CTR is not supported by the paper's own statistics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AutoEval-Main's 89.57% agreement may be an in-sample artifact: thresholds are chosen on the same 150k benchmark, and human ground-truth reliability is unreported (Table 2 lacks IAA/sample size), so the headline evaluator claim is unverified.","rationale":"The paper has two central pillars: online ad-performance gains and automated evaluator alignment with humans. The online A/B pillar has independent empirical support: Table 7 shows significant clicks and impressions lifts and a mobile CPC reduction, though the abstract mislabels the 9% clicks lift as CTR. The weaker pillar is the automated evaluator claim. For 89.57% agreement to be meaningful, two conditions are required: a reliable human reference and an unbiased evaluation protocol. Neither is established. Table 2 reports a 3.71% weighted error with no sample size or inter-annotator agreement, and Appendix B.2 explicitly mentions that the tool logs annotation metadata yet the paper omits the analysis. Section 6.3 selects the best threshold configuration on the same benchmark used to report the agreement, so the 89.57% is in-sample. These are validation and reporting gaps, not evidence that the system fails; the deployed experiments, human-evaluation scale, and reduction of the relevance-error rate from 10% to 1% are real supporting evidence. The concern is fixable by a held-out evaluation and an IAA report, and it is sufficiently central that neither ACCEPT nor REJECT is warranted. The reader's CONDITIONAL verdict therefore stands, with the condition being explicit validation of the evaluator's headline agreement rate.","tokens_in":17472,"tokens_out":9085,"duration_ms":95122,"concrete_test":"Use the 'contamination-free' verified golden dataset referenced in Section 6.4, or split the 150,000 human-labeled ad copies into a threshold-selection set and a held-out test set. Pre-specify the Context(2)+General(3) rule from the selection set only, apply it to the held-out test set, and report total agreement, FPR/FNR, and a bootstrapped 95% CI, together with Cohen's kappa and the sample size behind Table 2. If held-out agreement is materially below 89.57%, or if the optimal threshold on the selection set differs from (2,3), the headline agreement rate is an in-sample artifact.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central evaluator claim rests on AutoEval-Main's 89.57% agreement with human reviewers, but Section 6.3 reports testing six threshold configurations on the same 150,000 human-labeled ad copies and selecting the one with the lowest total disagreement (Table 8: Context(2)+General(3), 10.43% total). No validation split or pre-registered threshold is described for AutoEval-Main, so the headline number is the best of six fits, not an unbiased estimate. The ground truth itself is only weakly validated: Section 4/Table 2 gives a weighted error of 3.71% but no sample size or inter-annotator agreement, and Appendix B.2 states the tool 'logs annotation metadata, enabling further analysis of inter-annotator agreement' without reporting it. Error is concentrated on reject labels (29% mistaken), which disproportionately affects the FPR/FNR quantities in Table 8 and any agreement rate calibrated to those labels. If human labels are noisy and thresholds are tuned to them, the 89.57% may overstate AutoEval-Main's true alignment with reliable human judgment. The abstract's 9% CTR claim is also inconsistent with Table 7 (the 9% is a clicks lift; CTR p-values are 0.23 mobile and 0.11 desktop), but the evaluation-side gap is more load-bearing because it undermines the benchmark for every downstream AutoEval-Update claim as well.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes MarketingFM, a retrieval-augmented generation system for producing keyword-specific ad copy for e-commerce paid search, and two automated evaluation components: AutoEval-Main, a hybrid rule-based/LLM-as-a-Judge evaluator, and AutoEval-Update, an iterative prompt-refinement framework that uses active sampling and a critic LLM. The authors report offline human and automated evaluations on 150,000 generated ad copies for 10,000 keywords, and online A/B tests on 3,000 and 10,000 keywords. The headline claims are that keyword-focused ad copy outperforms template-based copy (stated as 'up to 9% higher CTR' in the abstract) and that AutoEval-Main achieves 89.57% agreement with human reviewers, with AutoEval-Update further improving alignment through automated prompt refinement.","tokens_in":17734,"tokens_out":6151,"duration_ms":52354,"significance":"If the results held, the work would be practically significant: it addresses a real bottleneck in scaling personalized ad content and reducing human review costs. The paper reports a large human-annotated dataset, a deployed system, and online A/B testing infrastructure, all of which are relatively rare in academic work and could be useful to practitioners. However, the current manuscript contains several internal inconsistencies and missing validation details that prevent accepting the central claims as stated.","major_comments":[{"comment":"The abstract's claim of 'up to 9% higher CTR' is not supported by the data in Table 7. Table 7 reports CTR lift of +4 basis points on mobile (p=0.234) and +24 basis points on desktop (p=0.110), neither of which is statistically significant. The 8–9% figure in Table 7 is the clicks lift, not the CTR lift, and Section 6.2 explicitly states that 'CTR remains stable due to broader reach.' The abstract should be corrected to report the clicks lift and impression lift, or to present the CTR effect with its confidence intervals and p-values.","section":"Abstract / §6.2, Table 7"},{"comment":"The text claims that 'All AutoEval-Update strategies improved upon the baseline' and that 'the baseline method performs the worst overall, with the highest FPR and FNR rates.' These statements are contradicted by Table 9: the diversity-based sampling strategy has accuracy 66.89% and F_beta 70.16%, both worse than the baseline (68.39% and 72.09%), and the baseline FNR (10.92%) is lower than that of random (15.52%) and diversity (22.41%) sampling. The conclusions in this section need to be revised to state that only random and uncertainty-based sampling improve over the baseline, and that no single strategy dominates on all metrics.","section":"§6.4, Table 9"},{"comment":"The reported 89.57% agreement between AutoEval-Main and human reviewers appears to be an in-sample artifact. Table 8 sweeps six threshold configurations over the same 150,000 human-labeled ad copies and the authors select the configuration with the lowest total disagreement (Context(2)+General(3), 10.43%). No validation split is described for AutoEval-Main, and no pre-registered thresholds are specified. The paper should report agreement on a held-out set using thresholds fixed before evaluation, or otherwise justify the threshold selection without using the test set.","section":"§6.3, Table 8"},{"comment":"The reliability of the human annotation ground truth is not established. Table 2 reports a weighted error rate of 3.71% but does not state the sample size used for manual validation or any inter-annotator agreement measure. Importantly, the reject labels have a 29% mistake rate, which directly affects the false-positive/false-negative quantities in Table 8 that AutoEval-Main is calibrated against. Appendix B.2 notes that the annotation tool logs metadata 'enabling further analysis of inter-annotator agreement,' but no such analysis is reported anywhere in the paper. The authors should report inter-annotator agreement and the number of validated samples.","section":"§4, Table 2"}],"minor_comments":[{"comment":"The reference to 'Figure 5' for the agreement result should be Table 5; Figure 5 is the AutoEval-Update pipeline diagram in Appendix C.","section":"§6.3"},{"comment":"The abstract reports '0.38% lower CPC', but Table 7 shows -0.35% (mobile) and -0.22% (desktop). Please clarify which number is being cited.","section":"Abstract"},{"comment":"The statement that AutoEval-Main 'reduces costs by 200× and processing time by 42×' lacks an explanation of how these factors were measured; please add a brief methodology or a reference.","section":"§6.3"},{"comment":"There is a typo in the sentence 'ensuring that evaluation remains adaptive and aligned with evolving data with little human involve' – 'involve' should be 'involvement'.","section":"§5.2"},{"comment":"The F_beta metric is not defined; please specify the beta value used.","section":"Table 9"}],"recommendation":"major_revision","confidential_remarks":"The paper is a work-in-progress arXiv preprint with internal inconsistencies that affect the validity of its headline claims. The issues are fixable: correcting the abstract, reanalyzing Table 9, adding a held-out validation split for AutoEval-Main, and reporting human annotation reliability would substantially strengthen the manuscript. I recommend major revision rather than rejection because the deployed-system context and large-scale evaluations are valuable, but the current presentation does not support the claims as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuine industry systems paper with a deployed pipeline and some real online wins, but the headline numbers are better than the evidence. The 9% CTR in the abstract is actually a clicks lift; CTR is not significant in Table 7. The 89.57% AutoEval-Main agreement is the best of six threshold configurations fit to the same 150k benchmark, with no validation split. And Table 9 contradicts the paper's claim that all AutoEval-Update sampling strategies beat baseline — diversity sampling has lower accuracy and F_beta than the baseline.\n\nWhat's actually good: MarketingFM is a sensible RAG pipeline for ad copy, and the engineering details are useful: task chaining for character limits, and replacing semantic retrieval with search-page product context reduced human-rated irrelevance from 10% to 1% in their 3,000-keyword test. The online A/B on 10,000 keywords shows significant lifts in clicks (+8%/+9%) and impressions (+8%/+12%) on mobile/desktop, which is real evidence the keyword-specific copy helped ranking and visibility. The context-aware LLM judge clearly reduces wasted rejections compared to query-only scoring (FNR 35.4% to 18.6%), and the AutoEval-Update idea — active sampling plus a critic LLM to propose prompt changes — is a reasonable approach to criteria drift, with a concrete case study.\n\nSoft spots in order of severity. First, the abstract misstates results: CTR lifts are not significant (p=0.23 mobile, p=0.11 desktop), and the 0.38% CPC figure doesn't match Table 7 (0.35% mobile, 0.22% desktop ns). That needs fixing. Second, the 89.57% agreement is in-sample: Table 8 sweeps six threshold combinations on the same 150k human-labeled ad copies and picks the best. Without a held-out split or pre-registered thresholds, that number is an upper-bound fit, not a performance estimate. Third, the ground truth itself is shaky: Table 2 reports a 3.71% weighted error but no sample size or inter-annotator agreement, and the rejection label is wrong 29% of the time. Since FPR/FNR are calibrated against those labels, the small error rates in Table 8 are suspect. Fourth, the AutoEval-Update results are simulations on historical human annotations, not live evaluation, and the diversity strategy in Table 9 underperforms baseline on accuracy.\n\nNone of this means the system doesn't work — the online engagement lifts are credible and directionally consistent. But the claims as written overstate the evidence. The paper deserves a serious referee and likely publication after major revision: correct the abstract, add a validation split for AutoEval-Main, report IAA, and reconcile the diversity sampling claim. It's aimed at practitioners building LLM eval for ads or e-commerce; academic readers will find the threshold-selection lesson useful.","headline":"Real deployed system with significant online engagement lifts, but the headline CTR and auto-evaluator agreement numbers are overstated and need correction.","tokens_in":18353,"tokens_out":3444,"would_cite":false,"duration_ms":31985,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Keyword-specific ad copy generated by a retrieval-augmented LLM outperforms template-based ads in large-scale search advertising tests, and the paper's automated judge agrees with human reviewers 89.57% of the time.","keywords":["large language models","retrieval-augmented generation","ad copy generation","LLM-as-a-Judge","paid search advertising","e-commerce marketing","automated evaluation","prompt refinement"],"falsifier":"Look at the raw click and impression counts behind the 10,000-keyword A/B test: if the CTR lifts remain non-significant, at about +4 to +24 basis points with p-values above 0.1, while click and impression lifts are significant, then the abstract's '9% higher CTR' would not be supported even though the engagement and cost-efficiency gains would stand.","tokens_in":17191,"feed_emoji":"📈","tokens_out":8443,"duration_ms":72314,"temperature":0.7,"pith_summary":"The paper argues that the main obstacle to effective offsite e-commerce advertising is generic template ad copy that ignores what the customer actually searched. It proposes MarketingFM, a retrieval-augmented generation system that grounds each ad in product metadata retrieved for the search keyword, and reports that in a 10,000-keyword online A/B test keyword-specific copy raised click volume and impressions and lowered cost-per-click relative to template ads. Because human review of millions of generated ads is expensive, the paper also proposes AutoEval-Main, a two-layer evaluator combining rule-based checks with LLM-as-a-Judge scoring, which it reports matches human accept/reject decisions in 89.57% of cases. A third component, AutoEval-Update, uses active sampling and a critic LLM to revise evaluation prompts as standards drift, reducing the human labor needed to keep the evaluator aligned. If these results hold, e-commerce marketers could personalize ad copy per keyword at scale while relying on cheap automated screening rather than large annotation teams.","feed_headline":"Keyword-focused LLM ads beat templates in paid search","feed_subtitle":"Grounding ads in product retrieval lifts clicks and impressions; a hybrid LLM judge matches humans 89.57% of the time.","key_machinery":"The load-bearing mechanism is retrieval-augmented generation grounded in the retailer's own search product index. A retriever maps each customer keyword to product documents, first via sentence-embedding similarity and later via the products that actually appear on the retailer's search page for that keyword, and the LLM generates headlines and descriptions from that retrieved context. A task-chaining step generates a surplus of candidates and then summarizes them into character-limited headlines. Evaluation is carried out by AutoEval-Main, which first applies rule-based rejections for safety, sale claims, and diversity, then an LLM-as-a-Judge that scores relevance and generalization on a 0-5 scale using the same retrieved context; AutoEval-Update closes the loop with active sampling of ad copies, a critic LLM that writes alignment reports from human-versus-LLM disagreements, and iterated prompt and threshold refinement.","core_discovery":"The central discovery is that grounding LLM ad copy in retrieval from the retailer's product search index converts a generic category template into a keyword-specific, product-focused ad, and that this conversion measurably improves paid search economics. In the paper's largest field experiment across 10,000 keywords, the treatment ads gained 8-12% more impressions and 8-9% more clicks on mobile and desktop, with modest CPC reductions; the abstract summarizes these gains as up to 9% higher CTR, 12% more impressions, and 0.38% lower CPC. For evaluation, the paper finds that an LLM judge that sees both the query and the retrieved product context and also scores how well the ad generalizes agrees with human reviewers 89.57% of the time, with most disagreements being the LLM rejecting ads humans accept. The paper further claims that a critic LLM can diagnose those disagreements and propose revised evaluation criteria, which the experiments indicate improve alignment beyond the unrefined prompt. Together these results are offered as evidence that LLM-based generation and evaluation can operate at industry scale with only minimal human oversight.","pith_inferences":["The disagreement analysis shows the LLM judge is more conservative than humans, rejecting ads humans accept in 8.90% of cases while accepting in only 1.53% of the reverse; a natural extension is threshold calibration that trades a small increase in accepted-risk for a large drop in wasted rejections, a tuning decision the paper leaves open.","The 3.71% weighted human annotation error rate implies the 89.57% agreement is close to the practical ceiling for human-level labels, so future gains may require adjudicated or consensus labels rather than better evaluation prompts.","The same retrieval-plus-evaluation loop could be tested on the other marketing channels the paper names, such as social ads and outbound email, where product context and length constraints differ; whether the gains transfer is a testable prediction.","One could expect diminishing returns from prompt refinement once evaluator disagreements are dominated by irreducible label noise, suggesting the stopping rule in AutoEval-Update could be tied to measured annotator agreement rather than a fixed threshold."],"forward_implications":["Search-marketing teams could generate per-keyword ad copy for millions of products without hand-writing templates, because grounding in retrieved product context keeps content relevant.","The 89.57% agreement between AutoEval-Main and human reviewers implies that a hybrid rule-plus-LLM judge can replace most human screening; the paper reports a 200x cost reduction and 42x faster processing.","Because AutoEval-Main is calibrated against a large human-annotated benchmark, that same benchmark could be reused to fine-tune generation models, as the paper notes.","AutoEval-Update's active-sampling loop implies evaluation criteria can track shifting product catalogs and ad policies using only a few thousand human labels per refresh cycle rather than full re-annotation.","The lower rejection rate from search-page product context, roughly 2.79% overall versus 15% with semantic retrieval, suggests retrieval design, not model choice, is the main lever for ad relevance."],"supporting_citations":[{"why":"Supplies the retrieval-augmented generation paradigm that grounds ad generation in retrieved product documents.","marker":"[21]"},{"why":"Supplies the sentence-embedding model used for semantic keyword-to-product retrieval.","marker":"[13]"},{"why":"Provides the vector-search library used to retrieve top-k product documents at scale.","marker":"[12]"},{"why":"Supplies the commercial LLM used as the ad copy generator.","marker":"[2]"},{"why":"Establishes the LLM-as-a-Judge methodology that AutoEval-Main adapts to marketing content.","marker":"[42]"},{"why":"Documents the criteria-drift problem that motivates the iterative prompt-refinement loop.","marker":"[32]"},{"why":"Describes the experimentation infrastructure used to run randomized keyword-level A/B splits.","marker":"[18]"},{"why":"Surveys prior LLM use in sponsored search that the generation framework builds on.","marker":"[29]"}],"fun_headline_variants":["Retrieval-based LLM ads beat generic templates in paid search","LLM ad copy tuned to keywords lifts CTR up to 9%","AutoEval-Main: LLM judge hits 89.57% human agreement","Critic LLM refines ad evaluation, cuts manual review","MarketingFM: ground ad generation in product search for gains"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The human accept/reject labels used as ground truth are accurate enough to serve as the benchmark for all the agreement numbers; the paper's own audit found a 3.71% weighted error rate, and if annotator noise is actually higher, the claimed 89.57% agreement and the refinement improvements are measured against an unstable target.","fun_headline_variants_meta":{"raw":{"variants":["Retrieval-based LLM ads beat generic templates in paid search","LLM ad copy tuned to keywords lifts CTR up to 9%","AutoEval-Main: LLM judge hits 89.57% human agreement","Critic LLM refines ad evaluation, cuts manual review","MarketingFM: ground ad generation in product search for gains"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000213,"raw_usage":{"total_tokens":1480,"prompt_tokens":1059,"completion_tokens":421,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":328}},"tokens_in":675,"tokens_out":421,"duration_ms":4696,"temperature":1.0,"reasoning_tokens":328,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:58:50.196287+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look at the raw click and impression counts behind the 10,000-keyword A/B test: if the CTR lifts remain non-significant, at about +4 to +24 basis points with p-values above 0.1, while click and impression lifts are significant, then the abstract's '9% higher CTR' would not be supported even though the engagement and cost-efficiency gains would stand.","supporting_citations":[{"cited_title":"Retrieval-augmented generation for knowledge- intensive nlp tasks","cited_arxiv_id":null,"evidence_quote":"Supplies the retrieval-augmented generation paradigm that grounds ad generation in retrieved product documents."},{"cited_title":"all-minilm-l6-v2, 2020","cited_arxiv_id":null,"evidence_quote":"Supplies the sentence-embedding model used for semantic keyword-to-product retrieval."},{"cited_title":"The claude 3 model family: Opus, sonnet, haiku.Claude-3 Model Card, 1, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the commercial LLM used as the ad copy generator."},{"cited_title":"Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences","cited_arxiv_id":null,"evidence_quote":"Documents the criteria-drift problem that motivates the iterative prompt-refinement loop."},{"cited_title":"Serp interference network and its applications in search advertising","cited_arxiv_id":null,"evidence_quote":"Describes the experimentation infrastructure used to run randomized keyword-level A/B splits."},{"cited_title":"Applying large language models to sponsored search advertising.URL: https://www","cited_arxiv_id":null,"evidence_quote":"Surveys prior LLM use in sponsored search that the generation framework builds on."}],"review_version":2}