Pith. sign in

REVIEW 5 major objections 7 minor 28 references

OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent

T0 review · 5 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that OMS, an LLM agent that clusters existing ad keywords, ranks them with a multi-metric TOPSIS score, and reflects on its own new candidates, beats prior LLM keyword generators and platform keyword tools on every metric…

desk verdict Solid applied agentic keyword-generation paper with a real online test and honest limitations; the main soft spot is an underspecified offline evaluation mapping, but the work deserves a serious referee. read the letter →

arxiv 2507.02353 v1 pith:22T6KOMZ submitted 2025-07-03 cs.AI

Bowen Chen , Zhao Wang , Shingo Takamatsu This is my paper
classification cs.AI
keywords sponsoredsearchadvertisingkeywordgenerationLLMagentmulti-objectiveoptimizationTOPSISrankingself-reflectiononlineA/Btestadquality
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

In sponsored search advertising, ads appear when a user's query matches an advertiser's keyword list, so choosing keywords determines whether ads show and whether they convert. This paper tries to show that a single LLM-driven agent can manage that choice end-to-end without any training data: it watches how current keywords perform, clusters them by user intent, scores them against several metrics at once, generates fresh candidates, critiques its own output, and repeats. If the claim holds, advertisers no longer need platform-internal query-keyword logs to automate keyword decisions, and they can optimize for conversions and cost simultaneously rather than for clicks alone. The evidence is an offline benchmark where OMS beats all baselines on every reported metric, an ablation showing each module contributes, and a 30-day A/B test in which OMS achieved more conversions at lower cost per acquisition than the OKG agent under the same budget.

What carries the argument

The load-bearing mechanism is a two-module loop rather than a single transformer. The first module, agentic clustering-ranking, converts the raw performance vectors into a unified TOPSIS score per keyword, where TOPSIS means the ratio of the distance from the anti-ideal point (all metrics at their worst) to the sum of the distances to the ideal and anti-ideal points, so that the LLM reasons about one scalar quality measure instead of juggling raw columns; entropy weighting lets metrics with more variance count more. The second module, multi-turn generation-reflection, couples generation to external tool returns, such as search-volume checks, rejection filters, lexical analysis, and category analysis, and adds an LLM reflection pass that grades each new keyword for relevance and alignment with real user search behavior, then regenerates until the set passes. Together they give the agent memory through clusters, a multi-objective objective through TOPSIS, and a critic through reflection, which the ablations show are all needed.

What would settle it

Count, in the paper's offline benchmark, how many OMS-generated keywords appear verbatim in the 800-keyword evaluation set; if most do not, their listed Clicks, CpC, and Search Volume values must be imputed, and restricting the comparison to exact matches would show whether the all-metrics advantage survives. Also, running the A/B test against a second baseline, which the paper says budget prevented, would check that the cost-per-conversion gain is not specific to the comparison with OKG.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that decomposing keyword generation into a cluster-rank-reflect loop lets an LLM agent turn live performance data into better keywords without any training. Given the current keyword set and their performance vectors, OMS has the LLM summarize each cluster's search intent, computes a TOPSIS score for every keyword from normalized metrics such as clicks, conversions, and cost, ranks keywords within and across clusters, constructs many-shot prompts that give either full keyword lists for good clusters or best and worst contrastive examples for bad clusters, generates candidates under tool-guided constraints, and runs an LLM reflection pass that scores each candidate on a 1-5 scale and says keep or replace, looping until the set is complete. The paper reports that this pipeline wins on all metrics in the OKG offline benchmark, outscored the strongest prior agent in human preference ratings, and, in a live Google Ads A/B test, produced 33 conversions versus 29 for OKG with a 12.1 percent lower CPA and a 12.0 percent higher conversion rate.

Load-bearing premise

The offline comparison only works if every generated keyword can be assigned realistic click, cost, and search-volume values from a fixed set of 800 known keywords, but the paper does not say how a new keyword is matched to those values.

Editorial extensions

If this is right

  • An advertiser can run a keyword campaign with no offline training data, since OMS learns from the live performance of its own deployed keywords.
  • Multiple campaign goals can be optimized at once; the TOPSIS score means improving clicks no longer has to come at the expense of cost or conversion rate.
  • Because keywords are re-clustered and reflected on each round, the system should resist semantic drift as campaigns run for many weeks.
  • The testable business effect is lower cost per conversion: under the same budget, the A/B test reports a 12.1 percent lower CPA and a 12.0 percent higher conversion rate.
  • The same agentic loop generalizes to different products and to platforms that expose performance APIs, since the method assumes only keyword lists, their metrics, and a product description.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The offline benchmark's way of assigning performance values to newly generated keywords is not described; a natural extension is to require the authors to report the exact-match rate of generated keywords against the 800-keyword evaluation set and to rerun the comparison restricted to exact matches.
  • Because the reflection pass uses a stronger reasoning LLM than the generator, part of the benefit may come from the critic's reasoning capacity rather than the architecture alone; swapping in a smaller critic would isolate that contribution.
  • TOPSIS weights are either hand-set or entropy-derived from the current keyword pool; an online weighting scheme that adapts weights to campaign goals, such as favoring conversions when conversion rate is the bottleneck, could sharpen the multi-objective claim.
  • The cluster-rank-reflect recipe may transfer to other search-advertising decisions, such as ad-copy selection or bid adjustment, where the same pattern of monitoring, multi-metric scoring, and self-correction applies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 7 minor

Summary. The paper proposes OMS, an LLM-agent framework for sponsored-search advertising keyword generation. OMS combines an agentic clustering-ranking module that computes multi-objective TOPSIS scores over active keywords and an iterative generation-reflection module that uses search, filtering, and reflection tools to generate and refine keywords. The authors claim that OMS outperforms existing LLM-based and tool-based baselines on an offline benchmark, in a human evaluation, and in a 30-day online A/B test on Google Ads, and they support this with ablations and a case study. The manuscript includes detailed prompt templates and workflow descriptions in the appendices.

Significance. If the empirical claims hold, OMS is a practically relevant contribution to automated ad-keyword generation: it addresses real-time multi-objective monitoring and self-reflective quality control, which are genuinely missing from prior agentic keyword-generation work. The inclusion of an online A/B test, a human evaluation with reported inter-annotator agreement, and detailed prompt/algorithm appendices are strengths. However, the significance is currently bounded by evaluation-protocol ambiguities and limited statistical support: the offline benchmark's matching rule for scoring generated keywords is unspecified, the online test is a single campaign with small differences, and the ablations cover one product. These issues need to be resolved before the central claim of consistent superiority across all metrics can be accepted.

major comments (5)
  1. [Section 5.1 and Appendix A.1] The offline benchmark evaluation does not specify how a generated keyword is matched to the 800-keyword held-out performance set. Appendix A.1 states that 800 keywords are randomly sampled as evaluation data and the remaining 200 are used for OKG Many-Shot/OMS, but it never states whether matching is exact string match, fuzzy match, embedding similarity, or aggregation over a cluster. OMS is explicitly designed to generate novel keywords (Section 4.2, Algorithm 1), so many generated keywords will not appear verbatim in the held-out set. Without a stated rule for handling unmatched keywords, the Click, CpC, Search Volume, and Comp Score values in Table 3 are undefined or must be imputed in some unspecified way, which directly affects the central claim that OMS 'outperforms baseline methods in all metrics'.
  2. [Table 3 and Appendix A.1] The averaged normalized results in Table 3 are reported without error bars, standard deviations, or significance tests, even though Appendix A.1 states that experiments were run five times with different random seeds. Several metric differences are small (e.g., Clicks 0.92 vs. 0.89, CpC 0.84 vs. 0.88, Comp Score 0.89 vs. 0.83), so without variance information or a statistical test the claim that OMS outperforms all baselines on all metrics is not fully supported. Per-seed results or confidence intervals should be reported.
  3. [Section 5.5, Table 7] The online A/B test is based on a single 30-day campaign for one product with a daily budget of 4,000 yen. The absolute differences in Clicks (724 vs. 713), CTR (6.25% vs. 6.24%), and CPC (¥128 vs. ¥130) are very small, and the total Cost is essentially identical (difference -0.01%). The headline 12.1% CPA gain and 12.0% CVR gain therefore rest on a difference of four conversions (33 vs. 29). No significance testing, confidence intervals, or bootstrap analysis is provided, and the campaign was not replicated. This is too weak a basis for the statement that OMS 'achieves lower CPA and CpC' as a general real-world conclusion.
  4. [Section 5.3, Table 4] The ablation study is performed on a single product (Alpha Camera) and reports no error bars or significance tests. The text claims that removing intent analysis, TOPSIS, reflection, or LLM-assisted re-clustering 'reduces performance' and that each component is effective, but with one product and no variance information these claims are not established. At minimum, the authors should report ablations across several products or explicitly label the results as illustrative rather than general.
  5. [Appendix A.1 and Section 5.5] The evaluation setup has a potential self-reference concern. Two of the three OMS authors also authored OKG, the offline benchmark is OKG's released dataset, and Appendix A.1 states that for non-agentic baselines the product information is obtained by first running OMS and saving its retrieved information. This means the baseline methods receive product information that was produced by the system under evaluation. The authors should clarify whether the baselines were given the same independently sourced product descriptions, or show that results are robust when baselines use their own retrieval.
minor comments (7)
  1. [Section 5.1] The sentence 'with normalized numerical values in Table 1' should refer to Table 3, not Table 1.
  2. [Figure 2] The label 'ALYSIS' in the figure appears to be a truncation of 'ANALYSIS'; this should be corrected.
  3. [Appendix C.1] The phrase 'inner-annotator agreement' should be 'inter-annotator agreement'.
  4. [Equation (5)] The entropy-based weighting formula has an undefined expression when any normalized value s_id is zero, since it computes s_id * ln(s_id). The authors should state how zero values are handled.
  5. [Algorithm 1] The thresholds τ (search volume) and θ (category rejection) are not given numerical values or a procedure for setting them; please provide the values used in the experiments or a sensitivity analysis.
  6. [Table 6] The human-preference evaluation reports averaged scores across five products but does not state the number of keywords evaluated per system or per product; adding this information would help assess the reliability of the differences.
  7. [References] The reference 'Shinn Yao et al. 2023' is likely intended to be 'Shunyu Yao et al.'; the citation should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: OMS's central comparison is empirically grounded and no prediction reduces to its inputs by construction.

full rationale

The paper's load-bearing claim is the empirical superiority of OMS in Section 5 (Tables 3, 6, and 7). The core comparison is not a derivation from definitions: TOPSIS scores (Eq. 4) and entropy weights (Eq. 5) are computed from observed keyword performance and used to generate new keywords, while the offline benchmark evaluates on a held-out set (Appendix A.1: 'We randomly sample the 800 keywords as the evaluation data, and the left 200 keywords with their performance are used for the OKG Many-shot or the OMS') and the online A/B test runs on Google Ads against OKG with equal budget. The OKG benchmark is authored by overlapping researchers, and the Search Volume Validator does filter generated keywords by a search-volume API that resembles the evaluated Search Volume metric; these are evaluation-design and fairness concerns, not circular reductions, because the reported metric is a held-out magnitude rather than the validator's binary pass/fail, and because the online A/B test independently tests cost and conversion outcomes. The paper's own Limitation section concedes the absence of A/B/n testing and of trainable query-keyword baselines; this weakens generalizability but does not make any claimed result equivalent to its input. No fitted parameter is renamed as a prediction, and no uniqueness theorem or load-bearing self-citation is invoked. Therefore the paper is not circular; the under-specified keyword-to-performance matching in the offline benchmark is a correctness risk rather than a circularity.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework relies on standard MCDM and clustering mathematics, but introduces several unspecified thresholds and trusts uncalibrated LLM judgments. No new entities are postulated.

free parameters (3)
  • cluster quality threshold lambda
    Used in Eq. 6 to decide whether a cluster gets full keyword context or only best/worst examples; value not reported.
  • search volume threshold tau
    Used in Algorithm 1 to reject keywords below a minimum search count; value not reported.
  • category rejection threshold theta
    Used in Algorithm 1 to drop clusters with a high rejection rate; value not reported.
assumptions (4)
  • standard math TOPSIS provides a valid ranking of keywords under multiple performance metrics
    Standard multi-criteria decision method cited from Hwang and Yoon (1981) and Zavadskas et al. (2006).
  • standard math Affinity Propagation with BERT embeddings yields semantically coherent keyword clusters
    Clustering from Frey and Dueck (2007) with bert-base-cased; no quantitative cluster-quality check.
  • ad hoc to paper LLM-generated intent summaries, reflection scores, and cluster assignments are accurate and beneficial
    The pipeline trusts GPT-4o and o3 outputs without calibration or verification (Sections 4.1.1 and 4.2).
  • domain assumption The OKG benchmark dataset provides reliable real-world keyword performance labels
    All offline metrics derive from this dataset; its collection methodology is not described here.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent." pith.science (2026). https://pith.science/paper/22T6KOMZ

@misc{pith2026250702353,
  author       = {Pith},
  title        = {Pith review of: OMS: On-the-fly, Multi-Objective, Self-Reflective Ad Keyword Generation via LLM Agent},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/22T6KOMZ}},
  note         = {Machine review of arXiv:2507.02353}
}
read the original abstract

Keyword decision in Sponsored Search Advertising is critical to the success of ad campaigns. While LLM-based methods offer automated keyword generation, they face three major limitations: reliance on large-scale query-keyword pair data, lack of online multi-objective performance monitoring and optimization, and weak quality control in keyword selection. These issues hinder the agentic use of LLMs in fully automating keyword decisions by monitoring and reasoning over key performance indicators such as impressions, clicks, conversions, and CTA effectiveness. To overcome these challenges, we propose OMS, a keyword generation framework that is On-the-fly (requires no training data, monitors online performance, and adapts accordingly), Multi-objective (employs agentic reasoning to optimize keywords based on multiple performance metrics), and Self-reflective (agentically evaluates keyword quality). Experiments on benchmarks and real-world ad campaigns show that OMS outperforms existing methods; ablation and human evaluations confirm the effectiveness of each component and the quality of generated keywords.

Figures

Figures reproduced from arXiv: 2507.02353 by the authors.

Figure 1
Figure 1. An illustrative example: among the three adver [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the OMS workflow. The process consists of two main modules: [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Performance Results on Benchmark Dataset. CpC is presented reversely, meaning a higher value means [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Performance change with Number of Shots and Generation Rounds (Under 80 examples). [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: An example workflow for keyword generation. [PITH_FULL_IMAGE:figures/full_fig_p014_5.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 12 canonical work pages

  1. [1]

    Xingye Chen, Wei Feng, Zhenbang Du, Weizhen Wang, Yanyin Chen, Haohan Wang, Linkai Liu, Yaoyu Li, Jinyuan Zhao, Yu Li, et al. 2025. Ctr-driven advertising image generation with multimodal large language models. In Proceedings of the ACM on Web Conference 2025, pages 2262--2275

    work page 2025

  2. [2]

    Yifan Chen, Gui-Rong Xue, and Yong Yu. 2008. https://doi.org/10.1145/1341531.1341564 Advertising keyword suggestion based on concept hierarchy . In Proceedings of the 2008 International Conference on Web Search and Data Mining, WSDM '08, page 251–260, New York, NY, USA. Association for Computing Machinery

    arXiv 2008

  3. [3]

    Fain and Jan Pedersen

    Daniel C. Fain and Jan Pedersen. 2006. https://api.semanticscholar.org/CorpusID:16068469 Sponsored search: A brief history . Bulletin of The American Society for Information Science and Technology, 32:12--13

    work page 2006

  4. [4]

    Sorouralsadat Fatemi and Yuheng Hu. 2024. https://doi.org/10.1145/3677052.3698686 Enhancing financial question answering with a multi-agent reflection framework . In Proceedings of the 5th ACM International Conference on AI in Finance, ICAIF ’24, page 530–537. ACM

    arXiv 2024

  5. [5]

    Frey and Delbert Dueck

    Brendan J. Frey and Delbert Dueck. 2007. https://doi.org/10.1126/science.1136800 Clustering by passing messages between data points . Science, 315(5814):972--976

    doi:10.1126/science.1136800 2007

  6. [6]

    Google. 2024. Google keyword planner. https://ads.google.com/home/tools/keyword-planner/. Google Ads

    work page 2024

  7. [7]

    Google . 2025. Keyword Planner - Google Ads . https://ads.google.com/home/tools/keyword-planner/. Accessed: 2025-05-18

    work page 2025

  8. [8]

    Ching-Lai Hwang and Kwangsun Yoon. 1981. Multiple Attribute Decision Making: Methods and Applications. Springer-Verlag, Berlin, Heidelberg

    work page 1981

Show all 28 references
  1. [9]

    Amruta Joshi and Rajeev Motwani. 2006. https://doi.org/10.1109/ICDMW.2006.104 Keyword generation for search engine advertising . In Sixth IEEE International Conference on Data Mining - Workshops (ICDMW'06), pages 490--496

    2006 doi

  2. [10]

    Mu-Chu Lee, Bin Gao, and Ruofei Zhang. 2018. https://doi.org/10.1145/3219819.3219850 Rare query expansion through generative adversarial networks in search advertising . In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '1...

    2018

  3. [11]

    Yijiang Lian, Zhijie Chen, Jinlong Hu, Kefeng Zhang, Chunwei Yan, Muchenxuan Tong, Wenying Han, Hanju Guan, Ying Li, Ying Cao, Yang Yu, Zhigang Li, Xiaochun Liu, and Yue Wang. 2019. https://arxiv.org/abs/1902.00592 An end-to-end generative retrieval method for sponsored search...

    2019 arXiv

  4. [12]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  5. [13]

    Microsoft . 2025. Keyword Planner - Microsoft Advertising . https://about.ads.microsoft.com/ja/tools/planning/keyword-planner. Accessed: 2025-05-18

    2025

  6. [14]

    Masato Mita, Soichiro Murakami, Akihiko Kato, and Peinan Zhang. 2024. https://doi.org/10.18653/v1/2024.acl-long.54 Striking gold in advertising: Standardization and exploration of ad text generation . In Proceedings of the 62nd Annual Meeting of the Association for Computation...

    2024 doi

  7. [15]

    Han Nie, Yanwu Yang, and Daniel Zeng. 2019. https://doi.org/10.1109/MIS.2019.2938881 Keyword generation for sponsored search advertising: Balancing coverage and relevance . IEEE Intelligent Systems, 34(5):14--24

    2019

  8. [16]

    Shakked Noy and Whitney Zhang. 2023. https://doi.org/10.1126/science.adh2586 Experimental evidence on the productivity effects of generative artificial intelligence . Science (New York, N.Y.), 381:187--192

    2023 doi

  9. [17]

    OpenAI. 2024. https://arxiv.org/abs/2410.21276 Gpt-4o system card . Preprint, arXiv:2410.21276

    2024 arXiv

  10. [18]

    Yang Wang, Zheyi Sha, Kunhai Lin, Chaobing Feng, Kunhong Zhu, Lipeng Wang, Xuewu Jiao, Fei Huang, Chao Ye, Dengwu He, Zhi Guo, Shuanglong Li, and Lin Liu. 2024. https://doi.org/10.1145/3589335.3651943 One-step reach: Llm-based keyword generation for sponsored search advertisin...

    2024

  11. [19]

    Zhao Wang, Briti Gangopadhyay, Mengjie Zhao, and Shingo Takamatsu. 2025 a . https://aclanthology.org/2025.coling-industry.10/ OKG : On-the-fly keyword generation in sponsored search advertising . In Proceedings of the 31st International Conference on Computational Linguistics:...

    2025

  12. [20]

    Zhao Wang, Sota Moriyama, Wei‑Yao Wang, Briti Gangopadhyay, and Shingo Takamatsu. 2025 b . Talk structurally, act hierarchically: A collaborative framework for llm multi-agent systems. arXiv preprint arXiv:2502.11098

    2025 arXiv

  13. [21]

    Hao Wu, Guang Qiu, Xiaofei He, Yuan Shi, Mingcheng Qu, Jing Shen, Jiajun Bu, and Chun Chen. 2009. https://doi.org/10.1145/1526709.1526873 Advertising keyword generation using active learning . In Proceedings of the 18th International Conference on World Wide Web, WWW '09, page...

    2009

  14. [22]

    Frank Xing. 2025. https://doi.org/10.1145/3688399 Designing heterogeneous llm agents for financial sentiment analysis . ACM Transactions on Management Information Systems, 16(1):1–24

    2025 doi

  15. [23]

    Shinn Yao et al. 2023. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629

    2023 arXiv

  16. [24]

    Edmundas Kazimieras Zavadskas, Algimantas Zakarevicius, and Jurgita Antucheviciene. 2006. Evaluation of ranking accuracy in multi-criteria decisions. Informatica, 17(4):601–618

    2006

  17. [25]

    Weinberger, and Yoav Artzi

    Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. https://arxiv.org/abs/1904.09675 Bertscore: Evaluating text generation with bert . Preprint, arXiv:1904.09675

    2020 arXiv

  18. [26]

    Xiaoxi Zhong et al. 2023. Agentverse: Facilitating multi-agent collaboration and exploration with llms. arXiv preprint arXiv:2309.07864

    2023 arXiv

  19. [27]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  20. [28]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.