Pith. sign in

REVIEW 4 major objections 5 minor 15 references

OKG: On-the-Fly Keyword Generation in Sponsored Search Advertising

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read OKG claims that an LLM agent that monitors live sponsored-search KPIs and regenerates keywords each cycle outperforms static LLM, extraction, and keyword-planner baselines.

desk verdict Useful dataset and a sensible agent architecture, but the headline performance claim is unproven because all KPIs are imputed from offline nearest neighbors, not live delivery. read the letter →

arxiv 2412.03577 v1 pith:TR3KPUXL submitted 2024-11-18 cs.IR cs.AI

Zhao Wang , Briti Gangopadhyay , Mengjie Zhao , Shingo Takamatsu This is my paper · ORCID
classification cs.IRcs.AI
keywords sponsoredsearchadvertisingkeywordgenerationLLMagentreal-timeKPImonitoringadaptiveexpansionpublicdatasetretrieval-augmented
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that sponsored-search keyword generation does not have to depend on large static datasets, which most advertisers cannot obtain. The proposed system, OKG, is an LLM agent that repeatedly searches for current product and market information, reads the live performance of its own generated keywords (clicks, cost per click, competitor score), and then produces the next keyword list by deepening winning categories and widening into new ones while keeping the total number of keywords fixed. The authors claim this on-the-fly loop yields higher clicks, lower cost per click, and lower competitor scores than GPT-4, Gemini-1.5-Pro, keyword extractors, and Google Keyword Planner in their evaluation, and that ablations show the adaptive split between deeper and wider expansion is what drives the improvement. The paper also releases a public dataset of real Japanese keyword records with KPIs across several product domains, which it says is the first such resource for the field.

What carries the argument

The load-bearing mechanism is the OKG agent loop, a closed cycle that turns live performance feedback into the next keyword list. At time step $t$, the agent observes the current keyword set $k_t$ and its measured KPIs $P_t$, pulls real-time product and market information $S_t$ through a search tool, and retrieves historical keyword performance from a vector memory via RAG; a calculation tool then splits the fixed budget of $n$ keywords between wider expansion $W_t$ (new keyword categories) and deeper expansion $D_t$ (more specific keywords in proven categories). The next list is $k_{t+1}=W_t \cup D_t$, with $|W_t| = \lfloor p^W_t \cdot n \rfloor$ and $|D_t| = n - |W_t|$, where $p^W_t$ and $p^D_t$ are the previous step's KPI shares of the wider and deeper keywords. This feedback loop is the entirety of the argument: live KPI changes redirect the agent's keyword choices, which static offline-trained generators cannot do.

What would settle it

Run the exact keyword lists produced by OKG and by GPT-4 as separate live Google Ads campaigns for the same products over the same three time steps, then compare actual clicks, cost per click, and competitor scores; if OKG's advantage over GPT-4 disappears or reverses under real delivery data, the offline nearest-neighbor KPI surrogate, rather than the generation method, produced the reported gains.

Watch

Extended reading notes

Core claim

The central discovery claimed in the paper is that keyword generation in sponsored search can be posed as an online decision problem—maximize total KPI over a time horizon T with a fixed per-step keyword budget n—and solved by an LLM agent that closes the loop between generation and live measurement. At each time step the agent observes the current keyword set and its KPIs, retrieves real-time product and market information, and computes the next keyword set as a union of wider expansions (new categories) and deeper expansions (more specific keywords in proven categories), with the split proportional to the previous step's KPI shares. The authors report that this approach outperforms LLM-based, keyword-extraction, and commercial-planner baselines on clicks, cost per click, and competitor score, and also produces higher relevance/coverage scores against online search results and higher similarity to offline real keywords. An ablation study is used to argue that neither wider-only nor deeper-only growth, nor a fixed split, nor a reflection-based variant matches the full adaptive mechanism, supporting the claim that real-time KPI feedback is the decisive component. The paper additionally contributes what it calls the first publicly accessible dataset of real keyword data with KPIs across diverse domains.

Load-bearing premise

The reported KPI comparisons assume that a generated keyword's real clicks, cost per click, and competitor score are accurately represented by the values of the most textually similar keyword in the offline dataset; if embedding similarity does not predict actual campaign delivery, the headline performance gains are measuring resemblance to old keywords rather than real advertising success.

Editorial extensions

If this is right

  • If OKG is correct, an advertiser can start a sponsored-search campaign from just a product name and let the agent maintain and refresh the keyword list from live performance data, eliminating the need for a large pre-collected keyword dataset.
  • Keyword lists would become time-varying assets: under a fixed per-step budget, the agent shifts slots toward categories that recently earned clicks, so budget is reallocated away from underperforming keywords without manual intervention.
  • The released public dataset with real keyword KPIs would give future researchers a shared benchmark, making keyword-generation methods comparable on identical real-world performance data.
  • The ablation result that reflection on past steps does not improve quality would suggest that, for keyword selection, current live signals are more valuable than accumulated historical experience.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A test the paper leaves implicit is whether the offline nearest-neighbor KPI surrogate tracks real delivery: one could deploy the generated keyword lists in a live campaign and compare actual clicks and costs to the surrogate values, which would separate generation quality from evaluation artifact.
  • The wider/deeper proportional allocation rule is a generic exploration-exploitation schedule that could transfer to other fixed-budget list-generation settings, such as query suggestions, trending hashtags, or product recommendations, wherever per-step performance is observable.
  • If real-time feedback indeed matters more than reflection on past experience, a natural next experiment is to vary the feedback delay (daily vs weekly KPI updates) to map how quickly the agent loses its advantage as signals age.
  • The paper's evaluation covers only Japanese keywords and Sony products; whether OKG's gains persist across languages, product categories, and advertising platforms (e.g., Bing Ads or Yahoo) is an open question that the released dataset and code make testable.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces OKG, an LLM-agent framework for sponsored-search keyword generation that uses an online search tool, a vector memory module with retrieval-augmented generation, and a calculation tool to allocate new keywords between wider (new categories) and deeper (existing categories) expansion based on previously observed KPI data. The authors also release a public dataset of real Japanese keyword KPI records for ten Sony products. The experimental section compares OKG with GPT-4, Gemini-1.5-Pro, keyword extraction baselines, and Google Keyword Planner on imputed KPIs, relevance/coverage scores, and similarity to offline keywords, and reports that OKG outperforms all baselines. The paper concludes that OKG significantly improves keyword adaptability and responsiveness.

Significance. The proposed setting is timely: keyword generation in sponsored search usually relies on static offline data, and a system that continuously adapts keyword lists from performance feedback would be practically valuable. The public dataset and released code are a concrete contribution that could support future work. However, the empirical evidence for the headline claim is currently not adequate: the KPI evaluation is based on nearest-neighbor imputation from offline data rather than live campaign delivery, and the adaptive feedback loop is exercised only on those imputed values. The usefulness of the dataset and the overall architecture are independent of that evidence, but the claimed performance advantage is not established by the experiments as reported.

major comments (4)
  1. [§5.1 (Table 1)] The central performance claim rests on KPI imputation, not on observed delivery. The paper states: 'For each generated keyword, we select the most similar keyword from offline data (highest similarity score and cosine similarity > 0.6) and use its KPIs to represent the generated keyword's KPIs.' Consequently, every Click, Search Volume, CPC, and Competitor Score in Table 1 is a copy of a historical keyword's KPI selected by embedding similarity. No evidence is given that BERT cosine similarity above 0.6 is predictive of actual advertising KPIs for a newly generated keyword. Because §4.2 uses Pt−1 (these imputed values) to compute pW_t and pD_t, the entire 'real-time adaptation' loop is simulated with synthetic feedback rather than measured against platform behavior. The headline claim that OKG 'significantly improves keyword performance' is therefore unsupported.
  2. [§5.1–§5.3] The evaluation is not an independent comparison. Generated keywords are scored against the same offline vocabulary that OKG's search and memory components are designed to exploit, and the KPI assignment rule gives higher scores to keywords that are embedding-neighbors of high-performing historical keywords. OKG is explicitly optimized to produce keywords similar to the offline data, so the comparison rewards matching the KPI database rather than delivering better campaign outcomes. A valid evaluation would require a held-out set of keywords with known KPIs, a validated surrogate, or a live A/B test with actual ad delivery; none of these is reported.
  3. [§5, Implementation Details; Fig. 3] No statistical support is provided for 'significantly improves.' The experiments use T=3 with a single run, and Table 1 reports only normalized point values without error bars, confidence intervals, or significance tests. The ablation in Fig. 3 is performed on a single product (Sony TV) with no repeated trials, so it cannot support the claim that full OKG is reliably better than the ablations. The paper should report multiple runs, per-product results, and appropriate uncertainty or significance measures.
  4. [§3 and §4.2] The problem setting defines Pt as 'observed KPI' from the ad platform, and the objective is to maximize the sum of Pt over T. However, in the experiments Pt is never observed from a platform; it is taken from static offline data via nearest-neighbor retrieval. This mismatch between the formal setting and the evaluation means that the paper does not demonstrate the core claim of on-the-fly adaptation to real-time KPI changes. The authors either need to run the system against the Google Ads API (which they cite as a component) for a real campaign, or explicitly reframe the experiments as a simulation and validate the surrogate.
minor comments (5)
  1. [References] The reference list contains two entries with the same key 'Google. 2024' (Google Ads API and Google Keyword Planner), and the in-text citations use both 'Google, 2024a' and 'Google, 2024b'; please disambiguate the keys consistently.
  2. [Section 3] The notation for the keyword set is inconsistent: the text alternates between lowercase 'kt' and uppercase 'Kt' for the same concept, and the definition 'K = union over t of Kt' makes the cumulative set and the per-step set easy to confuse in the objective function.
  3. [Table 1] The table header contains formatting typos ('Srch. V ol.', 'N.(0∼100)', 'Gemini1.5' missing a space, and inconsistent spacing in 'Kwd. Ext.'). Please clean up the table formatting.
  4. [Appendix C] Because the examples are translated from Japanese, the original Japanese keyword outputs should be included alongside the English translations so that the claimed specificity and brand alignment can be independently checked.
  5. [Section 5.1] The sentence 'Generated and original keywords are tokenized and embedded (using pooled embeddings) from a pretrained multilingual BERT model' should specify which pooling method and checkpoint were used, to support reproducibility.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline performance gains rest on KPI imputation from offline nearest neighbors; Table 1's 'performance' is constructed from the same offline data OKG exploits.

  1. fitted input called prediction [Section 5.1, Table 1]
    "The KPIs are obtained from our public dataset. Generated and original keywords are tokenized and embedded (using pooled embeddings) from a pre-trained multilingual BERT model (Google, 2024b) to measure cosine similarity. For each generated keyword, we select the most similar keyword from offline data (highest similarity score and cosine similarity > 0.6) and use its KPIs to represent the generated keyword's KPIs."

    Table 1's 'keyword performance' is not measured from any live campaign. By construction, every generated keyword's clicks, search volume, CPC, and competitor score are defined to equal those of its nearest offline neighbor under cosine similarity. Thus OKG's reported superiority in Clicks, CPC, and Competitor Score is a deterministic consequence of generating keywords that are close to high-KPI offline entries, not an independent measurement of real ad delivery. Moreover, the same imputed KPI is fed back as Pt-1 in Section 4.2 to decide Wt and Dt, so the adaptive allocation and the evaluation share the same unvalidated surrogate; the central claim of real-time, performance-driven adaptation is never tested against actual platform feedback.

full rationale

The main circular step is the evaluation protocol in Section 5.1. The paper assigns each generated keyword the KPI of its most similar offline keyword (cosine similarity > 0.6) and then reports this imputed value as real keyword performance. This makes the headline performance comparison partly self-confirming: 'better performance' means 'more similar to offline keywords with high KPIs,' which is the same information OKG retrieves and uses for its adaptive allocation. Sections 5.2 and 5.3 provide more independent evidence using BERTScore, BLEU, ROUGE, Jaccard, and cosine similarity against search results and offline keywords; those comparisons are not circular. However, the paper's strongest claim—that OKG significantly improves keyword performance through real-time KPI monitoring—rests on Table 1, which is built from offline nearest-neighbor imputation. No live delivery data, error bars, or significance tests are reported for the claimed improvements. This warrants a score of 6: the central performance claim partially reduces to the imputation rule, while the dataset and relevance/coverage evaluations retain independent content.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on two categories of unpaid assumptions: the KPI transfer from offline neighbors and the reliability of live search and ads APIs. The cosine threshold is the only explicit numeric free parameter. No new physical or conceptual entities are postulated.

free parameters (3)
  • cosine similarity threshold = 0.6
    Used in Section 5.1 to decide whether a generated keyword can borrow the KPI of the nearest offline keyword. No sensitivity analysis is provided.
  • LLM temperature = 0.1
    LLM sampling temperature in Section 5; hand chosen, affects output diversity and reproducibility.
  • time horizon T = 3
    Number of adaptation rounds in Section 5; chosen because three iterations reach ~100 keywords and runtime is about two hours. Not justified as optimal.
assumptions (4)
  • domain assumption A generated keyword's real performance equals the KPI of the most similar offline keyword when BERT cosine similarity exceeds 0.6.
    Section 5.1 states this is how all performance metrics for generated keywords are obtained; if this transfer is invalid, the reported Click, CPC, and Competitor values are not measurements.
  • domain assumption Real-time web search and Google Ads API data accurately reflect current market conditions and keyword KPIs.
    Section 4.1 describes the search tool and Google Ads API as trustworthy sources for St and Pt; no validation of retrieval accuracy is given.
  • domain assumption Multilingual BERT pooled embeddings capture enough semantic similarity between Japanese keywords to support KPI transfer and offline similarity evaluation.
    Section 5.1 and Table 3 rely on BERT embedding similarity; no calibration of embedding distance against actual keyword performance is provided.
  • ad hoc to paper Three iterations are sufficient to demonstrate the superiority of OKG.
    Section 5 states T=3 due to keyword cap and runtime; there is no evidence that longer horizons would not change the ranking.

how reviews work

0 comments
Cite this review

Pith. "Pith review of OKG: On-the-Fly Keyword Generation in Sponsored Search Advertising." pith.science (2026). https://pith.science/paper/TR3KPUXL

@misc{pith2026241203577,
  author       = {Pith},
  title        = {Pith review of: OKG: On-the-Fly Keyword Generation in Sponsored Search Advertising},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TR3KPUXL}},
  note         = {Machine review of arXiv:2412.03577}
}
read the original abstract

Current keyword decision-making in sponsored search advertising relies on large, static datasets, limiting the ability to automatically set up keywords and adapt to real-time KPI metrics and product updates that are essential for effective advertising. In this paper, we propose On-the-fly Keyword Generation (OKG), an LLM agent-based method that dynamically monitors KPI changes and adapts keyword generation in real time, aligning with strategies recommended by advertising platforms. Additionally, we introduce the first publicly accessible dataset containing real keyword data along with its KPIs across diverse domains, providing a valuable resource for future research. Experimental results show that OKG significantly improves keyword adaptability and responsiveness compared to traditional methods. The code for OKG and the dataset are available at https://github.com/sony/okg.

Figures

Figures reproduced from arXiv: 2412.03577 by the authors.

Figure 1
Figure 1. This visual contrasts the traditional keyword [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The architecture of OKG, which fulfills the functionality of online search, real-time keyword and KPI [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Comparison Results of Component Ablation. [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: An intuitive example of OKG generation prompt for Sony Bank’s Mortgage Service Step 5: Generating and Implementing New Keywords Outcome: Utilizing the insights gained from the above steps, OKG generates a tailored list of new keywords.. Iterative Refinement Everytime s…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

15 extracted references · 7 canonical work pages

  1. [4]

    Jeff Johnson, Matthijs Douze, and Hervé Jégou

    Olympicarena medal ranks: Who is the most intelligent ai so far? Preprint, arXiv:2406.16772. Jeff Johnson, Matthijs Douze, and Hervé Jégou

    arXiv

  2. [5]

    https: //github.com/facebookresearch/faiss

    Billion-scale similarity search with gpus. https: //github.com/facebookresearch/faiss. Ac- cessed: 2024-09-25. Mu-Chu Lee, Bin Gao, and Ruofei Zhang

    work page 2024

  3. [8]

    arXiv preprint arXiv:2403.05530

    Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Kay Römer, Benedikt Ostermaier, Friedemann Mat- tern, Michael Fahrmair, and Wolfgang Kellerer

    arXiv

  4. [10]

    https://serpapi.com/

    Serp: Real-time search engine data for seo and marketing. https://serpapi.com/. Accessed: 2024-09-25. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao

    work page 2024

  5. [11]

    arXiv preprint arXiv:1409.3215

    Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215. Yang Wang, Zheyi Sha, Kunhai Lin, Chaobing Feng, Kunhong Zhu, Lipeng Wang, Xuewu Jiao, Fei Huang, Chao Ye, Dengwu He, et al

    arXiv

  6. [12]

    In Companion Proceedings of the ACM on Web Conference 2024, pages 1604–1608

    One-step reach: Llm-based keyword generation for sponsored search advertising. In Companion Proceedings of the ACM on Web Conference 2024, pages 1604–1608. Yanwu Yang and Huiran Li

    work page 2024

  7. [13]

    arXiv preprint arXiv:1904.09675

    Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.09675. Hao Zhou, Minlie Huang, Yishun Mao, Changlei Zhu, Peng Shu, and Xiaoyan Zhu

    arXiv 1904

  8. [14]

    arXiv preprint arXiv:2305.09612

    Large language models are built- in autoregressive search engines. arXiv preprint arXiv:2305.09612. A An Example of OKG Generation Prompt In this section, we provide an intuitive example illustrating how OKG generates keyword sugges- tions through a structured, multi-step prompt as shown in Figure

    arXiv

Show all 15 references
  1. [15]

    Core Service

    Query Understanding The process begins with a user query to set up SSA keywords for Mortgage Service of Sony Bank. OKG parses this query to understand the specific requirements—such as the product focus (mort- gage services) and the target entity (Sony Bank). Step 1: Gathering...

    2024

  2. [2006]

    Bulletin-American Society For Information Science And Technology, 32(2):12

    Sponsored search: A brief history. Bulletin-American Society For Information Science And Technology, 32(2):12. Google. 2024a. About adjusting your keyword bids. Accessed: 2024-09-17. Google. 2024b. Bert multilingual model. https://github.com/google-research/bert/ blob/master/m...

    2024

  3. [2010]

    Proceedings of the IEEE, 98(11):1887–1902

    Real-time search for real-world entities: A survey. Proceedings of the IEEE, 98(11):1887–1902. Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley

    1902

  4. [2014]

    arXiv preprint arXiv:1411.1784

    Condi- tional generative adversarial nets. arXiv preprint arXiv:1411.1784. Han Nie, Yanwu Yang, and Daniel Zeng

    arXiv

  5. [2019]

    arXiv preprint arXiv:1902.00592

    An end-to-end generative retrieval method for sponsored search engine–decoding efficiently into a closed tar- get domain. arXiv preprint arXiv:1902.00592. Chin-Yew Lin

    1902 arXiv

  6. [2023]

    arXiv preprint arXiv:2303.08774

    Gpt-4 technical report. arXiv preprint arXiv:2303.08774. Choi

    arXiv

  7. [2024]

    https://developers

    Google ads api. https://developers. google.com/google-ads/api. Accessed: 2024- 09-25. Google

    2024

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.