Pith. Sign in

REVIEW

CHO-Bench: How Do Context Hints Affect Ad Rankings in LLM Advertising?

PaperPDF from arXiv

data files

ReviewThe objections and verdict, as JSON Paper breakdownThe authors' claims, conditions and measurements, as JSON Signed receiptProof this review has not been altered
X Bluesky LinkedIn Reddit Hacker News
Shiqi Zhang, Yang Fei, Xiaokui Xiao

Information Retrieval · Submitted 9 Nov 2026 · 42 references

Pith's verdict: Solid benchmark for context hints in LLM ads, but the offline rankers are a real limitation — worth engaging if you work on ad-side optimization.

T0 · AI review

How this review was made: An AI referee (deepseek/deepseek-v4-flash) read the paper on November 10, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

  1. T0 AI review An AI read the paper and answered a fixed list of questions.
  2. T1 Audit Automatic checks that the references and files the paper relies on exist and match.
  3. T2 Compute The paper's code or calculations re-run with the same results.
  4. T3 Formal statement The claims written out in a formal math language a computer can read.
  5. T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 4 major objections · 9 minor · Ask about this paper · The compiled record · Read the full review →
Challenge this review Comment (0) Read on arXiv ↗
Cite ▾
arXiv ID2611.11955 (version 1)
Pith ID5AAOEXB2, Pith's permanent record of this paper and its review
CitationShiqi Zhang, Xiaokui Xiao, Yang Fei. "CHO-Bench: How Do Context Hints Affect Ad Rankings in LLM Advertising?." arXiv:2611.11955 (2026). https://pith.science/paper/5AAOEXB2
Cite this reviewBibTeX for Pith's review of this paper
Claim paper

Why Pith says this

Highlight any text to comment on it.

Referee report

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

Context hints improve advertisement rankings in LLM advertising, but the best information strategy depends on the ranking model and hint budget.

The reading

Large language model (LLM) assistants like ChatGPT are becoming advertising platforms. Advertisers can provide short text descriptions called 'context hints' that describe what they offer, who might want it, and when it might be useful. These hints help the LLM decide which ads to show in response to a user's conversation, even when the user isn't explicitly searching for a product. This paper creates a benchmark called CHO-Bench to test how different types of context hints affect ad rankings. The authors developed seven strategies for writing hints, based on marketing frameworks, covering product features, customer needs, and usage situations. They tested these strategies across five datasets, using both embedding-based similarity and LLM-based ranking models, with over 1.19 million evaluations. The results show that context hints consistently improve ad rankings, but the best type of hint depends on the ranking model: embedding-based rankers prefer 'when' information (usage scenarios), while LLM rankers prefer 'who' information (customer segments). Combining all hint types does not consistently outperform individual strategies. Increasing the number of hints improves rankings but with diminishing returns, especially for embedding-based rankers.
Extended reading notes

Core claim

Context hints improve advertisement rankings under the evaluated offline rankers, achieving gains of 0.56–1.16 positions in Δ Rank and 5.42–14.40 percentage points in Δ Win across general and product advertising datasets (Table 2, Section 6.1).

Load-bearing premise

The evaluation assumes that the offline rankers (embedding-based similarity and LLM-based listwise ranking) are reasonable proxies for the proprietary ranking functions used in real LLM advertising systems like ChatGPT Ads. The paper acknowledges this limitation in Section 7: 'Our evaluation measures ranking changes rather than independently verified ad relevance or outcomes when competing advertisements also use context hints.' The ranking functions used are simplified versions that may not capture the complexity of production systems, which could include auction mechanisms, user personalization, and other factors.

Original abstract

LLM advertising selects advertisements based on user conversations, where user needs may extend beyond explicit product searches. ChatGPT Ads allows advertisers to provide natural-language context hints describing what they offer, who may find it useful, and when it may be useful, as additional information for ranking advertisements across diverse user conversations. However, how to effectively construct these hints remains unclear. We formulate Context Hint Optimization (CHO) as the problem of constructing context hints to improve an advertisement's ranking under a given budget. Drawing on established marketing frameworks, we develop seven context hint generation strategies covering What, Who, and When, along with a Unified strategy that combines all seven dimensions. We introduce CHO-Bench and conduct over 1.19 million paired evaluations across five datasets using embedding-based and LLM-based rankers. Our results show that context hints improve advertisement rankings under the evaluated offline rankers. While individual strategies achieve similar average gains and Unified offers no consistent advantage, strategy differences become more pronounced under specific generation-ranking configurations, with embedding-based and LLM-based rankers favoring different information dimensions. Increasing the hint budget further improves rankings with diminishing marginal gains. These findings establish an offline benchmark for studying ad-side ranking optimization in LLM advertising and highlight opportunities for model-specific hint selection and budget-aware generation and aggregation.

The compiled record

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. Numbers in the tables are the authors', checked by code. No sentence in this record was written by a model.

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-11-10: 1 request, 7,449 input tokens.

What the abstract itself states, as probabilities: claims first 0.10 · states limitation 0.75 · human participants 0.07 · controlled comparison 0.66 · randomized assignment 0.03 · states quantitative result 0.96 · compares with prior results 0.60 · mentions code or data availability 0.06.

The abstract, sentence by sentence

arXiv listing abstract v1. Results and the main claim are underlined; .

LLM advertising selects advertisements based on user conversations, where user needs may extend beyond explicit product searches. ChatGPT Ads allows advertisers to provide natural-language context hints describing what they offer, who may find it useful, and when it may be useful, as additional information for ranking advertisements across diverse user conversations. However, how to effectively construct these hints remains unclear. We formulate Context Hint Optimization (CHO) as the problem of constructing context hints to improve an advertisement's ranking under a given budget. Drawing on established marketing frameworks, we develop seven context hint generation strategies covering What, Who, and When, along with a Unified strategy that combines all seven dimensions. We introduce CHO-Bench and conduct over 1.19 million paired evaluations across five datasets using embedding-based and LLM-based rankers. Our results show that context hints improve advertisement rankings under the evaluated offline rankers. While individual strategies achieve similar average gains and Unified offers no consistent advantage, strategy differences become more pronounced under specific generation-ranking configurations, with embedding-based and LLM-based rankers favoring different information dimensions. Increasing the hint budget further improves rankings with diminishing marginal gains. These findings establish an offline benchmark for studying ad-side ranking optimization in LLM advertising and highlight opportunities for model-specific hint selection and budget-aware generation and aggregation.

background contribution method result interpretation main claim

What the classifier reports for each sentence. The number is the probability it gave the label.
#sentencerolethe authors' own workstates the main conclusionstrength
s0 LLM advertising selects advertisements based on user conversations, where user needs may extend beyond explicit product searches. background 1.00 0.06 0.02 hedged possibility 0.74
s1 ChatGPT Ads allows advertisers to provide natural-language context hints describing what they offer, who may find it useful, and when it may be useful, as additional information for ranking advertisements across diverse user conversations. background 1.00 0.07 0.02 asserted 0.99
s2 However, how to effectively construct these hints remains unclear. gap 1.00 0.05 0.01 asserted 0.73
s3 We formulate Context Hint Optimization (CHO) as the problem of constructing context hints to improve an advertisement's ranking under a given budget. contribution 0.83 0.83 0.03 asserted 0.95
s4 Drawing on established marketing frameworks, we develop seven context hint generation strategies covering What, Who, and When, along with a Unified strategy that combines all seven dimensions. method 0.71 0.94 0.03 asserted 1.00
s5 We introduce CHO-Bench and conduct over 1.19 million paired evaluations across five datasets using embedding-based and LLM-based rankers. method 0.78 0.97 0.04 asserted 0.99
s6 Our results show that context hints improve advertisement rankings under the evaluated offline rankers. result 1.00 0.96 0.54 asserted 1.00
s7 While individual strategies achieve similar average gains and Unified offers no consistent advantage, strategy differences become more pronounced under specific generation-ranking configurations, with embedding-based and LLM-based rankers favoring different information dimensions. result 0.95 0.96 0.40 asserted 1.00
s8 Increasing the hint budget further improves rankings with diminishing marginal gains. result 1.00 0.93 0.09 asserted 1.00
s9 These findings establish an offline benchmark for studying ad-side ranking optimization in LLM advertising and highlight opportunities for model-specific hint selection and budget-aware generation and aggregation. interpretation 0.80 0.82 0.43 asserted 0.99

What kind of paper the classifier reports this is: dataset or benchmark 0.80 · evidence from observation or experiment 0.94.

What was examined, and what was not

sources read
  • arXiv listing abstract v1
not examined
  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers
levels run
  • P abstract
not run
  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. Every number in the measurements table is the authors', checked by code. Record as JSON

Figures
Figure 1. Real example in ChatGPT [13]. Left: Queries and ad￾vertisements displayed in ChatGPT’s responses. Right: Chat￾GPT’s response to “How can I build muscle?” and the dis￾played advertisement for high-protein meals. query does not directly request meal delivery, ChatGPT’s response discusses protein intake, providing a connection to HelloFresh’s Protein Smart meals. This example illustrates how the same adver￾tisement can be associated with diverse user needs and situations. In ChatGPT Ads, advertisers can provide free-form natural￾language context hints describing what they offer, who…
Figure 2. Overview of CHO-Bench. For each ad (Steps 1–3), its ad corpus is used to generate context hints. For each query (Steps 4–5), the ad ranker ranks ads in its candidate pool based on ad cards and context hints, followed by rank-based evaluation. strategies corresponding to products and services, pain relievers, and gain creators. Product Features. This strategy captures factual information about an offering, including its features, capabilities, services, specifica￾tions, pricing, and availability, rather than customer outcomes. For HelloFresh, the collector identifies Protein Smart…
Figure 3. Ranking gains from context hints across generation
Figure 5. Effects of hint budget 𝑘 on Unified across five datasets. Rows show Δ Rank and Δ Win, with curves representing eight generation–ranking configurations. Hollow markers indicate unconstrained generation at the average number of generated hints. The 𝑘 = 3–5 interval is expanded for readability. strategies
Figure 6. Relative rankings of seven dimension-specific

Showing 5 of the paper's 6 figures. See all figures in the PDF ↗ Figures reproduced from arXiv:2611.11955 by the authors.

References

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.