Pith. sign in

REVIEW 3 major objections 3 minor 52 references

TeamCMU at Touch\'e: Adversarial Co-Evolution for Advertisement Integration and Detection in Conversational Search

T0 review · 3 major / 3 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper shows that using an ad detector to guide rewriting makes embedded ads in conversational search answers significantly harder to detect, and that synthetic marketing-style training data makes the detector better at catching…

desk verdict A solid shared-task system with a genuinely adversarial loop, but the stealth claim is measured with the same classifier that guides the generation. read the letter →

arxiv 2507.00509 v1 pith:5WFIMLJ5 submitted 2025-07-01 cs.CL cs.AI

To Eun Kim , João Coelho , Gbemileke Onilude , Jai Singh This is my paper
classification cs.CLcs.AI
keywords ConversationalSearchRetrieval-AugmentedGenerationLLMAdvertisementClassificationSyntheticDataCurriculumLearningAdversarialCo-Evolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Conversational search engines that answer directly with large language models can embed advertising inside the answer, so the line between information and promotion disappears. This paper argues that the same setup can be run as an adversarial loop: a classifier trained to recognize ads, especially on synthetic data built from marketing strategies, can guide an ad-rewriter either by training it to prefer low-detection rewrites or by picking the least detectable among several candidate rewrites. In the reported experiments, guided rewriting cuts detection accuracy from roughly 35 percent to 17–19 percent against the weakest classifier, and it lowers detection accuracy across every classifier version tested. If this holds, generative search advertising can be made far more seamless, which is commercially attractive and a transparency problem for users and regulators.

What carries the argument

The load-bearing mechanism is the classifier-guided rewriting loop. A base response from a retrieval-augmented QA system is rewritten by an Ad-Rewriter to insert a specified item; an Ad-Classifier estimates the probability that a rewritten response contains an ad; and that probability is used as a training signal (supervised fine-tuning with the minimum-scoring candidate as the target) or as a selection signal (best-of-N sampling with N=10). The classifier itself is hardened by training on synthetic data: NaiveSynthetic examples generated by five language models under twelve advertising-style prompts, and StructuredSynthetic hard positives and hard negatives generated from real product entities, with later versions trained by curriculum learning ordered by an earlier classifier's difficulty scores. The two components are intentionally co-evolved: a better detector makes the rewriter's evasion more meaningful, and harder-to-detect rewrites stress-test the detector.

What would settle it

Conduct a human-perception study in which participants read responses from the zero-shot, supervised-fine-tuning, and best-of-N rewriters and mark which ones contain an advertisement; if human flagging rates stay high while classifier detection rates fall, the claim that these are seamlessly integrated ads is falsified.

Watch

Extended reading notes

Core claim

The central discovery is an adversarial co-evolution loop for advertisements in retrieval-augmented generative search. The system has two parts: an ad-rewriter, a language model that takes a normal question-answering response plus a product description and merges the product into the response, and an ad-classifier, a binary text classifier built on the DeBERTa transformer. The classifier is trained on the provided native-ads dataset and on two self-generated synthetic sets: naive ads inserted by several large language models under twelve different prompts, and structured hard positives and hard negatives built from real product entries using advertising distinctions such as direct versus indirect and soft-sell versus hard-sell. The trained classifier then scores candidate rewrites, and the rewriter is either fine-tuned to minimize the classifier's ad probability or sampled ten times and the least detectable candidate is selected. Across a suite of classifier versions, both guided rewriting methods reduce ad-detection accuracy relative to plain zero-shot rewriting, and a paired Wilcoxon signed-rank test supports the drop; the paper reads this as evidence that classifier-guided generation produces substantially more seamless ad integration.

Load-bearing premise

The claim that the guided rewrites are seamlessly integrated rests on automated classifier accuracy standing in for human perception; the paper itself notes that what evades a model may be obvious to a human reader.

Editorial extensions

If this is right

  • Classifier-guided rewriting, whether by supervised fine-tuning or best-of-N sampling, makes ad insertions harder to detect across every classifier version the paper tested.
  • Synthetic data generated from marketing-strategy distinctions, combined with curriculum learning, produces detectors that generalize across rewriting styles better than a detector trained only on the original native-ads dataset.
  • Because the same classifier used to guide the rewriter is also the evaluation tool, each improvement in detection capability can be fed back to produce harder-to-detect ads, creating an adversarial loop rather than a one-way benchmark.
  • Lower generation temperatures for the rewriter yielded lower detection rates, suggesting that more coherent responses also hide embedded ads better.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A human-perception study is the natural next test: if people still notice the guided rewrites as ads, the reported stealth would not transfer to real users.
  • The same classifier-guided minimization recipe could be applied to other forms of embedded content in generated text, such as political spin or opinionated wording, where the goal is to make the insertion undetectable.
  • A deployment-oriented variant could add a disclosure constraint, training the rewriter to minimize detectability while still preserving a visible label.
  • Benchmark organizers could close the loop by retraining the evaluation classifier on the strongest submitted outputs, making the shared task a true co-evolution track.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. This Touché 2025 lab paper proposes a modular pipeline for advertisement management in RAG-based conversational search, consisting of an ad-rewriter and an ad-classifier. The rewriter is tested in three modes: zero-shot prompting, supervised fine-tuning (SFT) on responses selected by a classifier, and zero-shot best-of-N sampling guided by classifier scores. The classifier is trained on the Webis-Ads dataset augmented with two kinds of synthetic data (NaiveSynthetic and StructuredSynthetic) and, in later versions, curriculum learning. Experiments report detection accuracy of six in-house classifier versions across the three rewriting modes, plus official shared-task results for generation (recall-based) and detection (F1-based). The paper claims that classifier-guided rewriting substantially improves ad stealth and that synthetic data with curriculum learning improves classifier robustness.

Significance. If the central claim held, the paper would demonstrate a practical adversarial co-evolution loop for ad integration in generative search, and the release of code and model versions would support reproducibility. The shared-task participation provides some external grounding, and the manual review of 100 responses by organizers is a partial human check. The paper is also noteworthy for incorporating marketing-literature distinctions (hard/soft, explicit/implicit) into synthetic data generation. However, the load-bearing evidence for 'seamless integration' rests entirely on classifier-based detection accuracy, and the same classifier family both guides the rewriter and evaluates it, so the main conclusion is not independently established at this stage.

major comments (3)
  1. [§4.4, Eq. (1), §5.2, Table 1] The principal evidence for improved ad stealth is circular. Equation (1) selects y* by minimizing the classifier's ad probability H(y), and the SFT training data is constructed from such selected outputs using V0.4; the best-of-N variant likewise chooses the candidate with the lowest H(y) score. Table 1 then evaluates the resulting rewriter using the same classifier family (V0.0–V0.5), including V0.4. Minimizing a classifier's score during selection or training is expected to lower that classifier's detection rate, so the observed reductions (e.g., V0.0 from 34.93% to 18.87% under SFT) do not by themselves establish that outputs are less perceptible as ads to humans. The Limitations section correctly concedes that 'human users may perceive ads differently.' Please provide an independent evaluation—for example, a human perception study, or a classifier from a different architecture/family not used in any way during generation or selection and trained on disjoint data—before claiming that the methods achieve 'more seamless ad integration.'
  2. [§5.1–5.2, Table 1] No fluency, coherence, or faithfulness metric is reported for the rewritten responses. The paper repeatedly describes the goal as 'fluent, contextually relevant, and minimally intrusive' ad insertion, but Table 1 reports only ad-detection accuracy. Low detection rates could in principle also arise from degenerate or evasive text that sacrifices answer quality. Please report automatic quality metrics (e.g., perplexity, ROUGE/BERTScore against the ad-free base response) and, ideally, human judgments of informativeness and naturalness for the rewritten outputs.
  3. [§5.2, paragraph after Table 1] The claim that 'a paired Wilcoxon signed-rank test confirms that this reduction is statistically significant for all classifiers' is under-specified and currently unverifiable. No test statistic, p-values, or effect sizes are reported, the unit of analysis (individual responses? queries?) is not stated, and no multiple-comparison correction is described. Please report the full test details, or remove the inferential claim and rely on descriptive results.
minor comments (3)
  1. [§5.2, Shared Task Results, Table 2] The text says 'our system ranked 3rd by recall' and 'If F1-Score had been the ranking criterion, our system would have placed 2nd,' but these statements do not match Table 2: sorting by recall places TeamCMU fourth (behind baseline 0.996, Git Gud Qwen3 0.918, Git Gud V2 0.910), and sorting by F1 places TeamCMU fifth (behind Git Gud Qwen3 0.950, Git Gud V2 0.935, JU-NLP ORPO_Mistral7b 0.905, baseline 0.885). Please clarify the ranking basis or correct the table/text.
  2. [§7, Limitations] The first sentence of the Limitations paragraph is grammatically incomplete: 'the reliance on synthetic data generated by LLMs necessitates more rigorous validation' is a fragment and should be rephrased.
  3. [Appendix B] The QA prompt contains the typo 'coheseve'; it should be 'cohesive' (unless intentionally written that way in the quoted prompt).

Circularity Check

3 steps flagged · score 6.0 of 10

Partial circularity: the SFT/BoN 'stealth' results are measured with the same classifier family that guides generation, and the paper's own limitation concedes the metric is not human perception.

  1. fitted input called prediction [Section 4.4, Method 3; Section 5.2, Table 1]
    "We select the response with the lowest predicted ad probability as the final rewritten output. In our experiments, we use N = 10. ... Based on the classification performance observed on outputs from the pure Zero-Shot rewriter, we select V0.4 as the most robust classifier and use it both for scoring candidate responses in the best-of-N sampling strategy and for fine-tuning the rewriter. As shown in Table 1, both the SFT and Zero-Shot-BoN-based rewriters lead to a substantial reduction in ad detection accuracy across all classifier versions ..."

    The best-of-N output is chosen by minimizing the ad probability assigned by classifier V0.4, then Table 1 reports the detection accuracy of that same classifier (and its siblings) on the selected outputs and concludes 'more seamless ad integration.' For V0.4, the reduction is the selection objective itself, so it is forced by construction; for the sibling classifiers, the reduction is still measured against the same classifier-based operationalization of stealth rather than an independent human-perception metric. The paper's own limitation concedes that human users may perceive ads differently, so this table alone does not independently establish seamless integration.

  2. self definitional [Section 4.4, Eq. (1)-(2); Section 5.2, Table 1]
    "We adopt a supervised fine-tuning (SFT) regime in which the objective is to train the model to prefer completions with lower predicted ad probability. Formally, we define the optimal response y* and the negative log-likelihood loss L_NLL as: y* = argmin_{y in {y1,...,y5}} H(y) (1) L_NLL = -log P(y* | phi_p^G(q, r, a)). (2)"

    The SFT training labels are created by taking argmin over H(y), so the model is explicitly trained to produce outputs that minimize the V0.4 ad probability. Table 1 then uses V0.4 and its sibling classifiers to measure the detection accuracy of the SFT outputs and claims a 'substantial reduction in ad detection accuracy.' For V0.4 this is an evaluation of the same function used to construct the training data; the reported stealth improvement is a direct consequence of the training objective, not an independent demonstration that the advertisements are seamless to human readers.

1 more flagged steps
  1. other [Limitations section]
    "Additionally, the metric of ad detectability is grounded in classifier performance. However, human users may perceive ads differently, and what evades a model may still be obvious to a human reader."

    The paper itself identifies the construct gap: the only quantitative evidence for 'seamless integration' is classifier detection accuracy, and this is the same kind of classifier that provides the feedback in Eq. (1) and in best-of-N selection. Since no human-perception study or fluency/faithfulness metric is reported, the central claim reduces to 'the optimized outputs are less detectable by the detectors they were optimized against.' This limitation quote confirms that the evaluation does not establish human-level stealth, making the headline conclusion dependent on an unvalidated proxy.

full rationale

The paper's RQ2 claim, that SFT and best-of-N rewriting yield 'more seamless ad integration,' is supported almost entirely by Table 1, which reports the detection accuracy of in-house classifiers V0.0-V0.5. Those classifiers are not independent evaluators: Eq. (1) constructs the SFT target by taking argmin over H(y), and the BoN procedure selects the candidate with the lowest predicted ad probability from the same classifier family (V0.4). Hence the large drops in Table 1 for V0.4 are the optimization objective reported as a result; the drops for the sibling classifiers are at best evasion of the same detector family, not evidence about human perception. Even the zero-shot rewriter is prompted to 'Be hard to detect by current textual ad-detection systems,' so the entire generation side explicitly targets ad detectors from the start. The paper's Limitations section concedes exactly this: 'the metric of ad detectability is grounded in classifier performance. However, human users may perceive ads differently.' The external Touché evaluation (hidden organizer classifier plus manual review of 100 responses) provides some partial independence for the BoN submission, and the synthetic-data/curriculum-learning classifier training (RQ1) is an independent contribution, so this is not total circularity. Self-citations in the references ([3], [23], [44], [45]) are not load-bearing. Overall, the RQ2 'seamless integration' conclusion reduces in part to optimizing the same classifier metric that is then used to measure success, yielding a partial circularity score of 6.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central claims rest on assumptions that classifier probabilities measure real ad perceptibility, that LLM-generated synthetic data is representative, and that marketing-inspired prompts are useful for hard examples. Several hyperparameters are undisclosed, and the manual selection of data sources (Wikipedia infoboxes, Wikidata properties) adds unquantified bias.

free parameters (5)
  • N (best-of-N size) = 10
    Chosen for best-of-N sampling without an ablation; affects the stealth versus compute tradeoff.
  • Generation temperature = 0.5 and 1.0
    Two values tested; the paper notes lower temperature yields lower detectability but does not optimize or justify the choice.
  • Curriculum learning difficulty threshold = not disclosed
    Difficulty is estimated by V0.1 logits, but the threshold used to split easy/hard examples is not reported.
  • StructuredSynthetic upsampling factor = not disclosed
    Used in V0.5 to balance NaiveSynthetic and StructuredSynthetic data; the exact factor is not reported.
  • V0.4 classifier selection = V0.4
    Selected as the feedback classifier based on accuracy on zero-shot rewriter development outputs, which is a model-selection choice that may introduce bias.
assumptions (4)
  • domain assumption Automated classifier accuracy is a valid proxy for ad detectability by humans.
    State in Limitations: 'the metric of ad detectability is grounded in classifier performance,' yet human perception may differ. This is the foundation of the 'stealth' evaluation.
  • domain assumption LLM-generated synthetic data is representative of real user responses and advertisements.
    The paper calls for 'more rigorous validation' of synthetic data in Limitations and Future Direction, acknowledging this assumption is not proven.
  • domain assumption Marketing strategy distinctions (explicit/implicit, soft/hard sell) are useful for constructing training examples that improve classifier robustness.
    This motivates the StructuredSynthetic hard positive and negative generation, but the mapping from marketing categories to classifier robustness is assumed rather than tested.
  • domain assumption Wikipedia product entities and Wikidata properties sufficiently cover real-world ad-worthy products.
    The infobox and property selection is manual and not justified empirically; it may not generalize beyond the selected categories.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TeamCMU at Touch\'e: Adversarial Co-Evolution for Advertisement Integration and Detection in Conversational Search." pith.science (2026). https://pith.science/paper/5WFIMLJ5

@misc{pith2026250700509,
  author       = {Pith},
  title        = {Pith review of: TeamCMU at Touch\'e: Adversarial Co-Evolution for Advertisement Integration and Detection in Conversational Search},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5WFIMLJ5}},
  note         = {Machine review of arXiv:2507.00509}
}
read the original abstract

As conversational search engines increasingly adopt generation-based paradigms powered by Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG), the integration of advertisements into generated responses presents both commercial opportunities and challenges for user experience. Unlike traditional search, where advertisements are clearly delineated, generative systems blur the boundary between informational content and promotional material, raising concerns around transparency and trust. In this work, we propose a modular pipeline for advertisement management in RAG-based conversational systems, consisting of an ad-rewriter for seamless ad integration and a robust ad-classifier for detection. We leverage synthetic data to train high-performing classifiers, which are then used to guide two complementary ad-integration strategies: supervised fine-tuning of the ad-rewriter and a best-of-N sampling approach that selects the least detectable ad-integrated response among multiple candidates. Our evaluation focuses on two core questions: the effectiveness of ad classifiers in detecting diverse ad integration strategies, and the training methods that best support coherent, minimally intrusive ad insertion. Experimental results show that our ad-classifier, trained on synthetic advertisement data inspired by marketing strategies and enhanced through curriculum learning, achieves robust detection performance. Additionally, we demonstrate that classifier-guided optimization, through both fine-tuning and best-of-N sampling, significantly improves ad stealth, enabling more seamless integration. These findings contribute an adversarial co-evolution framework for developing more sophisticated ad-aware generative search systems and robust ad classifiers.

Figures

Figures reproduced from arXiv: 2507.00509 by the authors.

Figure 1
Figure 1. Overall Pipeline. A user query is first processed by the QA System to generate a base response. When an item is specified for advertisement, it is passed along with the base response to the Ad-Rewriter, which produces an ad-integrated version of the response. To further reduce ad detectability, we apply a best-of-𝑁 sampling strategy, selecting the rewritten response with the lowest ad probability as predicted by the… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

52 extracted references · 35 canonical work pages

  1. [1]

    Radlinski, N

    F. Radlinski, N. Craswell, A theoretical framework for conversational search, in: Proceedings of the 2017 conference on conference human information interaction and retrieval, 2017, pp. 117–126

    work page 2017

  2. [2]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W.-t. Yih, T. Rocktäschel, et al., Retrieval-augmented generation for knowledge-intensive nlp tasks, Advances in Neural Information Processing Systems 33 (2020) 9459–9474

    2020

  3. [3]

    T. E. Kim, A. Salemi, A. Drozdov, F. Diaz, H. Zamani, Retrieval-enhanced machine learning: Synthesis and opportunities, arXiv preprint arXiv:2407.12982 (2024)

    arXiv 2024

  4. [4]

    URL: https://www.perplexity

    Perplexity Team, Why we’re experimenting with advertising, 2024. URL: https://www.perplexity. ai/hub/blog/why-we-re-experimenting-with-advertising, accessed: 2025-04-30

    work page 2024

  5. [5]

    URL: https://help.openai.com/en/ articles/11146633-improved-shopping-results-from-chatgpt-search, accessed: 2025-04-30

    OpenAI, Improved shopping results from chatgpt search, 2025. URL: https://help.openai.com/en/ articles/11146633-improved-shopping-results-from-chatgpt-search, accessed: 2025-04-30

    work page arXiv 2025

  6. [6]

    Gleason, A

    J. Gleason, A. Koeninger, D. Hu, J. Teurn, Y. Bart, S. Knight, R. E. Robertson, C. Wilson, Search engine revenue from navigational and brand advertising, in: Proceedings of the International AAAI Conference on Web and Social Media, volume 18, 2024, pp. 488–501

    work page 2024

  7. [7]

    Zelch, M

    I. Zelch, M. Hagen, M. Potthast, A user study on the acceptance of native advertising in generative ir, in: Proceedings of the 2024 Conference on Human Information Interaction and Retrieval, CHIIR ’24, Association for Computing Machinery, New York, NY, USA, 2024, p. 142–152. URL: https://doi.org/10.1145/3627508.3638316. doi:10.1145/3627508.3638316

    arXiv 2024

  8. [8]

    Dubey, Z

    A. Dubey, Z. Feng, R. Kidambi, A. Mehta, D. Wang, Auctions with llm summaries, in: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, Association for Computing Machinery, New York, NY, USA, 2024, p. 713–722. URL: https://doi.org/ 10.1145/3637528.3672022. doi:10.1145/3637528.3672022

    arXiv 2024

Show all 52 references
  1. [9]

    Schmidt, I

    S. Schmidt, I. Zelch, J. Bevendorff, B. Stein, M. Hagen, M. Potthast, Detecting generated native ads in conversational search, in: Companion Proceedings of the ACM Web Conference 2024, WWW ’24, Association for Computing Machinery, New York, NY, USA, 2024, p. 722–725. URL: http...

    2024

  2. [10]

    Yi, Direct and indirect approaches to advertising persuasion: Which is more effective?, Journal of Business Research 20 (1990) 279–291

    Y. Yi, Direct and indirect approaches to advertising persuasion: Which is more effective?, Journal of Business Research 20 (1990) 279–291

    1990

  3. [11]

    Shapiro, H

    S. Shapiro, H. S. Krishnan, Memory-based measures for assessing advertising effects: A comparison of explicit and implicit memory effects, Journal of advertising 30 (2001) 1–13

    2001

  4. [12]

    Okazaki, B

    S. Okazaki, B. Mueller, C. R. Taylor, Measuring soft-sell versus hard-sell advertising appeals, Journal of Advertising 39 (2010) 5–20

    2010

  5. [13]

    E. L. Post, C. N. Sekharan, Comparative study and evaluation of online ad-blockers, in: 2015 2nd International Conference on Information Science and Security (ICISS), IEEE, 2015, pp. 1–4

    2015

  6. [14]

    Shiller, J

    B. Shiller, J. Waldfogel, J. Ryan, The effect of ad blocking on website traffic and quality, The RAND Journal of Economics 49 (2018) 43–63

    2018

  7. [15]

    Kiesel, Ç

    J. Kiesel, Ç. Çöltekin, M. Gohsen, S. Heineking, M. Heinrich, M. Fröbe, T. Hagen, M. Aliannejadi, T. Erjavec, M. Hagen, M. Kopp, N. Ljubešić, K. Meden, N. Mirzakhmedova, V. Morkevičius, H. Scells, I. Zelch, M. Potthast, B. Stein, Overview of Touché 2025: Argumentation Systems,...

    2025

  8. [16]

    Fröbe, M

    M. Fröbe, M. Wiegmann, N. Kolyada, B. Grahm, T. Elstner, F. Loebe, M. Hagen, B. Stein, M. Potthast, Continuous Integration for Reproducible Shared Tasks with TIRA.io, in: J. Kamps, L. Goeuriot, F. Crestani, M. Maistro, H. Joho, B. Davis, C. Gurrin, U. Kruschwitz, A. Caputo (Ed...

    2023

  9. [17]

    M. Seo, A. Kembhavi, A. Farhadi, H. Hajishirzi, Bidirectional attention flow for machine comprehension, in: International Conference on Learning Representations, 2017. URL: https: //openreview.net/forum?id=HJ0UKP9ge

    2017

  10. [18]

    Chen, Reading wikipedia to answer open-domain questions, arXiv preprint arXiv:1704.00051 (2017)

    D. Chen, Reading wikipedia to answer open-domain questions, arXiv preprint arXiv:1704.00051 (2017)

    2017 arXiv

  11. [19]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901

    2020

  12. [20]

    Kamalloo, N

    E. Kamalloo, N. Dziri, C. L. A. Clarke, D. Rafiei, Evaluating Open-Domain Question Answering in the Era of Large Language Models, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics, 2023

    2023

  13. [21]

    J. Li, X. Cheng, X. Zhao, J. Nie, J. Wen, HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023

    2023

  14. [22]

    Zamani, F

    H. Zamani, F. Diaz, M. Dehghani, D. Metzler, M. Bendersky, Retrieval-enhanced machine learning, in: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 2875–2886

    2022

  15. [23]

    F. Diaz, A. Drozdov, T. E. Kim, A. Salemi, H. Zamani, Retrieval-enhanced machine learning: Synthesis and opportunities, in: Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, 2024, ...

    2024

  16. [24]

    Petroni, A

    F. Petroni, A. Piktus, A. Fan, P. Lewis, M. Yazdani, N. De Cao, J. Thorne, Y. Jernite, V. Karpukhin, J. Maillard, V. Plachouras, T. Rocktäschel, S. Riedel, KILT: a benchmark for knowledge intensive language tasks, in: K. Toutanova, A. Rumshisky, L. Zettlemoyer, D. Hakkani-Tur,...

    2021

  17. [25]

    Bajaj, D

    P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, et al., Ms marco: A human generated machine reading comprehension dataset, arXiv preprint arXiv:1611.09268 (2016)

    2016 arXiv

  18. [26]

    Karpukhin, B

    V. Karpukhin, B. Oguz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, W.-t. Yih, Dense pas- sage retrieval for open-domain question answering, in: B. Webber, T. Cohn, Y. He, Y. Liu (Eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Pro- cessing (...

    2020 doi

  19. [27]

    Xiong, C

    L. Xiong, C. Xiong, Y. Li, K. Tang, J. Liu, P. N. Bennett, J. Ahmed, A. Overwijk, Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval, 2021

    2021

  20. [28]

    X. Li, Y. Zhou, Z. Dou, Unigen: A unified generative framework for retrieval and question answering with large language models, in: Proceedings of the AAAI Conference on Artificial Intelligence, 2024

    2024

  21. [29]

    D. Wang, Q. Huang, M. Jackson, J. Gao, Retrieve what you need: A mutual learning framework for open-domain question answering, Trans. Assoc. Comput. Linguistics (2024)

    2024

  22. [30]

    Y. Yin, G. Carenini, ARR: Question Answering with Large Language Models via Analyzing, Retrieving, and Reasoning, volume 2502.04689, 2025

    2025 arXiv

  23. [31]

    Joshi, E

    M. Joshi, E. Choi, D. S. Weld, L. Zettlemoyer, Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2017

    2017

  24. [32]

    Kwiatkowski, J

    T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, S. Petrov, Natural questions: A benchmark for question answering research,...

    2019

  25. [33]

    W. Sun, L. Yan, X. Ma, S. Wang, P. Ren, Z. Chen, D. Yin, Z. Ren, Is chatGPT good at search? investi- gating large language models as re-ranking agents, in: The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL: https://openreview.net/forum?id=3Q6LON8y2I

    2023

  26. [34]

    Hajiaghayi, S

    M. Hajiaghayi, S. Lahaie, K. Rezaei, S. Shin, Ad auctions for LLMs via retrieval augmented generation, in: The Thirty-eighth Annual Conference on Neural Information Processing Systems,

  27. [35]

    Soumalias, M

    E. Soumalias, M. J. Curry, S. Seuken, Truthful aggregation of llms with an application to online advertising, arXiv preprint arXiv:2405.05905 (2024)

    2024 arXiv

  28. [36]

    S. E. Robertson, H. Zaragoza, The probabilistic relevance framework: BM25 and beyond, Found. Trends Inf. Retr. (2009)

    2009

  29. [37]

    P. He, J. Gao, W. Chen, Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing, arXiv preprint arXiv:2111.09543 (2021)

    2021 arXiv

  30. [38]

    Stiennon, L

    N. Stiennon, L. Ouyang, J. Wu, D. M. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, P. Christiano, Learning to summarize from human feedback, in: Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Curran Associates Inc., Red...

    2020

  31. [39]

    Nakano, J

    R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, et al., Webgpt: Browser-assisted question-answering with human feedback, arXiv preprint arXiv:2112.09332 (2021)

    2021 arXiv

  32. [40]

    Bengio, J

    Y. Bengio, J. Louradour, R. Collobert, J. Weston, Curriculum learning, in: Proceedings of the 26th annual international conference on machine learning, 2009, pp. 41–48

    2009

  33. [41]

    A. Yang, B. Yang, B. Hui, B. Zheng, B. Yu, C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei, H. Lin, J. Tang, J. Wang, J. Yang, J. Tu, J. Zhang, J. Ma, J. Xu, J. Zhou, J. Bai, J. He, J. Lin, K. Dang, K. Lu, K. Chen, K. Yang, M. Li, M. Xue, N. Ni, P. Zhang, P. Wang, R. ...

    2024 arXiv

  34. [42]

    R. F. Woolson, Wilcoxon signed-rank test, Encyclopedia of biostatistics 8 (2005)

    2005

  35. [43]

    Mehrotra, J

    R. Mehrotra, J. McInerney, H. Bouchard, M. Lalmas, F. Diaz, Towards a fair marketplace: Counter- factual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems, in: Proceedings of the 27th acm international conference on information an...

    2018

  36. [44]

    Y. He, T. E. Kim, F. Diaz, J. Arguello, B. Mitra, Tip of the tongue query elicitation for simulated evaluation, arXiv preprint arXiv:2502.17776 (2025)

    2025 arXiv

  37. [45]

    well formed answer

    T. E. Kim, F. Diaz, Towards fair rag: On the impact of fair ranking in retrieval-augmented generation, arXiv preprint arXiv:2409.11598 (2024). A. Computational Cost All inference experiments, except for ablations on models larger than 7B parameters, were conducted on a single ...

    2024 arXiv

  38. [47]

    Think about how the advertised product/service genuinely relates to the user’s needs implied by the query

    Contextual Relevance is Key: The advertisement must feel relevant to the search query and the existing response text. Think about how the advertised product/service genuinely relates to the user’s needs implied by the query

  39. [48]

    Avoid overly promotional or sales-oriented phrasing

    Subtle and Natural Language: Use language that blends seamlessly with the original response. Avoid overly promotional or sales-oriented phrasing. Focus on informative and helpful language

  40. [49]

    Focus on Benefits, Not Just Features: Instead of directly listing qualities, rephrase them as benefits the user might gain in relation to the search query and response context

  41. [50]

    This might involve: * Briefly modifying an existing sentence to subtly incorporate the advertisement

    Strategic Placement: Choose the most natural and impactful location(s) within the response to subtly introduce the advertisement. This might involve: * Briefly modifying an existing sentence to subtly incorporate the advertisement. * Adding a short, relevant phrase or clause t...

  42. [51]

    The advertisement should enhance, not distort, the original information

    Maintain Original Meaning: Ensure the core meaning and factual accuracy of the original response remain unchanged. The advertisement should enhance, not distort, the original information

  43. [52]

    If it feels forced or obvious, refine your approach

    Review for Subtlety: Before returning the response, critically evaluate if the advertisement feels genuinely subtle and integrated. If it feels forced or obvious, refine your approach. Output: Return **only** the modified response with the subtly integrated advertisement. Sear...

  44. [2024]

    URL: https://openreview.net/forum?id=Ujo8V7iXmR

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.