Pith. sign in

REVIEW 3 major objections 5 minor 24 references

Evaluating Contrastive Feedback for Effective User Simulations

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLM-based simulated search users become more effective when the prompt contains summaries of both relevant and irrelevant documents they have already judged, compared with positive-only, negative-only, or no feedback.

desk verdict Clean but overreaching short study of contrastive prompt feedback for LLM user simulators; worth refereeing if the conclusion is scaled back. read the letter →

arxiv 2505.02560 v2 pith:2KPXUS6M submitted 2025-05-05 cs.IR

Andreas Konstantin Kruff , Timo Breuer , Philipp Schaer This is my paper · ORCID
classification cs.IR
keywords interactiveinformationretrievalusersimulationcontrastivefeedbackpromptengineeringlargelanguagemodelsrelevancesessionevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether showing an LLM-based simulated user—an agent that mimics a human searcher—summaries of both relevant and irrelevant documents during an interactive search session makes it find more relevant documents than simpler feedback schemes. Eight user configurations are compared on two TREC newswire collections, varying whether the prompt carries full topic information, only a title, positive feedback, negative feedback, contrastive feedback, or random relevance decisions. On the first collection the contrastive user is the top performer, and on the second it ties with the positive-only user; all feedback variants outperform topic-only baselines after the first few queries. The paper reads this as evidence that contrastive examples help the LLM internalize the distinction between desired and undesired documents, in line with contrastive-training results from fine-tuning. If correct, the finding gives interactive-retrieval evaluation a way to build more realistic simulated users without relying on explicit topic narratives.

What carries the argument

The carrying mechanism is the contrastive relevance feedback (CRF) prompt: a prompt that appends two summary blocks—one compressing previously judged relevant documents and one compressing previously judged irrelevant documents—to the topic context before asking the LLM to generate the next query or judge the next document. The summaries are refreshed after every relevance judgment, so the model's knowledge state evolves over the session. This mechanism is what lets the contrastive-training idea act at the prompting stage rather than the weight-update stage, and the experimental design isolates it by keeping the feedback type as the main difference between user configurations.

What would settle it

A replication run that fixes a single prompt template across all conditions—varying only whether the injected summary block lists relevant documents, irrelevant documents, both, or neither—would settle the claim; if the contrastive advantage disappears or reverses, the reported effect comes from template wording rather than from contrastive feedback.

Watch

Extended reading notes

Core claim

The central hypothesis is that the contrastive-learning principle—training a model on both positive and negative examples—carries over from fine-tuning to prompt engineering for user simulation. In the strongest configuration, every query-generation and relevance-judgment step receives a running summary of documents the simulated user has judged relevant plus a separate summary of documents judged irrelevant, updated after each judgment, alongside the full TREC topic context. On the Core17 collection this contrastive relevance feedback user is the most effective by both effort-effect information gain and session-discounted cumulative gain, while on Core18 it matches the positive-only user, which stays marginally ahead. The authors conclude that LLMs benefit from providing contrastive examples in the prompt throughout interactive retrieval simulations, and they note that a title-only contrastive variant performs comparably well to full-topic, no-feedback users, suggesting session context can partly replace detailed topic descriptions.

Load-bearing premise

The load-bearing premise is that the feedback type is the only difference between user configurations, since the exact prompt templates and summary formatting are not shown in the paper but left in a code repository.

Editorial extensions

If this is right

  • Simulated users should include session feedback summaries, not just static topic descriptions, when the goal is to emulate an evolving information need.
  • Contrastive feedback's advantage emerges only after several queries, so simulation evaluations need sessions long enough to capture the uptake phase.
  • A title-only contrastive variant outperforms full-topic, no-feedback users, so detailed topic narratives may not be necessary when the simulator accumulates session context.
  • The Core18 result, where positive-only feedback matches contrastive feedback, shows the benefit of negative examples is dataset-dependent and should not be assumed universal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: vary summary granularity (single sentence vs. paragraph, extractive vs. abstractive) to confirm the contrast set itself, rather than extra text, drives the gain.
  • The cross-collection difference suggests a hypothesis the authors leave open: negative examples matter most when judged-relevant documents are sparse or when many retrieved documents resemble the target without being relevant; comparing relevance-label density across collections would test it.
  • Because the prompt templates are only available in a code repository, a replication that fixes all template wording except the feedback block is needed before the effect can be attributed to contrastivity rather than incidental phrasing.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper evaluates whether providing contrastive relevance feedback — summaries of both judged-relevant and judged-irrelevant documents — in the prompt improves the effectiveness of LLM-based simulated users in interactive information retrieval. Using the SimIIR 3 framework, BM25 retrieval, and a Llama3.3 model, the authors compare eight user configurations on TREC Core17 and Core18, measuring Effort-Effect information gain and sDCG. They report that the contrastive user (CRF) is the top performer on Core17 but is on par with the positive-only user (PRF) on Core18, yet the conclusion states that LLMs benefit from contrastive examples in the prompt throughout interactive retrieval simulations.

Significance. If the central claim were robust, the paper would provide a useful prompt-engineering recipe for LLM-based user simulation and would connect contrastive training intuitions to in-context interaction. The work has clear strengths: the experimental setup is described in reasonable detail, the code and results are publicly available, the authors compare against random and topic-only baselines, and they are explicitly honest about the mixed Core18 result. However, the current evidence is not sufficient to support the stated conclusion. The contrastive advantage is observed on only one of two collections, and the experimental design fixes a single random seed with no repeated runs or significance testing, so the Core17 advantage could be sampling noise. The paper is therefore a useful preliminary study rather than a demonstrated result, and the conclusion needs to be substantially softened or the experiments strengthened.

major comments (3)
  1. [Section 4 and Section 5] The central claim that "LLMs benefit from providing contrastive examples in the prompt throughout interactive retrieval simulations" is not supported by the paper's own reported results. In Section 4, the authors state that on Core18, "CRF is on par with the PRF user, which remains the leading performer overall" and that "there is obviously no benefit to including summaries of irrelevant documents in the prompt." Since PRF is the strongest non-contrastive feedback condition, the contrastive advantage is observed only on Core17. The conclusion in Section 5 overstates this evidence and should be revised to reflect the collection-dependent nature of the finding.
  2. [Section 3.3] The experimental design fixes a single random seed (default 0.0) and uses temperature 1.0 for query generation, so each user configuration produces one stochastic trajectory per test collection. No variance estimates, confidence intervals, or significance tests are reported. Consequently, the observed Core17 advantage of CRF over PRF could be due to sampling noise, and the Core18 tie offers no support for the contrastive mechanism. The authors should run multiple seeds or repeated sessions and report variance, or at minimum frame the results as descriptive without claiming a reliable benefit.
  3. [Section 3.2] The prompt templates and summary-generation logic are only available in the GitHub repository and are not described in the paper. Since the entire comparison is between prompting strategies, differences in wording, summary formatting, or instruction phrasing between conditions could confound the observed effectiveness differences with the feedback-modality variable. The paper should include the actual templates or a detailed description of their structure in an appendix, even in compressed form, so that readers can verify that the conditions differ only in whether relevant, irrelevant, or both summaries are included.
minor comments (5)
  1. [Introduction] The phrase "the realism of the user simulation when proving the LLM with the explicitly formulated topic descriptions" appears to be a typo for "providing the LLM."
  2. [Related Work] "determinstic rules" should be spelled "deterministic rules."
  3. [Related Work] "theIR community" is missing a space and should read "the IR community."
  4. [Section 3.1] "While the CRF use can access the entire topic's contents" should read "While the CRF user can access".
  5. [Section 3.4, Eq. (1)] The notation in Eq. (1) is unclear: "rel_d" should be explicitly defined as the relevance value of a document d, and it would help to state the units or scale of the information gain contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central result is an external empirical comparison of prompting strategies against TREC benchmarks, not a derivation from fitted inputs or self-citations.

full rationale

The paper's central claim is that LLM-based simulated users benefit from contrastive relevance feedback in the prompt. This claim is supported by an experimental comparison across two external TREC collections (Core17 and Core18), with several user configurations differing in how feedback is presented. The evaluation measures in Equation (1) define search effectiveness directly from logged relevance judgments and are used only to score the resulting sessions; no parameter is fitted to the outcome and the CRF configuration is not constructed from the effectiveness measure. The prompting templates are external artifacts in a public repository, so any concern about incidental prompt wording is an experimental-validity issue, not a circularity issue. The paper cites several prior works by the same authors, including SimIIR 3 and context-driven query simulation, but these are used as implementation infrastructure or related work, not as evidence for the effectiveness advantage of contrastive prompting. The conclusion is directly supported by the observed comparison on Core17, and the paper itself reports the weaker result on Core18, so the analysis does not redefine or assume what it claims to test. No circular step, fitted-input-called-prediction, or load-bearing self-citation chain was found.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

The central claim rests entirely on standard IR domain assumptions: TREC topics and qrels as ground truth, the LLM as a stand-in for a user, BM25 as the retrieval function, and the summarization pipeline as a faithful compression of the user's feedback. No new physical or mathematical constants are introduced, and no numbers are fitted to data.

assumptions (5)
  • domain assumption TREC topic title, description, and narrative are a sufficient representation of a user's initial knowledge state.
    Section 3.1 defines FTTC and TTT users based on these topic sections; the simulation treats them as the knowledge state from which the user starts.
  • domain assumption The LLM's relevance judgments are a valid proxy for a real user's relevance decisions.
    Section 3.3 uses the LLM with temperature 0 to judge documents; the paper itself cites the ongoing debate on LLM relevance judgments in Section 2.
  • domain assumption BM25 is an adequate ranking function for evaluating simulated user interactions.
    Section 3.3 states BM25 was employed 'due to its efficiency' and that more effective rankers were not tried.
  • domain assumption The information gain and sDCG measures capture session-level search effectiveness.
    Section 3.4 defines these measures and uses them as the sole outcome metrics.
  • domain assumption The iterative document summarization in the repository preserves the information needed for feedback without introducing systematic bias.
    Section 3.2 says summarization was performed iteratively, but the exact procedure is only in the code repository.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Contrastive Feedback for Effective User Simulations." pith.science (2026). https://pith.science/paper/2KPXUS6M

@misc{pith2026250502560,
  author       = {Pith},
  title        = {Pith review of: Evaluating Contrastive Feedback for Effective User Simulations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2KPXUS6M}},
  note         = {Machine review of arXiv:2505.02560}
}
read the original abstract

The use of Large Language Models (LLMs) for simulating user behavior in the domain of Interactive Information Retrieval has recently gained significant popularity. However, their application and capabilities remain highly debated and understudied. This study explores whether the underlying principles of contrastive training techniques, which have been effective for fine-tuning LLMs, can also be applied beneficially in the area of prompt engineering for user simulations. Previous research has shown that LLMs possess comprehensive world knowledge, which can be leveraged to provide accurate estimates of relevant documents. This study attempts to simulate a knowledge state by enhancing the model with additional implicit contextual information gained during the simulation. This approach enables the model to refine the scope of desired documents further. The primary objective of this study is to analyze how different modalities of contextual information influence the effectiveness of user simulations. Various user configurations were tested, where models are provided with summaries of already judged relevant, irrelevant, or both types of documents in a contrastive manner. The focus of this study is the assessment of the impact of the prompting techniques on the simulated user agent performance. We hereby lay the foundations for leveraging LLMs as part of more realistic simulated users.

Figures

Figures reproduced from arXiv: 2505.02560 by the authors.

Figure 1
Figure 1. A flowchart of the analyzed user configuration described in Section 3.1. The color codes correspond to those in Figure 2 [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Results of the simulated sessions for different [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

24 extracted references · 6 canonical work pages

  1. [2]

    Voorhees

    James Allan, Donna Harman, Evangelos Kanoulas, Dan Li, Christophe Van Gysel, and Ellen M. Voorhees. 2017. TREC 2017 Common Core Track Overview. In TREC (NIST Special Publication, Vol. 500–324) . National Institute of Standards and Technology (NIST)

    work page 2017

  2. [3]

    Leif Azzopardi, Timo Breuer, Björn Engelmann, Christin Kreutz, Sean MacAvaney, David Maxwell, Andrew Parry, Adam Roegiest, Xi Wang, and Saber Zerhoudi

  3. [4]

    Krisztian Balog and ChengXiang Zhai. 2024. User Simulation for Evaluating Information Access Systems. Found. Trends Inf. Retr. 18, 1-2 (2024), 1–261. https: //doi.org/10.1561/1500000098

    doi:10.1561/1500000098 2024

  4. [5]

    Nicholas J. Belkin. 1980. Anomalous States of Knowledge as a Basis for Informa- tion Retrieval. Canadian Journal of Information Science 5 (1980), 133–143

    work page 1980

  5. [6]

    Timo Breuer. 2024. Data Fusion of Synthetic Query Variants With Generative Large Language Models. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region, SIGIR-AP 2024, Tokyo, Japan, December 9-12, 2024 , Tetsuya Sakai, Emi Ishita, Hiroaki Ohshima, Faegheh Hasibi...

    arXiv 2024

  6. [7]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2021. MS MARCO: Benchmarking Ranking Models in the Large-Data Regime. In SIGIR ’21: The 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, Virtual Event, Canada, July 11-15, 2021 , Fernando Diaz, Chirag Shah, Torsten Suel, Pablo Castells, R...

    arXiv 2021

  7. [8]

    Björn Engelmann, Timo Breuer, Jana Isabelle Friese, Philipp Schaer, and Norbert Fuhr. 2024. Context-Driven Interactive Query Simulations Based on Generative Large Language Models. arXiv:2312.09631 [cs.IR] https://arxiv.org/abs/2312. 09631

    arXiv 2024

  8. [9]

    Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. 2021. Su- pervised Contrastive Learning for Pre-trained Language Model Fine-tuning. arXiv:2011.01403 [cs.CL] https://arxiv.org/abs/2011.01403

    arXiv 2021

Show all 24 references
  1. [10]

    Price, Lois M

    Kalervo Järvelin, Susan L. Price, Lois M. L. Delcambre, and Marianne Lykke Nielsen. 2008. Discounted Cumulated Gain Based Evaluation of Multiple-Query IR Sessions. In Advances in Information Retrieval , 30th European Conference on IR Research, ECIR 2008, Glasgow, UK, March 30-...

    2008 doi

  2. [11]

    Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. 2021. PyTerrier: Declarative Experimentation in Python from BM25 to Dense Re- trieval. In Proceedings of the 30th ACM International Conference on Informa- tion & Knowledge Management (Virtual Event, Queensland...

    2021

  3. [12]

    David Maxwell and Leif Azzopardi. 2016. Simulating Interactive Information Retrieval: SimIIR: A Framework for the Simulation of Interaction. 1141–1144. https://doi.org/10.1145/2911451.2911469

    2016

  4. [13]

    David Maxwell, Leif Azzopardi, Kalervo Järvelin, and Heikki Keskustalo. 2015. Searching and Stopping: An Analysis of Stopping Rules and Strategies. InProceed- ings of the 24th ACM International on Conference on Information and Knowledge Management (Melbourne, Australia) (CIKM ...

    2015

  5. [14]

    Teemu Pääkkönen, Jaana Kekäläinen, Heikki Keskustalo, Leif Azzopardi, David Maxwell, and Kalervo Järvelin. 2017. Validating simulated interaction for retrieval evaluation. Inf. Retr. 20, 4 (Aug. 2017), 338–362. https://doi.org/10.1007/s10791- 017-9301-2

    2017 doi

  6. [15]

    Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L

    Hossein A. Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles L. A. Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Fag- gioli. 2024. LLMJudge: LLMs for Relevance Judgments. arXiv:2408.08896 [cs.IR] https://arxiv.org/abs/2408.08896

    2024 arXiv

  7. [16]

    Ian Soboroff. 2025. Don’t Use LLMs to Make Relevance Judgments. Information Retrieval Research 1, 1 (Mar. 2025), 29–46. https://doi.org/10.54195/irrj.19625

    2025 doi

  8. [17]

    Weiwei Sun, Zhengliang Shi, Shen Gao, Pengjie Ren, Maarten de Rijke, and Zhaochun Ren. 2022. Contrastive Learning Reduces Hallucination in Conversa- tions. arXiv:2212.10400 [cs.CL] https://arxiv.org/abs/2212.10400

    2022 arXiv

  9. [18]

    Voorhees, and Tetsuya Sakai

    Rikiya Takehi, Ellen M. Voorhees, and Tetsuya Sakai. 2024. LLM- Assisted Relevance Assessments: When Should We Ask LLMs for Help? arXiv:2411.06877 [cs.IR] https://arxiv.org/abs/2411.06877

    2024 arXiv

  10. [19]

    Voorhees and Angela Ellis (Eds.)

    Ellen M. Voorhees and Angela Ellis (Eds.). 2018.Proceedings of the Twenty-Seventh Text REtrieval Conference, TREC 2018, Gaithersburg, Maryland, USA, November 14- 16, 2018. NIST Special Publication, Vol. 500–331. National Institute of Standards and Technology (NIST). https://tr...

    2018

  11. [20]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations . https://openreview.net/forum?id=WE_vluYUL-X

    2023

  12. [21]

    Erhan Zhang, Xingzhu Wang, Peiyuan Gong, Yankai Lin, and Jiaxin Mao

  13. [22]

    Jing Zhang, Hui Gao, Peng Zhang, Boda Feng, Wenmin Deng, and Yuexian Hou

  14. [23]

    arXiv:2403.09142 [cs.IR] https://arxiv.org/abs/2403.09142

    USimAgent: Large Language Models for Simulating Search Users. arXiv:2403.09142 [cs.IR] https://arxiv.org/abs/2403.09142

    arXiv

  15. [25]

    LA-UCL: LLM-Augmented Unsupervised Contrastive Learning Framework for Few-Shot Text Classification. InProceedings of the 2024 Joint International Con- ference on Computational Linguistics, Language Resources and Evaluation (LREC- COLING 2024), Nicoletta Calzolari, Min-Yen Kan,...

    2024

  16. [26]

    Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, and Yu-Xiong Wang. 2024. Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models. arXiv:2310.04406 [cs.AI] https://arxiv.org/abs/2310.04406

    2024 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.