Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)

Andreas Konstantin Kruff; Christin Katharina Kreutz; Krisztian Balog; Philipp Schaer; Timo Breuer

REVIEW 2 major objections 5 minor 16 references

Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This workshop proposal argues that user simulators in information access can be validated in shared tasks that score simulated interactions against held-out real-user data, and that doing so on a small scale is the right path to a full…

desk verdict A well-scoped workshop proposal whose one real flaw is internal: Task B's synthetic conversational data cannot validate simulators against real users as §4.1 claims, and the paper deserves revision rather than rejection. read the letter →

arxiv 2505.11687 v1 pith:U2PAKEOU submitted 2025-05-16 cs.IR cs.AIcs.HC

Philipp Schaer , Christin Katharina Kreutz , Krisztian Balog , Timo Breuer , Andreas Konstantin Kruff This is my paper · ORCID

classification cs.IRcs.AIcs.HC

keywords InformationAccessSimulationUserModelsEvaluationSharedTasksConversationalSearchInteractiveRetrievalLLM-based

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper, a proposal for the second iteration of a workshop on simulations for information access, aims to turn user simulation from a promising idea into a testable evaluation method. It defines two micro shared tasks: one in which participants build a simulator of search-session interactions with ranked result lists, and one in which they predict the next user utterance in a conversational session. Submitted simulator configurations are run in a packaged environment and scored against held-out data from real or synthetic users. If the tasks work, the organizers will use what they learn about validation measures and pitfalls to design a full shared-task campaign at a major evaluation venue.

What carries the argument

The central object is a packaged development-and-evaluation loop: a dockerized environment containing a search system, a baseline user simulator, and data, so that participants change only the simulator configuration and its prompt design. The validation mechanism is held-out data—Task A uses real-user search sessions, Task B uses synthetic conversational exchanges derived from search logs—and the fidelity measures, which the workshop will define. Two properties carry the argument: participants need not build infrastructure, and organizers score against data the participants never see, so the comparison is a fidelity test rather than a system-effectiveness test.

What would settle it

The proposal would fail its own test if the shared tasks run and no submitted configuration scores closer to the held-out real-user data than the provided baseline does, or if organizers cannot agree on a measure because every candidate measure ranks the same configurations differently.

Watch

Extended reading notes

Core claim

The paper's central claim is that user simulators can be validated, not just built, and that a shared-task format is a workable way to do that validation. In Task A, participants design a simulator of a search session whose queries, result pages, and clicks come from longitudinal real-user interaction data; in Task B, they predict the next user utterance given a conversational history built from synthetic search logs. Submitted configurations are run in a packaged environment with a baseline simulator and then scored by held-out data. The scoring measures are deliberately not fixed by the paper; they are part of the workshop's agenda. If the two micro tasks succeed, the organizers take the lessons into a full shared-task campaign.

Load-bearing premise

The load-bearing premise is that comparisons between simulated and held-out real-user behavior, made with simple measures, can tell which simulator is more faithful, even though the paper leaves the choice of those measures entirely open to workshop discussion.

Editorial extensions

If this is right

If the micro shared tasks yield usable fidelity scores, the organizers can carry a concrete shared-task design into a full evaluation campaign.
Participants can enter through a packaged environment with a baseline simulator, so effort goes into modeling user behavior rather than system engineering.
LLM-based prompt designs become the primary experimental variable for both query formulation and conversational utterance prediction.
The workshop's outcomes—configurations and working notes—are shared openly, giving the community a reusable starting point rather than one-off results.
The evaluation protocol itself is part of the study: because the measures are left open, the workshop is also an experiment in how to measure simulation fidelity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper leaves implicit that a validated search-session simulator could eventually substitute for live-user studies in interactive retrieval evaluation, lowering cost and improving reproducibility.
Because Task A uses organic user data and Task B uses synthetic data, comparing the two tasks may reveal whether conversational simulators trained on synthetic logs transfer to real user behavior.
A natural extension would be a meta-task in which submitted configurations are scored by whether they generalize across unseen topics and result pages, testing robustness rather than a single held-out split.
The open choice of evaluation measures is both an enabler and a risk: the workshop may spend as much effort agreeing on what fidelity means as on producing leaderboards.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a workshop proposal, not a scientific result, and the one thing worth remembering is that the evaluation design has a genuine internal contradiction. Section 4.1 states the guiding principle: validate user simulations against held-out data collected from real users. Task A fits that principle—LongEval/CORE supplies real queries, SERPs, and clicks. Task B does not. Section 4.3 says the conversational data is "generated synthetically based on traditional search logs" and evaluation is semantic similarity to the "actual" utterances. Those actual utterances are synthetic, so no human ground truth exists. You can use Task B to compare simulators with each other or with the generative process that produced the data, but you cannot claim to validate fidelity to real users without an external human-utterance set. The stress-test note is right about this.

What the paper does well: it is a coherent, honest proposal for a community-building event. The micro shared tasks are concrete, the timeline is realistic, and the decision to package SimIIR, PyTerrier, and baseline simulators in a Docker environment is considerate of participants. Delegating the choice of evaluation measures to workshop discussion is reasonable at this stage, not a flaw. The related-work coverage is solid, and the self-citations are legitimate—Sim4IA 2024 and SimIIR 3 are the natural foundations.

Soft spots beyond Task B: there is no scientific result, but that is inherent to a workshop proposal, not a defect. The "simple measures" are unspecified, but the authors flag that intentionally and invite discussion. One minor issue: Figure 1's "held-out validation environment" is drawn generically and does not disclose that Task B's held-out data is synthetic, so the figure and §4.1 overstate the consistency of the two tasks.

Who this is for: IR evaluation researchers, shared-task organizers, and anyone planning a TREC/CLEF simulation track. The community value is real—there is a genuine gap in venues for user-simulation evaluation.

Recommendation: this deserves a serious referee, not a desk reject. A referee should ask the authors to either add a real-user conversational utterance set for Task B or explicitly re-scope Task B as a system-comparison pilot, and adjust §4.1 and Figure 1 accordingly. With that revision, it is a solid workshop proposal.

Referee Report

2 major / 5 minor

Summary. The paper is the workshop proposal for the second Sim4IA workshop at SIGIR 2025. Its central aim is to test two micro shared tasks for user simulation—Task A on interactions with ranked lists and Task B on conversational utterances—as a stepping stone toward a future TREC/CLEF campaign. The proposal describes the design principle of validating user simulators against held-out real-user data rather than measuring system effectiveness, the use of the SimIIR 3 framework with LLM-based user actions, the timeline for dataset release and submission, and the planned interactive workshop format (keynote, panel, lightning talks, breakout groups). The paper also summarizes the motivation from the previous Sim4IA 2024 workshop and the broader literature on simulation-based evaluation.

Significance. If the proposed design were fully realized, the workshop would provide a tangible infrastructure for community-wide validation of user simulators, addressing a recognized gap in simulation-based IR evaluation. The commitment to distributing a Dockerized SimIIR environment with baseline configurations lowers the barrier to entry, and the use of LongEval/CORE interaction data for Task A grounds one task in real user behavior. The paper is honest about open design questions and positions the workshop as a discussion forum, which is appropriate for a workshop proposal. However, as detailed below, the inconsistency between the stated validation principle and Task B's synthetic ground truth undermines the significance of one of the two tasks, and the underspecified evaluation measures leave the 'validation' deliverable incompletely defined.

major comments (2)

[§4.1 and §4.3] The manuscript's central validation claim is contradicted by the Task B data description. §4.1 states that 'The submitted user simulators are evaluated on the basis of held-out data collected from real users,' and the stated design principle is 'validating user simulations instead of measuring system effectiveness.' However, §4.3 specifies that the conversational data for Task B 'is generated synthetically based on traditional search logs,' so no real-user utterance exists as the 'actual user utterance' against which semantic similarity is computed. Semantic similarity to synthetic utterances can measure agreement with the generative process that produced the test data, but it cannot establish that a simulator 'mimic[s] the interactions of real users with a high degree of fidelity.' The paper should either add a real-user conversational dataset (e.g., from TREC iKAT or a similar resource) for Task B evaluation, or explicitly reframe Task B as a system-comparison pilot whose results are not evidence about real-user fidelity. As written, the paper delivers on its stated validation goal for only one of its two tasks.
[§4.1 and Table 1] The evaluation protocol is underspecified in a way that affects the feasibility of the shared tasks within the stated timeline. §4.1 says submitted simulators will be evaluated with 'simple measures' but explicitly defers the choice of measures to workshop discussion, even though Table 1 shows submissions due 27 June and the workshop on 17 July. It is unclear how the organizers can produce the promised evaluation—which presumably feeds the lightning talks and breakout discussions—if the measures are not fixed before submissions are due. Please specify at least a provisional set of candidate measures (e.g., click-through alignment, ranking correlation, utterance-level semantic similarity) or describe a pre-registration procedure, even if the final measures are to be discussed at the workshop. Without this, the micro shared tasks cannot operate as the validation exercises the paper claims them to be.

minor comments (5)

[Table 1] The day '13rd June' should be '13th June'.
[ACM Reference Format] 'InProceedings' lacks a space; it should read 'In Proceedings'.
[§3] The text refers to 'SimIIR 3.02' while the citation in [2] is titled 'SimIIR 3'; please align the version string.
[§4.3] Consider stating a tentative semantic-similarity measure (e.g., embedding cosine or BERTScore) so participants have a concrete target for their submissions.
[§4.2] The phrase 'so that simulations can be based on previous queries and interactions with the result lists' leaves unclear whether document-level content will be provided with the SERP data; specifying the exact data fields would improve reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular reasoning found; the workshop proposal makes no derivational claims that reduce to their own inputs.

full rationale

This paper is a workshop proposal, not a derivation or an experimental study. It contains no equations, no fitted parameters, and no predicted quantities that could be equivalent to inputs by construction. The central claims are organizational: to run two micro shared tasks, to discuss simulator evaluation, and to plan a future TREC/CLEF campaign. The paper relies on prior work, including the authors' own SimIIR framework and the FnTIR monograph on user simulation, but these citations are contextual background rather than load-bearing evidence for a derived result. The workshop does not assert that SimIIR is validated by the proposed tasks; it merely plans to use the framework as a packaged development environment. No uniqueness theorem is invoked, no ansatz is smuggled in via citation, and no known result is renamed as a new contribution. The reviewer's noted issue that Task B uses synthetically generated conversational data while Section 4.1 promises evaluation against 'held-out data collected from real users' is a genuine design inconsistency and a correctness risk, but it is not a circularity: the synthetic data is not defined in terms of the submitted simulators, and the proposed semantic-similarity evaluation does not make the simulator's output equal to its own input. Similarly, the fact that evaluation measures are left unspecified is a completeness concern, not a circularity. Because the paper makes no prediction that reduces to a fit or to a self-citation chain, the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The proposal rests on assumptions about the value of simulation, the availability and suitability of specific datasets, and the adequacy of synthetic data for a conversational task. No free parameters are fit to data because no empirical analysis is performed.

assumptions (3)

domain assumption User simulation can provide valid evaluations of information access systems.
The entire premise of the workshop relies on the claim that simulations are key contributors to IA evaluation, stated in the Motivation section.
domain assumption The LongEval 2025 dataset based on CORE interaction data will be available and suitable for Task A.
Section 4.2 states Task A will use this dataset, but its final availability and quality are not yet established in the paper.
domain assumption Synthetic conversational data for Task B is a reasonable proxy for real user behavior.
Section 4.3 says the conversational data is generated synthetically; the paper assumes this is sufficient for evaluating simulators, though real-world validation may differ.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)." pith.science (2026). https://pith.science/paper/U2PAKEOU

@misc{pith2026250511687,
  author       = {Pith},
  title        = {Pith review of: Second SIGIR Workshop on Simulations for Information Access (Sim4IA 2025)},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U2PAKEOU}},
  note         = {Machine review of arXiv:2505.11687}
}

read the original abstract

Simulations in information access (IA) have recently gained interest, as shown by various tutorials and workshops around that topic. Simulations can be key contributors to central IA research and evaluation questions, especially around interactive settings when real users are unavailable, or their participation is impossible due to ethical reasons. In addition, simulations in IA can help contribute to a better understanding of users, reduce complexity of evaluation experiments, and improve reproducibility. Building on recent developments in methods and toolkits, the second iteration of our Sim4IA workshop aims to again bring together researchers and practitioners to form an interactive and engaging forum for discussions on the future perspectives of the field. An additional aim is to plan an upcoming TREC/CLEF campaign.

Figures

Figures reproduced from arXiv: 2505.11687 by the authors.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 9 canonical work pages

[1]

Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffery Dal- ton, and Leif Azzopardi. 2024. TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview. arXiv:2401.01330 [cs.IR]

arXiv 2024
[2]

Leif Azzopardi, Timo Breuer, Björn Engelmann, Christin Kreutz, Sean MacAvaney, David Maxwell, Andrew Parry, Adam Roegiest, Xi Wang, and Saber Zerhoudi

work page
[3]

Leif Azzopardi, Kalervo Järvelin, Jaap Kamps, and Mark D. Smucker. 2010. Report on the SIGIR 2010 workshop on the simulation of interaction. SIGIR Forum 44, 2 (2010), 35–47. https://doi.org/10.1145/1924475.1924484

work page arXiv 2010
[4]

Krisztian Balog, David Maxwell, Paul Thomas, and Shuo Zhang. 2021. Report on the 1st simulation for information retrieval workshop (Sim4IR 2021) at SIGIR 2021. SIGIR Forum 55, 2 (2021), 10:1–10:16. https://doi.org/10.1145/3527546.3527559

work page arXiv 2021
[5]

Krisztian Balog and ChengXiang Zhai. 2023. Tutorial on User Simulation for Eval- uating Information Access Systems. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023 , Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhan...

arXiv 2023
[6]

Krisztian Balog and ChengXiang Zhai. 2024. User Simulation for Evaluating Information Access Systems. Foundations and Trends in Information Retrieval 18, 1-2 (2024), 1–261. https://doi.org/10.1561/1500000098

doi:10.1561/1500000098 2024
[7]

Timo Breuer, Christin Katharina Kreutz, Norbert Fuhr, Krisztian Balog, Philipp Schaer, Nolwenn Bernard, Ingo Frommholz, Marcel Gohsen, Kaixin Ji, Gareth J. F. Jones, Jüri Keller, Jiqun Liu, Martin Mladenov, Gabriella Pasi, Johanne Trip- pas, Xi Wang, Saber Zerhoudi, and ChengXiang Zhai. 2024. Report on the Workshop on Simulations for Information Access (S...

work page Pith review arXiv 2024
[8]

Matteo Cancellieri, Alaa El-Ebshihy, Tobias Fink, Petra Galuščáková, Gabriela Gonzalez-Saez, Lorraine Goeuriot, David Iommi, Jüri Keller, Petr Knoth, Philippe Mulhem, Florina Piroi, David Pride, and Philipp Schaer. 2025. LongEval at CLEF 2025: Longitudinal Evaluation of IR Model Performance. In Advances in Information Retrieval: 47th European Conference o...

work page 2025

Show all 16 references

[9]

Peter Ingwersen and Kalervo Järvelin. 2005. The turn - integration of information seeking and retrieval in context . Springer, Dordrecht

2005
[10]

Rolf Jagerman, Krisztian Balog, Philipp Schaer, Johann Schaible, Narges Tavakolpoursaleh, and Maarten de Rijke. 2017. Overview of TREC OpenSearch

2017
[11]

Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval Systems with Users. Foundations and Trends in Information Retrieval 3, 1—2 (2009), 1–224. https://doi.org/10.1561/1500000012

2009 doi
[12]

Johannes Kiesel, Çağrı Çöltekin, Maximilian Heinrich, Maik Fröbe, Milad Al- shomary, Bertrand De Longueville, Tomaž Erjavec, Nicolas Handke, Matyáš Kopp, Nikola Ljubešić, Katja Meden, Nailia Mirzhakhmedova, Vaidas Morke- vičius, Theresa Reitis-Münstermann, Mario Scharfbillig, ...

2024
[13]

Gustavo Penha, Arthur Câmara, and Claudia Hauff. 2022. Evaluating the Ro- bustness of Retrieval Pipelines with Query Variation Generators. In Advances in Information Retrieval - 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10-14, 2022, Proceedin...

2022
[14]

Philipp Schaer, Christin Katharina Kreutz, Krisztian Balog, Timo Breuer, and Norbert Fuhr. 2024. SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024). In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Re...

2024
[2017]

500-324), Ellen M

In Proceedings of The Twenty-Sixth Text REtrieval Conference, TREC 2017, Gaithersburg, Maryland, USA, November 15-17, 2017 (NIST Special Publication, Vol. 500-324), Ellen M. Voorhees and Angela Ellis (Eds.). National Institute of Standards and Technology (NIST). https://trec.n...

2017
[2024]

In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (Tokyo, Japan) (SIGIR-AP 2024)

SimIIR 3: A Framework for the Simulation of Interactive and Conversational Information Retrieval. In Proceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development in Information Retrieval in the Asia Pacific Region (Tokyo, Japan) (SIGIR-AP 2024)...

2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Mohammad Aliannejadi, Zahra Abbasiantaeb, Shubham Chatterjee, Jeffery Dal- ton, and Leif Azzopardi. 2024. TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview. arXiv:2401.01330 [cs.IR]

arXiv 2024

[2] [2]

Leif Azzopardi, Timo Breuer, Björn Engelmann, Christin Kreutz, Sean MacAvaney, David Maxwell, Andrew Parry, Adam Roegiest, Xi Wang, and Saber Zerhoudi

work page

[3] [3]

Leif Azzopardi, Kalervo Järvelin, Jaap Kamps, and Mark D. Smucker. 2010. Report on the SIGIR 2010 workshop on the simulation of interaction. SIGIR Forum 44, 2 (2010), 35–47. https://doi.org/10.1145/1924475.1924484

work page arXiv 2010

[4] [4]

Krisztian Balog, David Maxwell, Paul Thomas, and Shuo Zhang. 2021. Report on the 1st simulation for information retrieval workshop (Sim4IR 2021) at SIGIR 2021. SIGIR Forum 55, 2 (2021), 10:1–10:16. https://doi.org/10.1145/3527546.3527559

work page arXiv 2021

[5] [5]

Krisztian Balog and ChengXiang Zhai. 2023. Tutorial on User Simulation for Eval- uating Information Access Systems. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, CIKM 2023, Birmingham, United Kingdom, October 21-25, 2023 , Ingo Frommholz, Frank Hopfgartner, Mark Lee, Michael Oakes, Mounia Lalmas, Min Zhan...

arXiv 2023

[6] [6]

Krisztian Balog and ChengXiang Zhai. 2024. User Simulation for Evaluating Information Access Systems. Foundations and Trends in Information Retrieval 18, 1-2 (2024), 1–261. https://doi.org/10.1561/1500000098

doi:10.1561/1500000098 2024

[7] [7]

Timo Breuer, Christin Katharina Kreutz, Norbert Fuhr, Krisztian Balog, Philipp Schaer, Nolwenn Bernard, Ingo Frommholz, Marcel Gohsen, Kaixin Ji, Gareth J. F. Jones, Jüri Keller, Jiqun Liu, Martin Mladenov, Gabriella Pasi, Johanne Trip- pas, Xi Wang, Saber Zerhoudi, and ChengXiang Zhai. 2024. Report on the Workshop on Simulations for Information Access (S...

work page Pith review arXiv 2024

[8] [8]

Matteo Cancellieri, Alaa El-Ebshihy, Tobias Fink, Petra Galuščáková, Gabriela Gonzalez-Saez, Lorraine Goeuriot, David Iommi, Jüri Keller, Petr Knoth, Philippe Mulhem, Florina Piroi, David Pride, and Philipp Schaer. 2025. LongEval at CLEF 2025: Longitudinal Evaluation of IR Model Performance. In Advances in Information Retrieval: 47th European Conference o...

work page 2025