SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions

Saber Zerhoudi

arxiv: 2604.27878 · v1 · submitted 2026-04-30 · 💻 cs.IR

SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions

Saber Zerhoudi This is my paper

Pith reviewed 2026-05-07 07:01 UTC · model grok-4.3

classification 💻 cs.IR

keywords user simulatorsinteractive information retrievalevaluation toolkitbehavioral realismtester reliabilitysession embeddingsFréchet distancesystem ranking validity

0 comments

The pith

Human-likeness tests for user simulators show almost no correlation with whether they produce valid system rankings, while session-distance metrics do.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces SimEval-IR to treat behavioral realism and tester reliability as distinct goals for user simulators in interactive search. It supplies a unified session schema, dataset adapters, and three benchmarks that measure realism, reliability via RATE-style estimation, and their relationship. Across four real datasets and four simulator families the dominant classifier-based human-likeness test shows essentially zero pooled correlation with ranking validity. In contrast, marginal click-depth distance and Fréchet distance on session embeddings yield correlations of roughly 0.4. The toolkit and all reproduction scripts are released so others can apply the same separation of concerns.

Core claim

The classifier-discriminator human-likeness check has essentially no pooled predictive power for system-ranking validity (r=+0.09, n=48), while marginal click-depth distance and Fréchet distance over session embeddings give a much stronger signal (|r|=0.43 and 0.40, p≤0.005).

What carries the argument

The SimEval-IR benchmark suite that enforces a canonical session schema, runs separate realism and RATE-style reliability benchmarks, and correlates the two via embedding distances.

If this is right

Evaluations of user simulators must report ranking validity separately from behavioral realism metrics.
Session-level distance measures such as click-depth and Fréchet embedding distance supply usable signals for ranking validity.
Future simulator design can target the distance metrics directly rather than optimizing only for classifier discriminability.
The released toolkit allows direct comparison of new simulators against the same four datasets and RATE procedure.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same separation of realism and reliability benchmarks could be applied to conversational search or recommendation simulators.
If distance metrics remain stronger predictors, simulator training objectives might shift toward minimizing embedding distances to real sessions.
Requiring only ranking-validity benchmarks would change how papers report simulator quality in interactive IR.
Extending the schema to multi-turn conversational logs would test whether the same correlation patterns hold outside classic search sessions.

Load-bearing premise

The RATE-style estimation procedure correctly captures tester reliability and the four chosen real-world datasets plus four simulator families are representative enough for the pooled correlation findings to generalize.

What would settle it

Re-running the full correlation analysis on an independent collection of real user sessions and simulator families and checking whether the near-zero correlation for the classifier human-likeness test persists.

Figures

Figures reproduced from arXiv: 2604.27878 by Saber Zerhoudi.

**Figure 1.** Figure 1: The same simulator can appear realistic under be view at source ↗

**Figure 3.** Figure 3: Canonical session schema unifying session search and conversational interactions as ordered event sequences with typed events view at source ↗

**Figure 4.** Figure 4: Classifier-discriminator interpretation: the main view at source ↗

**Figure 5.** Figure 5: Realism does not equal reliability. Each panel plots view at source ↗

read the original abstract

User simulators are increasingly central to interactive information retrieval, yet the community lacks standardized evaluation tools. Simulators serve two objectives, behavioral realism (matching real user behavior) and tester reliability (producing valid system rankings), and these are often conflated despite being distinct and sometimes conflicting. We present SimEval-IR, an open-source toolkit and benchmark suite that makes this distinction measurable. SimEval-IR provides: (1) a canonical session schema unifying session search and conversational interactions, with validated dataset adapters and explicit loss accounting; (2) three executable benchmarks covering behavioral realism, tester reliability with RATE-style estimation, and an analysis linking the two; and (3) baseline results across four real datasets in two languages and four simulator families. Our key finding: the classifier-discriminator ''human-likeness'' check, the dominant realism test in the literature, has essentially no pooled predictive power for system-ranking validity ($r{=}{+}0.09$, $n{=}48$), while marginal click-depth distance and Fr\'{e}chet distance over session embeddings give a much stronger signal ($|r|{=}0.43$ and $0.40$, $p{\leq}0.005$). SimEval-IR is released with all configurations and scripts to reproduce the reported analysis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

SimEval-IR gives a reusable toolkit that separates behavioral realism from tester reliability and shows human-likeness checks have almost no link to valid system rankings, though the result rests on a narrow set of datasets and simulators.

read the letter

The main thing to take from this paper is that SimEval-IR supplies an open toolkit with a canonical session schema, dataset adapters, and three linked benchmarks that let people measure two goals separately: how closely simulated sessions match real ones, and whether the simulator produces the same system orderings that real users would. The headline empirical result is that the usual classifier-based human-likeness test shows essentially no pooled correlation with ranking validity (r = +0.09), while simpler distance measures on click depth and session embeddings do better (|r| around 0.40–0.43). They release the code and configs, which is the right move for a methods paper like this. The work is grounded in four real datasets across two languages and four simulator families, and it reports concrete numbers rather than just claiming a new framework. That combination of schema, benchmarks, and the specific correlation finding is what is actually new here. The authors are careful not to overclaim the distinction between the two objectives, and the release of reproduction scripts makes the numbers checkable. The soft spot is the limited sample. Four datasets and four simulator families leave open the possibility that the low correlation for human-likeness is tied to this particular collection rather than a general fact; if the simulators miss important behavioral regimes or if the RATE-style estimator has unstated assumptions about session independence, the pooled r values could shift. The abstract also omits error bars, exact exclusion rules, and power calculations, so the full paper needs to show those details clearly before the result can be treated as robust. This is aimed at people already working on interactive IR simulators who need standardized evaluation tools. A reader who wants to move beyond ad-hoc human-likeness tests will find the benchmarks and code useful even if they end up disagreeing with the exact numbers. It is worth sending to peer review because the toolkit is concrete and the empirical separation is worth testing on wider data, but referees should focus on generalizability and the statistical details of the RATE procedure.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces SimEval-IR, an open-source toolkit and benchmark suite for evaluating user simulators in interactive information retrieval. It provides a canonical session schema unifying search and conversational interactions, three executable benchmarks (behavioral realism, tester reliability via RATE-style estimation, and their linkage), and baseline results across four real datasets (two languages) and four simulator families. The central empirical claim is that the dominant classifier-discriminator human-likeness metric has essentially no pooled predictive power for system-ranking validity (r=+0.09, n=48), whereas marginal click-depth distance and Fréchet distance over session embeddings yield stronger signals (|r|=0.43 and 0.40, p≤0.005).

Significance. If the reported correlations are robust, the work has clear significance for the IR community: it supplies the first standardized, reproducible toolkit that explicitly separates behavioral realism from tester reliability, supplies concrete evidence against over-reliance on discriminator-based checks, and releases all configurations and scripts. This directly supports better simulator development and evaluation practices. The open toolkit and baseline results across multiple datasets are concrete strengths that lower barriers to adoption and future comparisons.

major comments (2)

[Results section (pooled correlation analysis)] Results section (pooled correlation analysis): The claim that human-likeness has 'essentially no pooled predictive power' rests on r=+0.09 with n=48. It is unclear whether the pooling treats the 48 units as independent or accounts for clustering by dataset or simulator family; without this, the effective sample size and uncertainty around the coefficient are unknown, directly affecting the strength of the central claim.
[Experimental design (datasets and simulators)] Experimental design (datasets and simulators): The pooled findings rely on only four real-world datasets and four simulator families. No sensitivity analysis, bootstrap over subsets, or explicit discussion of between-setup variance is described; if the low correlation is driven by this narrow sample rather than a general property, the conclusion that click-depth and Fréchet distances are reliably superior would not generalize.

minor comments (2)

[Abstract] Abstract: The correlation values use nonstandard LaTeX spacing ($r{=}{+}0.09$); conventional notation (r = +0.09) would improve readability without changing meaning.
[Toolkit release description] Toolkit release description: The canonical schema and loss-accounting mechanism are introduced, but the manuscript would benefit from a short table or pseudocode showing how loss is computed for session-search versus conversational cases to aid immediate use by readers.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their constructive and detailed feedback. We address each major comment below and have revised the manuscript to incorporate additional statistical analyses and sensitivity checks that strengthen the robustness of our central claims.

read point-by-point responses

Referee: Results section (pooled correlation analysis): The claim that human-likeness has 'essentially no pooled predictive power' rests on r=+0.09 with n=48. It is unclear whether the pooling treats the 48 units as independent or accounts for clustering by dataset or simulator family; without this, the effective sample size and uncertainty around the coefficient are unknown, directly affecting the strength of the central claim.

Authors: We thank the referee for this important statistical point. The reported Pearson r = +0.09 was computed across the 48 simulator–dataset observations, treating each as an independent unit. To account for potential clustering by dataset and simulator family, we have re-analyzed the data with a linear mixed-effects model that includes random intercepts for both factors. The fixed-effect coefficient for human-likeness remains near zero (β = 0.07, p = 0.61), while the coefficients for marginal click-depth distance and Fréchet distance stay significant (β = 0.41 and 0.38, p < 0.005). Cluster-robust standard errors were also computed. These results have been added to the Results section, with full model specifications now provided in the appendix. revision: yes
Referee: Experimental design (datasets and simulators): The pooled findings rely on only four real-world datasets and four simulator families. No sensitivity analysis, bootstrap over subsets, or explicit discussion of between-setup variance is described; if the low correlation is driven by this narrow sample rather than a general property, the conclusion that click-depth and Fréchet distances are reliably superior would not generalize.

Authors: We acknowledge that the evaluation is limited to four datasets (covering two languages) and four simulator families. In the revised manuscript we have added a bootstrap sensitivity analysis (1,000 resamples of the 48 observations) and a leave-one-dataset-out analysis. The bootstrap 95% CI for the human-likeness correlation is [−0.13, 0.28], which includes zero and excludes the intervals for the other two metrics. Leave-one-dataset-out correlations for human-likeness remain |r| < 0.14 in every subset, while click-depth and Fréchet distances retain |r| > 0.35. We have also expanded the discussion section to quantify between-setup variance and to emphasize that SimEval-IR is released as an extensible toolkit precisely to support future additions of datasets and simulators. These changes directly address the generalizability concern. revision: yes

Circularity Check

0 steps flagged

No significant circularity in the empirical derivation chain

full rationale

The paper's central claim is an empirical result: the classifier-discriminator human-likeness metric shows near-zero pooled correlation (r=+0.09, n=48) with system-ranking validity as measured by RATE-style estimation, while click-depth and Fréchet distances show stronger signals. This correlation is computed directly from running the proposed benchmarks on four external real-world datasets (two languages) and four independent simulator families. No equations reduce by construction to their inputs, no fitted parameters are relabeled as predictions, and no load-bearing self-citations or uniqueness theorems are invoked. The analysis is self-contained against external benchmarks and data, making the reported r values falsifiable outside the paper's own definitions.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based solely on the abstract, the work introduces no free parameters, new mathematical entities, or ad-hoc axioms beyond standard statistical assumptions for correlation analysis; it builds on existing datasets and simulators.

axioms (1)

standard math Standard assumptions of Pearson correlation and associated p-value calculations
Invoked when reporting r values and statistical significance for the predictive power of different realism metrics.

pith-pipeline@v0.9.0 · 5524 in / 1463 out tokens · 55714 ms · 2026-05-07T07:01:33.841289+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

28 extracted references · 28 canonical work pages

[1]

AOL. 2006. AOL Search Query Log. https://web.archive.org/web/20061103185436/ http://research.aol.com/. Anonymized web search query log released for research

work page arXiv 2006
[2]

Leif Azzopardi, Timo Breuer, Björn Engelmann, Christin Kreutz, Sean MacA- vaney, David Maxwell, Andrew Parry, Adam Roegiest, Xi Wang, and Saber Zerhoudi. 2024. SimIIR 3: A framework for the simulation of interactive and con- versational information retrieval. InProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development i...

work page 2024
[3]

Krisztian Balog, Nolwenn Bernard, Saber Zerhoudi, and ChengXiang Zhai. 2025. Theory and Toolkits for User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluation. (2025), 4138– 4141

work page 2025
[4]

Ben Carterette, Mike Hall, et al. 2016. Evaluating Retrieval over Sessions: The TREC Session Track 2011–2014. InProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

work page 2016
[5]

Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. InProceedings of the 18th international conference on SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions World wide web. 1–10

work page 2009
[6]

Jia Chen, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019. TianGong- ST: A new dataset with large-scale refined real-world web search sessions. In Proceedings of the 28th ACM International Conference on Information and Knowl- edge Management. 2485–2488

work page 2019
[7]

Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Biller- beck. 2020. Orcas: 18 million clicked query-document pairs for analyzing search. InProceedings of the 29th ACM International Conference on Information & Knowl- edge Management. 2983–2989

work page 2020
[8]

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An ex- perimental comparison of click position-bias models. InProceedings of the 2008 international conference on web search and data mining. 87–94

work page 2008
[9]

Jeffrey Dalton, Chenyan Xiong, et al. 2020. TREC Conversational Assistance Track (CAsT). InProceedings of the Twenty-Ninth Text REtrieval Conference (TREC 2020)

work page 2020
[10]

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel two-sample test.The journal of machine learning research13, 1 (2012), 723–773

work page 2012
[11]

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems30 (2017)

work page 2017
[12]

Andreas Konstantin Kruff, Christin Katharina Kreutz, Timo Breuer, Philipp Schaer, and Krisztian Balog. 2026. Sim4ia-bench: a user simulation benchmark suite for next query and utterance prediction. InEuropean Conference on Infor- mation Retrieval. Springer, 594–609

work page 2026
[13]

Sahiti Labhishetty and ChengXiang Zhai. 2021. An Exploration of Tester-based Evaluation of User Simulators for Comparing Interactive Retrieval Systems. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1598–1602. doi:10.1145/3404835.3463091

work page doi:10.1145/3404835.3463091 2021
[14]

Sahiti Labhishetty and ChengXiang Zhai. 2022. RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators. InAdvances in Infor- mation Retrieval: 44th European Conference on IR Research (ECIR 2022), Pro- ceedings (Lecture Notes in Computer Science, Vol. 13185). Springer, 336–350. doi:10.1007/978-3-030-99736-6_23

work page doi:10.1007/978-3-030-99736-6_23 2022
[15]

Bill Yuchen Lin, Yuxiang Deng, Khyathi Raghavi Chandu, Alisa Liu, et al. 2025. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild.International Conference on Learning Representations (ICLR)(2025). To appear

work page 2025
[16]

LMSYS. 2024. LMSYS-Chat-1M: A Large-Scale Multi-Model Conversational Dataset. https://huggingface.co/datasets/lmsys/chat-1m

work page 2024
[17]

Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2022. Streamlining Eval- uation with ir_measures. InAdvances in Information Retrieval: 44th European Conference on IR Research (ECIR 2022), Proceedings (Lecture Notes in Computer Science). Springer

work page 2022
[18]

Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. 2021. PyTerrier: Declarative experimentation in Python from BM25 to dense retrieval. InProceedings of the 30th acm international conference on information & knowledge management. 4526–4533

work page 2021
[19]

NIST. 2018. trec_eval – TREC Evaluation Software. https://trec.nist.gov/trec_ eval/

work page 2018
[20]

Andrew Parry, Maik Fröbe, Harrisen Scells, Ferdinand Schlatt, Guglielmo Fag- gioli, Saber Zerhoudi, Sean MacAvaney, and Eugene Yang. 2025. Variations in relevance judgments and the shelf life of test collections. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3387–3397

work page 2025
[21]

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 3982–3992

work page 2019
[22]

Navid Rekabsaz, Markus Zlabinger, and Allan Hanbury. 2021. TripClick: A Large Health Web Search Click Dataset. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

work page 2021
[23]

Emine Yilmaz, Javed A Aslam, and Stephen Robertson. 2008. A new rank cor- relation coefficient for information retrieval. InProceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 587–594

work page 2008
[24]

Saber Zerhoudi and Michael Granitzer. 2024. Beyond Conventional Metrics: Assessing User Simulators in Information Retrieval. InProceedings of the Italian Information Retrieval Workshop (IIR 2024) (CEUR Workshop Proceedings)

work page 2024
[25]

Saber Zerhoudi, Michael Granitzer, Jörg Schlötterer, and Christin Seifert. 2021. Query change as a contextual Markov model for simulating user search behaviour. InProceedings of the 13th Annual Meeting of the Forum for Information Retrieval Evaluation. 43–51

work page 2021
[26]

Saber Zerhoudi, Sebastian Günther, Kim Plassmeier, Timo Borst, Christin Seifert, Matthias Hagen, and Michael Granitzer. 2022. The SimIIR 2.0 Framework: User Types, Markov Model-Based Interaction Simulation, and Advanced Query Gen- eration. InProceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM 2022). 4661–4666

work page 2022
[27]

Saber Zerhoudi, Christin Seifert, and Michael Granitzer. 2022. Evaluating Simu- lated User Interaction and Search Behaviour in Digital Libraries. InProceedings of the 44th European Conference on Information Retrieval (ECIR 2022)

work page 2022
[28]

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. Miracl: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics11 (2023), 1114–1131

work page 2023

[1] [1]

AOL. 2006. AOL Search Query Log. https://web.archive.org/web/20061103185436/ http://research.aol.com/. Anonymized web search query log released for research

work page arXiv 2006

[2] [2]

Leif Azzopardi, Timo Breuer, Björn Engelmann, Christin Kreutz, Sean MacA- vaney, David Maxwell, Andrew Parry, Adam Roegiest, Xi Wang, and Saber Zerhoudi. 2024. SimIIR 3: A framework for the simulation of interactive and con- versational information retrieval. InProceedings of the 2024 Annual International ACM SIGIR Conference on Research and Development i...

work page 2024

[3] [3]

Krisztian Balog, Nolwenn Bernard, Saber Zerhoudi, and ChengXiang Zhai. 2025. Theory and Toolkits for User Simulation in the Era of Generative AI: User Modeling, Synthetic Data Generation, and System Evaluation. (2025), 4138– 4141

work page 2025

[4] [4]

Ben Carterette, Mike Hall, et al. 2016. Evaluating Retrieval over Sessions: The TREC Session Track 2011–2014. InProceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval

work page 2016

[5] [5]

Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. InProceedings of the 18th international conference on SimEval-IR: A Unified Toolkit and Benchmark Suite for Evaluating User Simulators and Search Sessions World wide web. 1–10

work page 2009

[6] [6]

Jia Chen, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019. TianGong- ST: A new dataset with large-scale refined real-world web search sessions. In Proceedings of the 28th ACM International Conference on Information and Knowl- edge Management. 2485–2488

work page 2019

[7] [7]

Nick Craswell, Daniel Campos, Bhaskar Mitra, Emine Yilmaz, and Bodo Biller- beck. 2020. Orcas: 18 million clicked query-document pairs for analyzing search. InProceedings of the 29th ACM International Conference on Information & Knowl- edge Management. 2983–2989

work page 2020

[8] [8]

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An ex- perimental comparison of click position-bias models. InProceedings of the 2008 international conference on web search and data mining. 87–94

work page 2008

[9] [9]

Jeffrey Dalton, Chenyan Xiong, et al. 2020. TREC Conversational Assistance Track (CAsT). InProceedings of the Twenty-Ninth Text REtrieval Conference (TREC 2020)

work page 2020

[10] [10]

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. 2012. A kernel two-sample test.The journal of machine learning research13, 1 (2012), 723–773

work page 2012

[11] [11]

Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. Gans trained by a two time-scale update rule converge to a local nash equilibrium.Advances in neural information processing systems30 (2017)

work page 2017

[12] [12]

Andreas Konstantin Kruff, Christin Katharina Kreutz, Timo Breuer, Philipp Schaer, and Krisztian Balog. 2026. Sim4ia-bench: a user simulation benchmark suite for next query and utterance prediction. InEuropean Conference on Infor- mation Retrieval. Springer, 594–609

work page 2026

[13] [13]

Sahiti Labhishetty and ChengXiang Zhai. 2021. An Exploration of Tester-based Evaluation of User Simulators for Comparing Interactive Retrieval Systems. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval. 1598–1602. doi:10.1145/3404835.3463091

work page doi:10.1145/3404835.3463091 2021

[14] [14]

Sahiti Labhishetty and ChengXiang Zhai. 2022. RATE: A Reliability-Aware Tester-Based Evaluation Framework of User Simulators. InAdvances in Infor- mation Retrieval: 44th European Conference on IR Research (ECIR 2022), Pro- ceedings (Lecture Notes in Computer Science, Vol. 13185). Springer, 336–350. doi:10.1007/978-3-030-99736-6_23

work page doi:10.1007/978-3-030-99736-6_23 2022

[15] [15]

Bill Yuchen Lin, Yuxiang Deng, Khyathi Raghavi Chandu, Alisa Liu, et al. 2025. WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild.International Conference on Learning Representations (ICLR)(2025). To appear

work page 2025

[16] [16]

LMSYS. 2024. LMSYS-Chat-1M: A Large-Scale Multi-Model Conversational Dataset. https://huggingface.co/datasets/lmsys/chat-1m

work page 2024

[17] [17]

Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2022. Streamlining Eval- uation with ir_measures. InAdvances in Information Retrieval: 44th European Conference on IR Research (ECIR 2022), Proceedings (Lecture Notes in Computer Science). Springer

work page 2022

[18] [18]

Craig Macdonald, Nicola Tonellotto, Sean MacAvaney, and Iadh Ounis. 2021. PyTerrier: Declarative experimentation in Python from BM25 to dense retrieval. InProceedings of the 30th acm international conference on information & knowledge management. 4526–4533

work page 2021

[19] [19]

NIST. 2018. trec_eval – TREC Evaluation Software. https://trec.nist.gov/trec_ eval/

work page 2018

[20] [20]

Andrew Parry, Maik Fröbe, Harrisen Scells, Ferdinand Schlatt, Guglielmo Fag- gioli, Saber Zerhoudi, Sean MacAvaney, and Eugene Yang. 2025. Variations in relevance judgments and the shelf life of test collections. InProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval. 3387–3397

work page 2025

[21] [21]

Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 3982–3992

work page 2019

[22] [22]

Navid Rekabsaz, Markus Zlabinger, and Allan Hanbury. 2021. TripClick: A Large Health Web Search Click Dataset. InProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval

work page 2021

[23] [23]

Emine Yilmaz, Javed A Aslam, and Stephen Robertson. 2008. A new rank cor- relation coefficient for information retrieval. InProceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. 587–594

work page 2008

[24] [24]

Saber Zerhoudi and Michael Granitzer. 2024. Beyond Conventional Metrics: Assessing User Simulators in Information Retrieval. InProceedings of the Italian Information Retrieval Workshop (IIR 2024) (CEUR Workshop Proceedings)

work page 2024

[25] [25]

Saber Zerhoudi, Michael Granitzer, Jörg Schlötterer, and Christin Seifert. 2021. Query change as a contextual Markov model for simulating user search behaviour. InProceedings of the 13th Annual Meeting of the Forum for Information Retrieval Evaluation. 43–51

work page 2021

[26] [26]

Saber Zerhoudi, Sebastian Günther, Kim Plassmeier, Timo Borst, Christin Seifert, Matthias Hagen, and Michael Granitzer. 2022. The SimIIR 2.0 Framework: User Types, Markov Model-Based Interaction Simulation, and Advanced Query Gen- eration. InProceedings of the 31st ACM International Conference on Information and Knowledge Management (CIKM 2022). 4661–4666

work page 2022

[27] [27]

Saber Zerhoudi, Christin Seifert, and Michael Granitzer. 2022. Evaluating Simu- lated User Interaction and Search Behaviour in Digital Libraries. InProceedings of the 44th European Conference on Information Retrieval (ECIR 2022)

work page 2022

[28] [28]

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. Miracl: A multilingual retrieval dataset covering 18 diverse languages. Transactions of the Association for Computational Linguistics11 (2023), 1114–1131

work page 2023