Pith. sign in

REVIEW 5 major objections 3 minor 65 references

Query–document relevance is linearly decodable from instruction-tuned LLM activations, with the signal strongest in middle-to-late layers, and probes trained on those activations can preserve system rankings as well as or better than the mo

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 22:58 UTC pith:JEY7GGT4

load-bearing objection A careful, honest probing study showing relevance is linearly decodable from LLM residual streams in a depth-dependent way, with the main caveat that the readout position (final token) is an assumption and the probe-vs-generation comparison is supervised versus zero-shot. the 5 major comments →

arxiv 2607.15555 v1 pith:JEY7GGT4 submitted 2026-07-17 cs.IR

LLMs Encode Relevance as a Layer-Wise Cross-Lingual Signal

Pietro Bernardelle , Samaneh Mohtadi , Stefano Civelli , Joel Mackenzie , Gianluca Demartini This is my paper
classification cs.IR
keywords relevance assessmentlinear probingresidual streamlayer-wise analysislarge language modelsinformation retrieval evaluationmultilingual transferinterpretability
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that query–document relevance is not just an output an LLM produces, but an internal representation that can be read out with a simple linear model. The authors extract the last-token residual-stream activation at every layer of six instruction-tuned LLMs during a relevance-judgment forward pass, then train linear probes on those activations. They find that relevance is weakly decodable in early layers and strongest in middle-to-late layers, across model families and under graded and binary relevance definitions. In several models these probes agree with human judgments as well as or better than the model's own generated labels, and they preserve the ranking of retrieval systems more faithfully. The same signal transfers partially across seven languages, suggesting a shared but incomplete multilingual relevance structure.

Core claim

The central discovery is that relevance is encoded as a depth-dependent linear signal in the residual stream: a probe reading the final-token state at middle-to-late layers recovers human relevance labels, while early layers carry little such signal. Furthermore, validation-selected probes in several models achieve higher agreement with human judgments and better preserve official system rankings than the model's own generated relevance labels, revealing a representation–expression gap: the model internally encodes relevance evidence that it fails to write down faithfully. Cross-lingual probes show partial transfer, indicating the representation is partly language-agnostic while still benefi

What carries the argument

The load-bearing mechanism is layer-wise linear probing of the residual stream. For each transformer layer, a regularized linear probe (logistic regression for binary labels, ordinal logistic regression for graded labels) is trained on the activation of the final input token immediately before answer generation — the position that has attended to the full query, document, and relevance instruction. Training probes independently at each layer yields a depth profile of relevance decodability, with the probing layer chosen by validation performance and an oracle test-selected layer as an upper-bound diagnostic.

Load-bearing premise

The load-bearing premise is that the final prompt token's activation, immediately before answer generation, is a faithful snapshot of the model's query–document relevance representation; if the relevant state is distributed across other token positions or only assembled during generation, the layer-wise profile may be an artifact of where the authors chose to read.

What would settle it

Train the same linear probes on activations extracted at a different token position — for example the first document token or a mid-prompt token — rather than the final prompt token. If the middle-to-late layer advantage disappears or reverses, the depth-dependent claim would reflect the readout position rather than an internal relevance representation. A complementary test: remove the relevance-judgment instruction while keeping query and document; if probes still decode relevance at the same depths, the signal is intrinsic to the query–document pair rather than an artifact of the judging pro

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Layer-wise probing can serve as a diagnostic for where relevance emerges inside LLM judges, distinguishing models that internally encode relevance from those that only produce surface-level labels.
  • Probe-derived pseudo-labels can support IR evaluation: in several models they preserve human-judgment system rankings better than the model's own generated labels, offering a representation-level evaluation signal.
  • The representation–expression gap implies that a model's generated relevance judgment is not a direct readout of its internal estimate; output calibration methods may need to address generation-stage transformations.
  • Partial cross-lingual portability of the relevance signal suggests internal relevance representations share structure across languages, which could support lower-resource or multilingual evaluation, though native-language supervision remains stronger.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If relevance is linearly readable at middle layers, one could build a cheap relevance classifier that reads a single layer's activation instead of running full generation, potentially giving large speedups for large-scale judging or reranking.
  • The layer-wise profile suggests a 'relevance region' in the network; an editorial extension is that activation steering or patching within those layers might correct output bias in LLM judges more directly than prompt engineering.
  • A testable extension beyond the paper: concatenating activations from several middle layers into the probe input may exceed any single-layer probe, indicating distributed encoding of relevance.
  • If the linear-decodability finding generalizes to larger and retrieval-specialized models, it could enable training-free relevance estimation directly from activations of off-the-shelf LLMs.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 3 minor

Summary. This paper asks whether query-document relevance is linearly decodable from the residual-stream activations of instruction-tuned LLMs, how such internal probes compare to generated relevance judgments, and whether the signal transfers across languages. The authors use TREC DL20 and a seven-language MIRACL sample, run UMBRELA-style prompts through six 4-9B models, extract last-token activations at every layer, train ridge-regularized linear (or ordinal-logistic) probes under grouped query splits, and compare validation-selected probes with parsed generation outputs at both label and system-ranking levels. They report a depth-dependent decodability profile, cases where probes outperform generated judgments, and partial cross-lingual portability.

Significance. The paper addresses a timely question in IR evaluation and interpretability. Its strengths are the grouped query-level splits, validation-based selection of both regularization and layer with an explicit oracle upper-bound, deterministic decoding, six models from multiple families, and released code. The layer-wise decodability result, if restricted to the specific readout, is a useful diagnostic. However, the headline RQ2 comparison is between a supervised linear probe and zero-shot generated judgments, and the central readout-position assumption is untested; these issues currently make the stronger claims conditional.

major comments (5)
  1. [Section 4.2, Table 2] The probe-versus-inference comparison is asymmetric. The probing pipeline is supervised on human relevance labels, including the label distribution and grade thresholds, while the generated-judgment baseline is zero-shot instruction following. Showing that a validation-selected linear probe sometimes has higher Cohen's kappa than generated labels establishes that supervised readouts can beat zero-shot outputs, but it does not establish a 'representation-expression gap' or that relevance evidence is 'not faithfully expressed' in the final label (Section 5). A minimal control would be to fit the same supervised calibration to the generated labels (e.g., a validation-fitted mapping of output scores/labels) or to compare probe performance against a zero-shot probe that is not trained on human labels.
  2. [Section 3.4] The entire layer-wise profile and the RQ2 comparison depend on the choice of the final input token as the probe readout. The paper asserts that this position 'represents the model state from which the final relevance judgment is generated,' but it does not test this against alternative positions. Because relevance could be distributed across query/document tokens or crystallize during decoding, the reported depth profile and probe-vs-output gap may be artifacts of the readout position. The Section 5 scope note acknowledges final-token activations as a limitation, but the Conclusions still phrase the findings as properties of how the LLM encodes relevance. Please add a position ablation (e.g., mean-pooled document/query representations or first generated-token activations) or explicitly reframe the claims as being about final-token residual streams.
  3. [Section 3.3, Table 2] It is unclear whether the inference-output baseline is evaluated on all q-d pairs or only on the subset that parsed successfully. Table 1 shows parsing success ranging from 87.8% to 100%, so dropping unparsed cases would change both the evaluation set and the label distribution. If probe and inference are scored on different instances, part of the observed gap could be an artifact. The manuscript should state the scoring denominator and rerun the comparison on the common subset of successfully parsed examples.
  4. [Section 4.2, Table 3] The claims that probes 'match or outperform' generated judgments and 'better preserve system rankings' are not supported by significance testing. Table 2 reports five-fold means with standard deviations, and many probe-vs-inference differences are within one standard deviation; Table 3 reports single correlation values without intervals, and the relative advantage is metric-dependent (e.g., RBO sometimes favors inference). Please provide paired significance tests or bootstrap confidence intervals over the folds and over system rankings for the main comparisons.
  5. [Section 4.3, Figure 4] The cross-lingual transfer protocol is underspecified. It is not stated whether regularization strength and probing layer for a source-language probe are selected using source-language validation, target-language validation, or a fixed rule. If target validation is used in any form, the transfer result would overstate portability, since the target set would contribute to model selection. Please specify the selection protocol and, if target validation was used, report a version in which all hyperparameters are fixed from the source language.
minor comments (3)
  1. [Section 3.1] The reference to the TREC DL20 passage-ranking task is cited as [8], but [8] is the ranking-LLM probing paper; the TREC DL20 overview is [12]. Please correct the citation.
  2. [Section 3.4 / Figure 2] Figure 2 caption says 'shaded bands indicate variation across folds,' while Figure 4 uses standard error of the mean; please make the interval definitions uniform and state them in each caption.
  3. [Section 3.5] The text says 'we report the mean performance across the five outer folds and use the standard deviation to summarize fold-level variability.' With only five folds, the standard deviation is noisy; reporting the per-fold values in an appendix or supplement would help readers assess stability.

Circularity Check

0 steps flagged

No significant circularity: held-out linear probing against external human labels; self-citations are background only.

full rationale

The paper's derivation chain is not circular. The central claim—that q-d relevance is linearly decodable from residual-stream activations—is tested by training linear probes on human relevance labels from one set of queries and evaluating on disjoint held-out query groups: 'We split examples by query rather than by individual q-d pair... all documents associated with the same query are assigned to the same split.' Layer and regularization strength are selected on validation labels, and the selected probe is then evaluated once on test data: 'The selected probe is then evaluated once on test data that was not used for either training or model selection.' The probe-versus-output and leaderboard comparisons are therefore external measurements against TREC DL20 and MIRACL gold labels, not quantities that reduce to the paper's own inputs. The oracle-layer rows are explicitly labeled as an upper-bound diagnostic ('This is not a valid model-selection procedure... Oracle results are therefore not used to support the main claims'), so they do not constitute a fitted input being called a prediction. The choice of final-token residual-stream activations is a measurement assumption, and the paper itself lists 'final-token residual-stream activations' as a scope limitation; but a possibly limiting readout choice is an epistemic caveat, not a circular reduction. Self-citations (e.g., [9], [21], [34], [35]) appear only as background or as interpretive consistency checks about multilingual internals and relevance-scale effects; none of them is load-bearing for the main held-out decodability result. Thus no circular step rises to the defined threshold; the low score reflects one or more minor non-load-bearing self-citations rather than any circular derivation.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No new entities, forces, or objects are introduced; the paper only reads out existing activations. The probe is a statistical tool, not a new physical/mathematical entity.

free parameters (2)
  • Probe regularization strength alpha = validation-selected from 13-value log grid (per model/layer/dataset)
    Controls regularized logistic/ordinal probes; chosen by validation Cohen's kappa/QWK, so probe performance depends on this selection.
  • Probing layer = validation-selected layer per fold (not enumerated in the paper)
    The central layer-wise claim and the probe-vs-output comparison depend on which layer is chosen; validation selection is a model-selection choice, while the oracle variant uses test labels.
axioms (4)
  • domain assumption Human labels in TREC DL20 and MIRACL define q-d relevance.
    All probe and output agreement metrics are computed against these labels (Sections 3.1, 3.7); if labels are unreliable, all kappas and rankings are relative to that standard.
  • domain assumption The final input-token residual-stream activation is a faithful summary of the model's relevance representation.
    Stated in Section 3.4 ('represents the model state from which the final relevance judgment is generated'); not independently validated.
  • domain assumption Linear decodability by a supervised probe is evidence that the model internally represents relevance.
    Section 3.5: 'Success under this setting indicates that relevance is available in a simple, linearly accessible form'; this is the standard probing assumption and is not separately conditioned on.
  • domain assumption Grouped query splitting and 54 TREC DL20 queries supply stable fold-level estimates.
    Section 3.5 uses grouped 5-fold CV with ~10-11 test queries per fold; stability of the reported means and system-ranking correlations depends on this small-sample assumption.

pith-pipeline@v1.3.0-alltime-deepseek · 19140 in / 15765 out tokens · 177913 ms · 2026-08-01T22:58:08.850854+00:00 · methodology

0 comments
read the original abstract

Large language models (LLMs) are increasingly used in information retrieval (IR) pipelines as relevance judges and re-rankers. Yet most analyses remain output-centric, evaluating generated labels or scores while offering limited insight into how relevance is represented inside the model. In this work, we study whether query-document (q-d) relevance is linearly decodable from residual-stream activations in instruction-tuned LLMs, how this signal compares with generated relevance judgments, and whether it transfers across languages. Using the TREC DL20 and MIRACL evaluation collections, we guide medium-scale LLMs (4-9B parameters) with UMBRELA-style relevance judgment prompts, extract last-token activations from every transformer layer, and train linear probes to predict relevance labels. We compare probe predictions with generated judgments and use TREC DL20 to test whether probe-derived pseudo-labels preserve system rankings against human judgments. Our results suggest that q-d relevance is encoded as a depth-dependent signal: probe performance is weak in early layers and strongest in middle-to-late layers, indicating that relevance becomes more linearly accessible after contextual integration. Most importantly, in several models, validation-selected probes match or outperform generated judgments and better preserve system rankings, revealing a separation between internal relevance representation and external expression. Multilingual experiments suggest partial cross-language portability, although transfer remains weaker than within-language decoding. Overall, this work provides a representation-level perspective on LLM-based relevance assessment. Layer-wise probing can help diagnose where relevance emerges, when generated judgments fail to reflect internally available evidence, and how relevance representations vary across languages, datasets, and model families.

Figures

Figures reproduced from arXiv: 2607.15555 by Gianluca Demartini, Joel Mackenzie, Pietro Bernardelle, Samaneh Mohtadi, Stefano Civelli.

Figure 1
Figure 1. Figure 1: The prompt used for relevance assessment on the [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Layer-wise probing performance on TREC DL20 and MIRACL. TREC DL20 results use grouped query splits and report binary relevance prediction with Cohen’s 𝜅 after binarizing labels, alongside graded relevance prediction with quadratic weighted 𝜅. For MIRACL, we report the English subset results using Cohen’s 𝜅. Lines show held-out probe performance, shaded bands indicate variation across folds. Markers identif… view at source ↗
Figure 3
Figure 3. Figure 3: Distribution of query-level probe informativeness across relative model depth on TREC DL20 and MIRACL (en). Each panel shows one probing setup: binary probes are evaluated with Cohen’s 𝜅, while graded probes are evaluated with quadratic weighted 𝜅. Rows correspond to models and columns to relative-depth bins. Cell values report the percentage of fractional Top-1 query mass assigned to each bin, splitting t… view at source ↗
Figure 4
Figure 4. Figure 4: Layer-wise portability of binary UMBRELA relevance probes on native MIRACL. The blue curve shows the mean same-language held-out Cohen’s 𝜅 across languages, while the red curve shows the mean cross-language Cohen’s 𝜅 when probes are applied to held-out examples from other languages. Shaded bands indicate the standard error of the mean across language pairs. Dots mark the layer with the highest mean Cohen’s… view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of peak probe portability across relative model depth on native MIRACL. Each cell reports the percentage of language-pair evaluations whose highest held-out Cohen’s 𝜅 occurs in the corresponding relative-depth bin, splitting ties evenly across equally best-performing layers. The same-language panel summarizes probes trained and tested within the same language, while the cross-language transfer… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

65 extracted references · 3 canonical work pages

  1. [1]

    Guillaume Alain and Yoshua Bengio. 2016. Understanding intermediate layers using linear classifier probes.arXiv preprint arXiv:1610.01644(2016)

    Pith/arXiv arXiv 2016

  2. [2]

    Avishek Anand, Lijun Lyu, Maximilian Idahl, Yumeng Wang, Jonas Wallat, and Zijian Zhang. 2022. Explainable information retrieval: A survey.arXiv preprint arXiv:2211.02405(2022)

    Pith/arXiv arXiv 2022

  3. [3]

    de Vries, and Emine Yilmaz

    Peter Bailey, Nick Craswell, Ian Soboroff, Paul Thomas, Arjen P. de Vries, and Emine Yilmaz. 2008. Relevance Assessment: Are Judges Exchangeable and Does It Matter?. InProceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 667–674. doi:10. 1145/1390334.1390447

    arXiv 2008

  4. [4]

    Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2018. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268 [cs.CL] https://arxiv.org/abs/1611.09268

    Pith/arXiv arXiv 2018

  5. [5]

    Ben Carterette and Ian Soboroff. 2010. The Effect of Assessor Error on IR System Evaluation. InProceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, 539–546. doi:10.1145/ 1835449.1835540

    arXiv 2010

  6. [6]

    Catherine Chen, Jack Merullo, and Carsten Eickhoff. 2024. Axiomatic Causal In- terventions for Reverse Engineering Relevance Computation in Neural Retrieval Models(SIGIR ’24). Association for Computing Machinery, New York, NY, USA, 1401–1410. doi:10.1145/3626772.3657841

    arXiv 2024

  7. [7]

    Haodong Chen, Shengyao Zhuang, Zheng Yao, Guido Zuccon, and Teerapong Leelanupab. 2026. Where Relevance Emerges: A Layer-Wise Study of Internal Attention for Zero-Shot Re-Ranking.arXiv preprint arXiv:2602.22591(2026)

    Pith/arXiv arXiv 2026

  8. [8]

    Tanya Chowdhury, Atharva Nijasure, and James Allan. 2025. Probing ranking llms: A mechanistic analysis for information retrieval. InProceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR). 336–346

    2025

  9. [9]

    Stefano Civelli, Pietro Bernardelle, Nicolò Brunello, and Gianluca Demartini

  10. [10]

    Jacob Cohen. 1960. A coefficient of agreement for nominal scales.Educational and Psychological Measurement20, 1 (1960), 37–46. doi:10.1177/001316446002000104

    doi:10.1177/001316446002000104 1960

  11. [11]

    Jacob Cohen. 1968. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit.Psychological Bulletin70, 4 (1968), 213–220. doi:10.1037/h0026256

    doi:10.1037/h0026256 1968

  12. [12]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2021. Overview of the TREC 2020 deep learning track. arXiv:2102.07662 [cs.IR] https://arxiv.org/ abs/2102.07662

    Pith/arXiv arXiv 2021

  13. [13]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2021. Ms marco: Benchmarking ranking models in the large-data regime. InProceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 1566–1576

    2021

  14. [14]

    John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Sala- manca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac, Acyr Locatelli, Hangyu Lin, Dwarak Talupuru, Bharat Ve...

    Pith/arXiv arXiv 2024

  15. [15]

    Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, et al. 2021. A mathematical framework for transformer circuits.Transformer Circuits Thread1, 1 (2021), 12

    2021

  16. [16]

    Furnas, Thomas K

    George W. Furnas, Thomas K. Landauer, Louis M. Gomez, and Susan T. Dumais

  17. [17]

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yixin Dai, Jiawei Sun, Haofen Wang, Haofen Wang, et al. 2023. Retrieval-augmented generation for large language models: A survey.arXiv preprint arXiv:2312.10997 2, 1 (2023), 32

    Pith/arXiv arXiv 2023

  18. [18]

    Ariel Gera, Odellia Boni, Yotam Perlitz, Roy Bar-Haim, Lilach Eden, and Asaf Yehudai. 2025. JuStRank: Benchmarking LLM Judges for System Ranking. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Vienna, Austria, 682–712. doi:10.18653/v1/2025.acl-long.34

    work page doi:10.18653/v1/2025.acl-long.34 2025

  19. [19]

    Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 5484–5495

    2021

  20. [20]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models.arXiv preprint arXiv:2407.21783 (2024)

    Pith/arXiv arXiv 2024

  21. [21]

    Lei Han, Kevin Roitero, Eddy Maddalena, Stefano Mizzaro, and Gianluca Demar- tini. 2019. On transforming relevance scales. InProceedings of the 28th ACM international conference on information and knowledge management. 39–48

    2019

  22. [22]

    Arthur E Hoerl and Robert W Kennard. 1970. Ridge regression: Biased estimation for nonorthogonal problems.Technometrics12, 1 (1970), 55–67

    1970

  23. [23]

    Kalervo Järvelin and Jaana Kekäläinen. 2002. Cumulated Gain-Based Evaluation of IR Techniques.ACM Transactions on Information Systems20, 4 (2002), 422–446. doi:10.1145/582415.582418

    arXiv 2002

  24. [24]

    Jüri Keller, Maik Fröbe, Björn Engelmann, Fabian Haak, Timo Breuer, Birger Larsen, and Philipp Schaer. 2026. Formalized Information Needs Improve Large- Language-Model Relevance Judgments.arXiv preprint arXiv:2604.04140(2026)

    Pith/arXiv arXiv 2026

  25. [25]

    Maurice G. Kendall. 1938. A New Measure of Rank Correlation.Biometrika30, 1-2 (1938), 81–93. doi:10.1093/biomet/30.1-2.81

    doi:10.1093/biomet/30.1-2.81 1938

  26. [26]

    Daoyang Li, Haiyan Zhao, Qingcheng Zeng, and Mengnan Du. 2025. Exploring multilingual probing in large language models: A cross-language analysis. In Proceedings of the 1st Joint Workshop on Large Language Models and Structure Modeling (XLLM 2025). 61–70

    2025

  27. [27]

    Qi Liu, Haozhe Duan, Jiaxin Mao, and Ji-Rong Wen. 2025. How do large language models understand relevance? a mechanistic interpretability perspective.ACM Transactions on Information Systems44, 2 (2025), 1–27

    2025

  28. [28]

    Tie-Yan Liu. 2009. Learning to rank for information retrieval.Foundations and Trends®in Information Retrieval3, 3 (2009), 225–331

    2009

  29. [29]

    William Lugoloobi and Chris Russell. 2025. LLMs Encode How Difficult Problems Are.arXiv preprint arXiv:2510.18147(2025)

    arXiv 2025

  30. [30]

    2008.Introduction to information retrieval

    Christopher D Manning. 2008.Introduction to information retrieval. Syngress Publishing,

    2008

  31. [31]

    Simone Merlo, Stefano Marchesin, Guglielmo Faggioli, and Nicola Ferro. 2025. A cost-effective framework to evaluate llm-generated relevance judgements. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. 2115–2126. CIKM’26, November 7–11, 2026, Rome, Italy Bernardelle et al

    2025

  32. [32]

    Simone Merlo, Stefano Marchesin, Guglielmo Faggioli, and Nicola Ferro. 2026. Reducing Human Effort to Validate LLM Relevance Judgements via Stratified Sampling. InEuropean Conference on Information Retrieval. Springer, 418–433

    2026

  33. [33]

    Alistair Moffat and Justin Zobel. 2008. Rank-biased precision for measurement of retrieval effectiveness. 27, 1, Article 2 (Dec. 2008), 27 pages. doi:10.1145/1416950. 1416952

    work page doi:10.1145/1416950 2008

  34. [34]

    Samaneh Mohtadi and Gianluca Demartini. 2026. Query–Document Dense Vectors for LLM Relevance Judgment Bias Analysis. InEuropean Conference on Information Retrieval. Springer, 88–103

    2026

  35. [35]

    Samaneh Mohtadi, Kevin Roitero, Stefano Mizzaro, and Gianluca Demartini. 2026. The Effect of Document Summarization on LLM-Based Relevance Judgments. In European Conference on Information Retrieval. Springer, 70–87

    2026

  36. [36]

    Atharva Nijasure, Tanya Chowdhury, and James Allan. 2025. How rele- vance emerges: Interpreting lora fine-tuning in reranking llms.arXiv preprint arXiv:2504.08780(2025)

    Pith/arXiv arXiv 2025

  37. [37]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback.Advances in neural information processing systems35 (2022), 27730–27744

    2022

  38. [38]

    Ponte and W

    Jay M. Ponte and W. Bruce Croft. 1998. A language modeling approach to information retrieval. InProceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval(Melbourne, Australia)(SIGIR ’98). Association for Computing Machinery, New York, NY, USA, 275–281. doi:10.1145/290941.291008

    arXiv 1998

  39. [39]

    Qwen Team. 2026. Qwen3.5: Towards Native Multimodal Agents. https://qwen. ai/blog?id=qwen3.5

    2026

  40. [40]

    Filip Radlinski and Nick Craswell. 2017. A theoretical framework for conversa- tional search. InProceedings of the 2017 conference on conference human informa- tion interaction and retrieval. 117–126

    2017

  41. [41]

    Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L

    Hossein A. Rahmani, Clemencia Siro, Mohammad Aliannejadi, Nick Craswell, Charles L. A. Clarke, Guglielmo Faggioli, Bhaskar Mitra, Paul Thomas, and Emine Yilmaz. 2025. Judging the Judges: A Collection of LLM-Generated Relevance Judgements.arXiv preprint arXiv:2502.13908(2025). https://arxiv.org/abs/2502. 13908

    Pith/arXiv arXiv 2025

  42. [42]

    Hossein A Rahmani, Emine Yilmaz, Nick Craswell, Bhaskar Mitra, Paul Thomas, Charles LA Clarke, Mohammad Aliannejadi, Clemencia Siro, and Guglielmo Faggi- oli. 2024. Llmjudge: Llms for relevance judgments.arXiv preprint arXiv:2408.08896 (2024)

    Pith/arXiv arXiv 2024

  43. [43]

    2009.The probabilistic relevance frame- work: BM25 and beyond

    Stephen Robertson and Hugo Zaragoza. 2009.The probabilistic relevance frame- work: BM25 and beyond. Vol. 4. Now Publishers Inc

    2009

  44. [44]

    Lisa Schut, Yarin Gal, and Sebastian Farquhar. 2025. Do Multilingual LLMs Think In English?. InICLR 2025 Workshop on Building Trust in Language Models and Applications

    2025

  45. [45]

    Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval.Journal of documentation28, 1 (1972), 11–21

    1972

  46. [46]

    Charles Spearman. 1987. The Proof and Measurement of Association Between Two Things.The American Journal of Psychology100, 3/4 (1987), 441–471. doi:10. 2307/1422689

    1987

  47. [47]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 Technical Report. arXiv:2503.19786 [cs.CL] https: //arxiv.org/abs/2503.19786

    Pith/arXiv arXiv 2025

  48. [48]

    Paul Thomas, Seth Spielman, Nick Craswell, and Bhaskar Mitra. 2024. Large Language Models Can Accurately Predict Searcher Preferences. InProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’24). 1930–1940. doi:10.1145/3626772.3657707

    arXiv 2024

  49. [49]

    Shivani Upadhyay, Ehsan Kamalloo, and Jimmy Lin. 2024. LLMs Can Patch Up Missing Relevance Judgments in Evaluation.arXivabs/2405.04727 (2024). https://arxiv.org/abs/2405.04727

    Pith/arXiv arXiv 2024

  50. [50]

    Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, and Jimmy Lin. 2025. A Large- Scale Study of Relevance Assessments with Large Language Models Using UM- BRELA. InProceedings of the 2025 International ACM SIGIR Conference on In- novative Concepts and Theories in Information Retrieval (ICTIR ’25). 3...

    arXiv 2025

  51. [51]

    Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, and Jimmy Lin. 2024. Umbrela: Umbrela is the (open-source reproduction of the) bing relevance assessor.arXiv preprint arXiv:2406.06519(2024)

    Pith/arXiv arXiv 2024

  52. [52]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need.Advances in neural information processing systems30 (2017)

    2017

  53. [53]

    Ellen M Voorhees. 2001. The philosophy of information retrieval evaluation. In Workshop of the cross-language evaluation forum for european languages. Springer, 355–370

    2001

  54. [54]

    Yumeng Wang, Jirui Qi, Catherine Chen, Panagiotis Eustratiadis, and Suzan Verberne. 2026. How role-play shapes relevance judgment in zero-shot LLM rankers. InEuropean Conference on Information Retrieval. Springer, 228–242

    2026

  55. [55]

    William Webber, Alistair Moffat, and Justin Zobel. 2010. A Similarity Measure for Indefinite Rankings.ACM Transactions on Information Systems28, 4 (2010), 20:1–20:38. doi:10.1145/1852102.1852106

    arXiv 2010

  56. [56]

    Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024. Do llamas work in english? on the latent language of multilingual transformers. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 15366–15394

    2024

  57. [57]

    An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report.arXiv preprint arXiv:2505.09388(2025). https://arxiv.org/abs/2505.09388

    Pith/arXiv arXiv 2025

  58. [58]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al . 2025. Qwen2.5 Technical Report.arXiv preprint arXiv:2412.15115(2025). https://arxiv.org/abs/ 2412.15115

    Pith/arXiv arXiv 2025

  59. [59]

    Riccardo Zamolo, Riccardo Lunardi, Michael Soprano, Gianluca Demartini, Ste- fano Mizzaro, and Kevin Roitero. 2026. Large Language Models as Assessors: On the Impact of Relevance Scales. InEuropean Conference on Information Retrieval. Springer, 338–348

    2026

  60. [60]

    Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin

  61. [61]

    Yu Zhao, Xiaotang Du, Giwon Hong, Aryo Pradipta Gema, Alessio Devoto, Hon- gru Wang, Xuanli He, Kam-Fai Wong, and Pasquale Minervini. 2024. Analysing the residual stream of language models under knowledge conflicts. InNeurIPS 2024-Workshop on Foundation Model Interventions. Neural Information Processing Systems Foundation (NeurIPS), 1–12

    2024

  62. [62]

    Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2025. Large language models for information retrieval: A survey.ACM Transactions on Information Systems44, 1 (2025), 1–54

    2025

  63. [1987]

    ACM 30, 11 (1987), 964–971

    The vocabulary problem in human-system communication.Commun. ACM 30, 11 (1987), 964–971

    1987

  64. [2023]

    Transactions of the Association for Computational Linguistics11 (2023), 1114–1131

    MIRACL: A Multilingual Retrieval Dataset Covering 18 Diverse Languages. Transactions of the Association for Computational Linguistics11 (2023), 1114–1131. doi:10.1162/tacl_a_00595

    doi:10.1162/tacl_a_00595 2023

  65. [2026]

    In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Maria Liakata, Viviane P

    A Shared Geometry of Difficulty in Multilingual Language Models. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Maria Liakata, Viviane P. Moreira, Jiajun Zhang, and David Jurgens (Eds.). Association for Computational Linguistics, San Diego, California, United States, 796–807. doi:10.186...

    work page doi:10.18653/v1/2026.acl-short.66 2026