Modeling Ranking Properties with In-Context Learning

Andrew Parry; Debasis Ganguly; Nilanjan Sinhababu; Pabitra Mitra

REVIEW 2 major objections 4 minor 1 cited by

Modeling Ranking Properties with In-Context Learning

T0 review · 2 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An in-context example ranking built from a similar query can steer a large language model's reranking toward fairness or diversity, with no supervised training and no explicit instruction about the auxiliary objective.

desk verdict A clever training-free idea for steering LLM rankings, but the causal claim is over-reached: the ablations don't isolate the target distribution, and the target distribution itself is under-specified. read the letter →

arxiv 2505.17736 v1 pith:VY5ATNVM submitted 2025-05-23 cs.IR

Nilanjan Sinhababu , Andrew Parry , Debasis Ganguly , Pabitra Mitra This is my paper

classification cs.IR

keywords in-contextlearninglistwisererankingrankingfairnesssearchresultdiversityLLMdemonstrationengineeringtargetdistribution

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a single in-context example — a ranking of documents for a query similar to the current one — can control whether a large language model reranker favours relevance alone or also balances fairness and diversity, without supervised training and without explicit instructions about the auxiliary objective. The example is built by reordering retrieved documents so that their attribute composition (gender, stance, or topic cluster) matches a target distribution at every prefix. The paper reports experiments on four test collections showing that this demonstration-based control improves the auxiliary objectives while largely preserving nDCG, and that inverting or degrading the example significantly hurts the auxiliary metric. If the claim holds, operators could steer search behaviour by editing examples rather than retraining or rewriting prompts, which matters because fairness and diversity objectives are hard to express in natural language and change over time.

What carries the argument

The central mechanism is a target distribution $\tau(R(Q))$ over $k$ attribute categories combined with a greedy rank-induction step: documents retrieved for the similar query are partitioned by attribute value, and at each position the next document is chosen from the candidate heads of the category lists so that the prefix's empirical distribution has minimum KL divergence to $\tau(R(Q))$ (Equation 3). The resulting permutation is inserted as the 'example ordering' in a listwise ranking prompt whose instruction is merely to rank by relevance, making the example the only channel that carries the fairness or diversity objective. The machinery is attribute-agnostic: the same construction handles binary gender, binary stance, and query-dependent topic clusters derived by hierarchical agglomerative clustering.

What would settle it

Fix the fairness target distribution in advance (e.g., uniform gender split) without consulting any test-query relevance judgments, rebuild the demonstrations from a query log like MS MARCO, and measure whether the reranked top-10 on TREC Fairness 2022 moves toward that uniform split; if it does not, the demonstration is not the causal control signal claimed and the label-free premise fails.

Watch

Extended reading notes

Core claim

The authors claim that in-context learning examples are a causal, sufficient control signal for listwise ranking behaviour in instruction-tuned LLMs. For a test query, they retrieve the most similar query from a large unlabeled query log, retrieve that query's top documents, and greedily reorder them so that each prefix of the example list has an attribute-category distribution closest (in KL divergence) to a target distribution $\tau(R(Q))$; the resulting list is placed in the prompt before the test query's documents, with the instruction only to rank by relevance. Across TREC Deep Learning 2019/2020 (topical diversity), Touché (pro–con balance), and TREC Fairness 2022 (gender exposure), the method with GPT-4o-mini improves $\alpha$nDCG or AWRF over zero-shot and prompt-instructed baselines while keeping nDCG competitive, and it matches or exceeds post-hoc methods such as MMR and FA*IR on several settings. The causal claim rests on ablations in which swapping the target proportions, using uniform examples, or replacing the localized example with a static one degrades the auxiliary objective, while relevance-only examples still improve nDCG.

Load-bearing premise

That the target distribution over attribute categories used to build each demonstration can be specified without relying on the test query's relevance judgments, even though the paper's fairness description ties that distribution to group labels in relevance judgments, leaving the claimed label-free control unresolved.

Editorial extensions

If this is right

Operators can change a deployed reranker's behaviour on fairness or diversity by replacing one demonstration, without retraining or prompt rewriting, so objectives can be updated as requirements change.
Auxiliary objectives that resist natural-language instruction — such as balanced stance or topic coverage — become controllable because they are encoded as a distribution over examples rather than as words.
The same pipeline works across different attribute types (gender, stance, topics), suggesting the demonstration mechanism generalizes to any ranking property expressible as a target category distribution.
Demonstrations only steer behaviour when they are local to the query: static examples degrade both relevance and auxiliary measures, so example selection is a necessary component, not a decoration.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural next test is whether the target distribution can be user-specified (e.g., request a 50/50 gender split) rather than derived from relevance judgments; the paper's fairness description ties $\tau(R(Q))$ to judgement-set group labels, so a label-free version of the method lives or dies on that specification.
If the causal reading is right, the method turns a ranking objective into a versionable artefact: audits and fairness fixes could be expressed as diffs to example rankings, which changes how search systems are governed.
The greedy KL-matching construction suggests the principle extends beyond fairness and diversity to any per-prefix distributional target — e.g., recency, sentiment, or geographic coverage — and testing across languages and domains would reveal how much depends on the base LLM's world knowledge.
Because only one example is used, multiple demonstrations might encode complex trade-offs more reliably, or expose sensitivity to example quality; the paper leaves that parameter space open.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The one thing to know: this is a genuinely new combination—using in-context demonstrations of target-distribution rankings to get an LLM to rerank for fairness or diversity without training. If it works, it's a nice tool for the IR toolkit. The evaluation is broad: four collections, two auxiliary objectives, two first-stage retrievers, and several ablations. Code is provided.

What's good: the core idea is simple and testable. The greedy construction of example rankings (an MMR-style selection toward a target distribution) is clean. The comparison against PAO (0-shot with explicit instruction) shows demonstrations can be more effective than instruction-prompting, at least for GPT-4o-mini. That's a useful empirical result even if the larger causal story is shaky.

The soft spots: the causal claim in Section 1 ("demonstrations can change behaviour in a causal manner") is not supported by Table 3. Each ablation differs from +Target in multiple ways: different document order, different selection of documents, different query. Appendix C shows that simply randomizing the example ordering changes metrics by amounts comparable to the gaps between +Target and the ablation baselines. So the target distribution is confounded with a known order effect. The ablation as designed cannot isolate the target distribution.

Second, the target distribution itself: Equation (1) is defined over R(Q), the relevant set, but the paper never says how R(Q) is obtained for fairness examples. Section 4.1 says "group labels in relevance judgments inform the target distribution," which contradicts the claim of label-free control. If the target distribution is computed from test-query relevance judgments, then the method is not label-free and the transfer story weakens. If it's user-specified, then the sensitivity to that choice should be analyzed. This is not a minor detail; the whole method rests on this distribution.

Third, the numbers don't all line up. The text says "approximately a 52% improvement in nDCG compared to DELTR and FA*IR," but Table 2 shows a few percent relative gain. There's also a contradiction between the main text (k=1) and Appendix A.3 (top-5 similar queries) that needs resolving.

None of these are fatal to the core idea—the concept is plausible and the preliminary results are suggestive. But as it stands, the evidence is not clean enough to back the causal language. A well-scoped revision with order-controlled ablations and a transparent definition of tau would make this a solid paper.

Who's it for: people working on LLM-based ranking, fairness/diversity in IR, or ICL for control. I'd send it to a serious referee—the idea deserves scrutiny—but my own recommendation would be conditional: address the confound and the target-distribution ambiguity before publication.

Referee Report

2 major / 4 minor

Summary. The paper proposes an in-context learning (ICL) method for steering listwise neural rerankers toward auxiliary ranking objectives such as group fairness and topical diversity. For a test query, the method retrieves the most similar query from the MS MARCO training set, retrieves its top documents from a target collection, and reorders those documents with a greedy KL-minimizing procedure (Eq. 3) so that the sequence approximately matches a target distribution over an attribute (e.g., gender, stance, or topic cluster). This example ranking is inserted into a RankGPT-style prompt, and the LLM is asked to rank the current query's documents. Experiments on TREC Fairness 2022, Touché 2020, and TREC Deep Learning 2019/2020, with BM25 and ColBERT first-stage rankers and several LLMs, show gains in AWRF/M1 and αnDCG over zero-shot baselines while maintaining nDCG. The paper claims causal control via demonstrations and a label-free adaptation mechanism, and it releases source code.

Significance. If the central causal claim held, the paper would provide a practical, training-free way to control ranking behavior by demonstration, avoiding task-specific prompt engineering and supervised listwise training. The experimental breadth is a strength: four collections, two first-stage retrievers, three LLM families, and released code. However, two load-bearing issues weaken the contribution as presented: the ablations in Table 3 do not isolate the target distribution from example-document ordering, and the construction of the target distribution for fairness appears to require relevance judgments and group labels, undercutting the label-free framing. The significance is therefore conditional on resolving these points.

major comments (2)

[Section 3, Eq. (1); Section 4.1] The target distribution τ(R(Q)) in Eq. (1) is defined over the set of relevant documents R(Q) for the example query Q, but the paper never states how R(Q) is obtained. Section 4.1 says that for fairness, 'group labels in relevance judgments inform the target distribution', which implies that relevance judgments (or at least group labels for relevant documents) are required to construct the example ranking. This is in tension with the abstract's claim that the method 'eliminates the need for such training' and with the Introduction's statement that the approach works 'without requiring relevance assessments'. Please clarify whether the target distribution is user-specified or derived from labeled data, and if the latter, revise the label-free claims accordingly.
[Section 4.3, Table 3; Appendix C] The ablations in Table 3 are intended to establish that the target distribution, rather than other properties of the example, causes the observed changes in ranking behavior. However, each ablation changes both the target distribution and the sequence of example documents (for +Static, the example query also changes). Appendix C demonstrates a substantial sensitivity to example ordering: on Touche-2020, random vs first-stage ordering changes AWRF from .5800 to .5410 and M1 from .2023 to .2180; on Fair-2022, nDCG changes from .5697 to .6029 and M1 from .3526 to .4013. These order-induced shifts are of the same magnitude as, or larger than, the differences between +Target and the ablation conditions in Table 3. Because example document order is not held fixed across conditions, the ablation results do not isolate the target distribution, and the claim in Section 1 that 'ablations confirm demonstrations as the causal factor' is not yet supported. Please add a condition that controls for document order (e.g., identical input ordering across target/adversarial/relevant examples) or restate the claim as a correlational finding.

minor comments (4)

[Section 4.3] The text states that the approach yields 'improvements of up to 19%' in αnDCG over the 0-shot baseline, but the largest such improvement visible in Table 1 is about 5.5% (DL-2019, BM25, GPT-4o-mini: rows 5 vs 7, .6761 to .7135). Please correct the percentage or show the exact computation.
[Section 4.3, Table 2] The sentence 'our method achieves approximately a 52% improvement in nDCG performance compared to DELTR and FA*IR' holds only for the Fair-2022 dataset; on Touche-2020 the corresponding improvements are around 5-6%. Please specify the dataset in that sentence to avoid over-generalization.
[Appendix A.3 vs Section 3] Appendix A.3 says 'We retrieve the top-5 most similar queries' while Section 3 states k=1. These statements should be reconciled, for example by clarifying that k=1 is used in the main experiments and the top-5 retrieval is used only for candidate generation or analysis.
[Table 3 caption] The caption says suffixes a to e denote statistical significance of +Target against first-stage, 0-shot, Adversarial, Relevant, and Static, but the table rows show only a subset of these letters. Please ensure the legend matches the displayed markers.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the claimed demonstration-to-ranking control is evaluated as a held-out transfer test, and the self-citations set auxiliary hyperparameters rather than carrying the derivation.

full rationale

The derivation chain is not circular by construction. The target distribution tau(R(Q)) (Eq. 1) is an input specification; Eq. 3 greedily reorders retrieved documents for a similar query Q to approximate that target, producing one demonstration. The prompt (Fig. 4) then conditions a frozen LLM on this demonstration and asks it to rerank a disjoint test query Qc. Evaluation metrics (AWRF, M1, and alpha-nDCG) are computed on the test outputs, not on the example rankings, so the reported gains are a transfer and manipulation result rather than a restatement of the construction objective. The model is not trained or fitted to the test auxiliary metrics, and the test queries and documents are not used to build the examples. The main limitations are non-circular: Appendix C shows document-order sensitivity that can confound the causal attribution in Table 3, and Section 4.1's statement that 'group labels in relevance judgments inform the target distribution' leaves unresolved how tau is set for similar queries from MS-MARCO. These are validity and operationalization gaps, not reductions of the prediction to the input. Self-citations, such as fixing k=1 following Sinhababu et al. (2024) or ordering analyses from Parry et al. (2024a), set design choices or motivate ablations; the paper's own Static ablation independently supports locality, so the central claim does not rest solely on those citations. Accordingly no step satisfies the required standard of exhibiting an equation-level equivalence or a fitted-parameter renaming.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper relies on the LLM's ICL behavior, on a user-specified target distribution, and on a greedy list-construction heuristic; none of these are derived from first principles. The target distribution for fairness is the least documented input, and the number of similar queries used is internally contradictory.

free parameters (4)

target_distribution_tau = unspecified (e.g., uniform for Touche and diversity; unclear for Fair-2022)
The greedy example construction in Eq. (3) minimizes KL to this target; its values are never listed for the fairness experiments.
k_num_similar_queries = 1 (Section 3) or 5 aggregated (Appendix A.3)
Main text states k=1, appendix states top-5 similar queries are aggregated; the discrepancy is unresolved.
m_example_list_size = not reported
The prompt template uses {m} and Figure 2 shows 3 passages, but the paper never states the example list size used in experiments.
clustering_threshold_diversity = not reported
Hierarchical agglomerative clustering with complete linkage produces a query-dependent number of clusters; the cutoff threshold is not given.

assumptions (4)

domain assumption A single in-context example ranking suffices for the LLM to infer the desired auxiliary objective and transfer it to a new query.
Central to the method; tested only via ablations, not guaranteed by the paper's design.
domain assumption The BM25 most similar query from MS MARCO is topically aligned enough with the test query for demonstrations to transfer.
Used in Section 3 and Appendix A.3; static examples degrade, implying topicality matters.
ad hoc to paper The greedy KL-minimizing selection (Eq. 3) produces example list prefixes that match the target distribution well enough for the LLM to perceive the pattern.
No analysis is provided of how close the generated permutations are to the target distribution.
domain assumption Relevance can be preserved because the instruction is 'rank by relevance' even when the example shows a different ordering.
The paper assumes the model balances the instruction with the example's implied objective rather than just copying the example's exact order.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Modeling Ranking Properties with In-Context Learning." pith.science (2026). https://pith.science/paper/VY5ATNVM

@misc{pith2026250517736,
  author       = {Pith},
  title        = {Pith review of: Modeling Ranking Properties with In-Context Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VY5ATNVM}},
  note         = {Machine review of arXiv:2505.17736}
}

read the original abstract

While standard IR models are mainly designed to optimize relevance, real-world search often needs to balance additional objectives such as diversity and fairness. These objectives depend on inter-document interactions and are commonly addressed using post-hoc heuristics or supervised learning methods, which require task-specific training for each ranking scenario and dataset. In this work, we propose an in-context learning (ICL) approach that eliminates the need for such training. Instead, our method relies on a small number of example rankings that demonstrate the desired trade-offs between objectives for past queries similar to the current input. We evaluate our approach on four IR test collections to investigate multiple auxiliary objectives: group fairness (TREC Fairness), polarity diversity (Touch\'e), and topical diversity (TREC Deep Learning 2019/2020). We empirically validate that our method enables control over ranking behavior through demonstration engineering, allowing nuanced behavioral adjustments without explicit optimization.

Figures

Figures reproduced from arXiv: 2505.17736 by the authors.

**Figure 1.** Proposed ICL method for reranking a set of top-retrieved documents. An example constitutes a localized query along with its top-retrieved arranged to satisfy a desired ranking property, such as relevance, fairness, diversity, etc. relevance, which is an individual property of a document itself, these additional objectives, such as diversity (Clarke et al., 2008) or fair representation (Craswell et al., 2008), are i… view at source ↗

**Figure 2.** ICL Example for a Touche query. For this [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** The figure shows a sample input query from [PITH_FULL_IMAGE:figures/full_fig_p013_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: The prompt template used in our work with the [PITH_FULL_IMAGE:figures/full_fig_p015_4.png]

**Figure 5.** Figure 5: An example showing five localized queries [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Scaling Search Relevance: Augmenting App Store Ranking with LLM-Generated Judgments
cs.IR 2026-02 conditional novelty 5.0 of 10

Using a fine-tuned 3B LLM to generate millions of textual relevance labels for App Store search improves the ranker's behavioral/textual Pareto frontier and lifts conversion by 0.24%.

Reference graph

Works this paper leans on

41 extracted references · 37 canonical work pages · cited by 1 Pith paper

[1]

tenure is needed to protect academic freedom| {z } CON

work page
[2]

without tenure teachers may be fired.| {z } CON

work page
[3]

text-generation

it is difficult to remove under-performing teachers.| {z } PRO Rank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [ ] > [ ], e.g., [1] > [2]. Figure 3: The figure shows a sample input quer...

work page arXiv 2024
[4]

How big can goldfish grow?

work page
[5]

What is a goldfish? Localized Queries from TREC DL-2020 Test Query Why is pete rose banned from hall of fame? Similar Query from MS-MARCO

work page 2020
[10]

In Findings of the Association for Computa- tional Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 14880– 14891

Multi-task transfer matters during instruction- tuning. In Findings of the Association for Computa- tional Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 14880– 14891. Association for Computational Linguistics. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng

work page 2024
[12]

Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibration

Improving few-shot performance of language models via nearest neighbor calibration. CoRR, abs/2212.02216. Andrew Parry, Sean MacAvaney, and Debasis Ganguly. 2024a. Exploiting positional bias for query-agnostic generative content in search. In Findings of the As- sociation for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, Augu...

work page Pith review arXiv 2024
[15]

ACM Trans

I know what you need: Investigating document retrieval effectiveness with partial session contexts. ACM Trans. Inf. Syst., 40(3):53:1–53:30. Ashudeep Singh and Thorsten Joachims. 2018. Fairness of exposure in rankings. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, KDD 2018, London, UK, August 19-23,...

work page 2018

Show all 41 references

[20]

[k] {Dm} The {m} passages above are ranked based on their relevance to the search query

{D2} ... [k] {Dm} The {m} passages above are ranked based on their relevance to the search query. Output: [4] > [1] > ... > [6]| {z } Example ordering Current (Input) Query Rank the passages based on their relevance to query: {Qc}
[21]

[k] {Dc m} Rank the {m} passages above based on their rele- vance to the search query

{Dc 2} ... [k] {Dc m} Rank the {m} passages above based on their rele- vance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [ ] > [ ], e.g., [1] > [2]. Fig...

2023
[22]

How to grow goldfish faster?
[23]

How big do shubunkin goldfish grow?
[24]

How fast do baby goldfish grow?
[27]

Why is pete rose banned from mlb?
[28]

Where does pete rose do autographs?
[29]

How old is pete rose?
[30]

When was pete rose born?
[31]

How many catchers are in the hall of fame? Localized Queries from Touche-2020 Test Query Should teachers get tenure? Similar Query from MS-MARCO

2020
[32]

How many years does it take to get tenure as a teacher?
[33]

What is the illinois mandate for tenured teacher evaluation?
[34]

What are tenure protections?
[35]

What is tenure mean?
[36]

Greenspan tenure? Localized Queries from Fair-2022 Test Query Architecture Similar Query from MS-MARCO

2022
[37]

What do you do in architecture?
[38]

What is it architecture?
[39]

What is an architecture do?
[40]

What is architectural?
[41]

How roman architecture influenced mod- ern architecture? Figure 5: An example showing five localized queries that are retrieved for a test query in each test collection. 15
[126]

Ferdinand Schlatt, Maik Fröbe, Harrisen Scells, Shengyao Zhuang, Bevan Koopman, Guido Zuc- con, Benno Stein, Martin Potthast, and Matthias Ha- gen

National Institute of Standards and Technology (NIST). Ferdinand Schlatt, Maik Fröbe, Harrisen Scells, Shengyao Zhuang, Bevan Koopman, Guido Zuc- con, Benno Stein, Martin Potthast, and Matthias Ha- gen. 2024. Set-encoder: Permutation-invariant inter- passage attention for list...

2024 arXiv
[1513]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S

ACM. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Nat- ural Language Proce...

2020 arXiv
[1578]

Meike Zehlike and Carlos Castillo

ACM. Meike Zehlike and Carlos Castillo. 2020. Reducing disparate exposure in ranking: A learning to rank approach. In WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 2849–2855. ACM / IW3C2. Yadong Zhu, Yanyan Lan, Jiafeng Guo, Xueqi Cheng, and Shuzi ...

2020
[1994]

In Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithers- burg, Maryland, USA, November 2-4, 1994, volume 500-225 of NIST Special Publication , pages 109–

Okapi at TREC-3. In Proceedings of The Third Text REtrieval Conference, TREC 1994, Gaithers- burg, Maryland, USA, November 2-4, 1994, volume 500-225 of NIST Special Publication , pages 109–

1994
[2013]

The whens and hows of learning to rank for web search. Inf. Retr., 16(5):584–628. Marco Morik, Ashudeep Singh, Jessica Hong, and Thorsten Joachims. 2020. Controlling fairness and bias in dynamic learning-to-rank. In Proceedings of the 43rd International ACM SIGIR conference on...

2020
[2016]

MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the Workshop on Cognitive Computation: Integrat- ing neural and symbolic approaches 2016 co-located with the 30th Annual Conference on Neural Infor- mation Processing Systems (NIPS 2016), Barc...

2016
[2019]

In Proceedings of the 42nd Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1403–1404

Fairness and discrimination in retrieval and recommendation. In Proceedings of the 42nd Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, pages 1403–1404. ACM. Michael D. Ekstrand, Graham McD...

2019
[2020]

Overview of touché 2020: Argument retrieval: Extended abstract. In Experimental IR Meets Mul- tilinguality, Multimodality, and Interaction: 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22–25, 2020, Proceedings, page 384–395,...

2020 arXiv
[2022]

In SIGIR ’22: The 45th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, pages 759–769

Fairness of exposure in light of incomplete exposure estimation. In SIGIR ’22: The 45th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrieval, Madrid, Spain, July 11 - 15, 2022, pages 759–769. ACM. Yoichi Ishibashi, Danushka Bollegala, Kat...

2022
[2024]

In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 1504–

Fairness-aware exposure allocation via adap- tive reranking. In Proceedings of the 47th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pages 1504–

2024
[2292]

Debasis Ganguly, Manisha Ganguly, Johannes Leveling, and Gareth J

ACM. Debasis Ganguly, Manisha Ganguly, Johannes Leveling, and Gareth J. F. Jones. 2013. Topicvis: a GUI for topic-based feedback and navigation. In The 36th International ACM SIGIR conference on research and development in Information Retrieval, SIGIR ’13, Dublin, Ireland - Ju...

2013 arXiv
[2425]

Sean MacAvaney, Craig Macdonald, Roderick Murray- Smith, and Iadh Ounis

ACM. Sean MacAvaney, Craig Macdonald, Roderick Murray- Smith, and Iadh Ounis. 2021. Intent5: Search result diversification using causal language models. CoRR, abs/2108.04026. Craig Macdonald, Rodrygo L. T. Santos, and Iadh Ounis

2021 arXiv
[5139]

Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin

Association for Computational Linguistics. Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. 2024. Fine-tuning llama for multi-stage 10 text retrieval. In Proceedings of the 47th Interna- tional ACM SIGIR Conference on Research and De- velopment in Information Retrie...

2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

tenure is needed to protect academic freedom| {z } CON

work page

[2] [2]

without tenure teachers may be fired.| {z } CON

work page

[3] [3]

text-generation

it is difficult to remove under-performing teachers.| {z } PRO Rank the 3 passages above based on their relevance to the search query. The passages should be listed in descending order using identifiers. The most relevant passages should be listed first. The output format should be [ ] > [ ], e.g., [1] > [2]. Figure 3: The figure shows a sample input quer...

work page arXiv 2024

[4] [4]

How big can goldfish grow?

work page

[5] [5]

What is a goldfish? Localized Queries from TREC DL-2020 Test Query Why is pete rose banned from hall of fame? Similar Query from MS-MARCO

work page 2020

[6] [10]

In Findings of the Association for Computa- tional Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 14880– 14891

Multi-task transfer matters during instruction- tuning. In Findings of the Association for Computa- tional Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 14880– 14891. Association for Computational Linguistics. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng

work page 2024

[7] [12]

Improving Few-Shot Performance of Language Models via Nearest Neighbor Calibration

Improving few-shot performance of language models via nearest neighbor calibration. CoRR, abs/2212.02216. Andrew Parry, Sean MacAvaney, and Debasis Ganguly. 2024a. Exploiting positional bias for query-agnostic generative content in search. In Findings of the As- sociation for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, Augu...

work page Pith review arXiv 2024

[8] [15]

ACM Trans

I know what you need: Investigating document retrieval effectiveness with partial session contexts. ACM Trans. Inf. Syst., 40(3):53:1–53:30. Ashudeep Singh and Thorsten Joachims. 2018. Fairness of exposure in rankings. In Proceedings of the 24th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, KDD 2018, London, UK, August 19-23,...

work page 2018