Pith. sign in

REVIEW 4 major objections 5 minor 46 references

Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Reranking the input of a symmetric task—putting query-relevant elements where a model's attention lingers—can bring an LLM's accuracy to within about 1% of the optimal ordering.

desk verdict A useful wrapper for symmetric LLM tasks with a solid debiasing core, but the exposure-transfer claim needs a direct test before the headline numbers are trusted. read the letter →

arxiv 2412.00546 v1 pith:MK3YKCS4 submitted 2024-11-30 cs.LG cs.DBcs.IR

Mohsen Dehghankar , Abolfazl Asudeh This is my paper · ORCID
classification cs.LGcs.DBcs.IR
keywords LLMinputrerankingsymmetrictaskspositionexposurerelevanceestimationbipartitedebiasingSinkhornscalingaggregatequerieslongcontext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models forget parts of long inputs, and the forgetting is not uniform: items in some positions are much more likely to be missed than items in others. For symmetric tasks—queries about an unordered bag of elements, where any ordering is logically valid—this position bias can be exploited instead of suffered. The paper introduces input reranking: learn a position-by-position exposure profile for the target LLM, estimate how relevant each input element is to the query using a small helper model, then reorder the input so relevant elements sit in high-exposure positions. Experiments on graph-degree and database-aggregation tasks report that this wrapper improves accuracy by up to 99% of the gap between random ordering and an optimal ordering. If the claim holds, accuracy gains are available without retraining, new data, or changing the prompt content.

What carries the argument

The exposure function $X_L(i)$ is the central object: it is the probability-like score that the target LLM $L$ will not miss an item at rank position $i$. The paper estimates it from random permutations of a toy token-counting task by solving $R^\top X = \vec{\epsilon}$ in the least-squares sense, where $R$ records which tokens were relevant in each permutation and $\vec{\epsilon}$ holds the inverse of the observed output error (Eq. 10); a $t$-distribution confidence interval controls the number of permutations needed. For query-time relevance, the paper builds a bipartite evaluation graph whose left nodes are input elements and right nodes are helper-LLM evaluations of shuffled partitions, then alternates updates of element scores and evaluation-bias coefficients exactly like Sinkhorn scaling, which converges to a doubly stochastic limit and removes per-evaluation over- or under-scoring bias. Combining the two through Eq. 2 yields the reranking: relevant elements are moved to the positions the model actually sees.

What would settle it

Run the exposure-discovery phase on a graph-degree task itself and compare the measured per-position miss rates to the profile learned from token counting; if the two differ, apply the claimed reranking and show its error is no better than random ordering, which would directly falsify the transfer claim.

Watch

Extended reading notes

Core claim

LLMs systematically overlook certain positions in long prompts, and the paper's central claim is that this position-dependent exposure can be measured and inverted into a better input order. The model treats the expected utility of a ranking as the position-weighted sum of element relevance, $E[utility(\pi|q)] = \sum_i X_L(i) \cdot Rel_q(e_{\pi(i)})$, where $X_L(i)$ is the exposure of position $i$ and $Rel_q$ is relevance to the query. Exposure is learned offline by fitting a least-squares system on a synthetic token-counting task whose per-token relevance is known, and query-time relevance is estimated by asking a helper LLM for coarse scores on shuffled partitions and debiasing those scores through iterative row/column scaling on a bipartite evaluation graph. In experiments on graph-degree and database aggregation tasks, the resulting rerankings reach roughly 97–99% of the optimal ranking utility in most settings and cut normalized output error nearly to zero, with the best helper models. The paper's claim is that reranking alone—no model change, no prompt-content change—recovers most of the accuracy that long-input forgetting loses.

Load-bearing premise

The whole pipeline rests on the belief that the position profile learned on a synthetic single-token counting task is a fixed property of the model that carries over unchanged to graph and database tasks, and that the error falls linearly with the exposure-weighted sum of relevance.

Editorial extensions

If this is right

  • The same black-box wrapper applies to any symmetric task—edge lists, table rows, unordered collections—without task-specific prompt engineering.
  • Exposure profiles are model-specific, so the preprocessing must be repeated per target LLM; the paper observes that one evaluated model remembers the beginning best while another remembers the middle.
  • Query-time relevance estimation costs only $\sigma m$ helper-LLM calls, so the overhead is small relative to the accuracy gain.
  • The bipartite debiasing procedure generalizes beyond reranking: any pointwise scoring by an LLM can be debiased the same way.
  • When reranking is used, normalized output error on several graph and database tasks lands close to the optimum, not just better than random.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The exposure-transfer assumption is the part most worth stress-testing: if per-position miss rates are measured directly on a graph or database task and they disagree with the token-counting profile, the reranking could be no better than random; a content-conditioned exposure model would be the natural fix.
  • The same machinery could be applied to retrieval-augmented generation, where the order of retrieved passages is controllable; the exposure profile would need to be re-estimated for that context structure.
  • A direct testable extension is to compare Eq. 9's linear model against a content-aware or interaction-aware error model on tasks where relevant items cluster semantically, since the linear model assumes position alone drives forgetting.
  • The bipartite debiasing step could be reused as a standalone bias-correction tool for any LLM-based scoring pipeline, including preference and rubric-based evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies symmetric tasks—queries over an unordered bag of elements—and proposes to improve LLM accuracy by reranking the input before the LLM is invoked. The proposed pipeline is two-stage: offline, the paper estimates a position-dependent "exposure" function X_L(i) for the target LLM, and online it estimates query-dependent relevance scores for each input element using a smaller helper LLM. Relevance estimation is formulated as a bipartite debiasing problem and solved by an iterative Sinkhorn-style matrix scaling, with Theorem 1 claiming convergence. Experiments cover a synthetic graph-degree task and three real-world database aggregation tasks, using GPT-3.5 Turbo and GPT-4o Mini as target LLMs and five open-weight models as helpers. The paper reports ranking utilities approaching the optimal ordering and output errors close to a normalized optimum, with the abstract claiming up to 99% proximity to the optimal upper bound.

Significance. If the central claims hold, the paper offers a useful black-box wrapper for symmetric data-management tasks: it requires no access to the target LLM's internals and only a cheap helper model at query time. The bipartite debiasing formulation is a clean adaptation of matrix scaling to pointwise relevance estimation, and the convergence argument is essentially sound modulo a support-condition detail. The problem formulation is clear and the experimental scope (two commercial LLMs, five helper models, four task settings) is reasonable for a first study. However, the main intellectual load is carried by an unvalidated exposure model and an asserted transfer of exposure learned on a synthetic token-counting task to downstream tasks. The paper currently does not establish either the functional form in Eq. (9) or the cross-task consistency claimed in Section 6.2, so the abstract's 99% proximity figure is not yet backed by the evidence.

major comments (4)
  1. [Section 4, Eq. (9)] The exposure model 1/E[ε] ∝ (1/n) Σ X_L(i) · Rel_q(t_i) is stated without derivation or validation. Expected error is naturally a sum of per-element miss probabilities, and the reciprocal of such a sum is not generally linear in an exposure-weighted relevance sum. Since this equation is the basis for the least-squares estimator in Eq. (10) and for the entire reranking utility in Eq. (2), it is load-bearing. The paper should either derive this form from a concrete generative model of LLM misses, or validate it empirically by comparing predicted inverse errors with observed inverse errors on held-out permutations. Without that, the estimated X_L may be an artifact of the assumed functional form rather than a meaningful position-exposure curve.
  2. [Section 5.5 and Section 6.2] The transfer of exposure from the synthetic token-counting task in Section 5.1 to graph-degree and database tasks is asserted but not demonstrated. Figure 4, which is the only direct transfer evidence, reports GPT-4o Mini, two datasets, no variance or error bars, and no comparison against a task-specific exposure baseline. Section 6.2 claims "our experiments demonstrate that this recall pattern is consistent across different tasks within the same category," but no experiment in the paper measures the recall pattern on a downstream task; the cited Machlab et al. [28] actually reports that recall is heavily dependent on prompt structure and content. A direct validation is needed: estimate exposure on each downstream task (e.g., by the window-placement procedure) and compare reranking with transferred exposure, task-specific exposure, and no exposure. This is essential because the abstract's improvement claim depends on the transferred exposure being valid.
  3. [Section 5.1 and Tables 1–2] There is a mismatch between the proposed estimator and the experimental exposure discovery. Section 4.1 derives a least-squares solution for X from Eq. (10), but Section 5.1 instead measures exposure as the inverse of the average error when relevant tokens are placed in windows, and no confidence intervals from Eqs. (12)–(13) are reported. Additionally, Table 1 computes ranking utility using an arbitrary exposure X_L(i)=1/i, not the learned exposure, while Table 2 normalizes errors so that Random is 1.00 and Optimum is 0.00 in every subtable. The abstract's "up to 99% proximity to the optimum" therefore appears to refer to the ranking-utility metric with a fixed 1/i exposure, not to the final accuracy improvement obtained with the learned exposure. The paper should report absolute errors, state explicitly whether Table 2 uses the learned exposure or relevance-only sorting, and separate the claims about relevance estimation from the claims about exposure transfer.
  4. [Theorem 1 and Section 3.2] The convergence proof identifies the iterative updates with Sinkhorn's algorithm, but the stated support condition ("at least one positive diagonal") is not sufficient for Sinkhorn--Knopp convergence; the support of W must be fully indecomposable (or at least have total support). This is likely satisfied for random shuffles but should be stated correctly. More importantly, the bias model in Eq. (3) assumes that each evaluation E_j has a single multiplicative bias β_j shared by all elements in that partition. This is a strong assumption that is not tested, and the fixed point of the scaling is only meaningful if the assumed bias structure holds. The authors should at least show a robustness check, e.g., correlation between the debiased scores and ground-truth relevance on a labeled subset.
minor comments (5)
  1. [Figure 3] The axis labels contain typos: "Relative Posi ion in Promp " and "Expos re" should be "Relative Position in Prompt" and "Exposure".
  2. [Section 5.5, Figure 4] The figure reports single-point errors without error bars or statistical significance tests, although each value is stated to be an average of 10 runs; adding standard deviations or confidence intervals would materially strengthen the transfer claim.
  3. [Section 5 experimental setup] The paper states that the code is "accessible through this repository" but does not provide a URL; a working link should be included for reproducibility.
  4. [Tables 2a/2b] The normalization procedure is underspecified: "errors are normalized for each helper model to align them in the interval [0,1]" does not say whether the normalization preserves absolute differences or forces Random and Optimum to 1.00 and 0.00; the original, unnormalized errors should be reported in addition.
  5. [Section 4.2] The confidence-interval analysis in Eqs. (11)–(13) is not used anywhere in the experiments; reporting the estimated variances for the exposure curves in Figure 3 would connect the theory to the measurements.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: exposure is learned on a separate synthetic calibration task and the reranking pipeline is evaluated on held-out graph/database output errors.

full rationale

The derivation chain is not closed by construction. The exposure vector X_L is estimated in Section 5.1 on a synthetic token-counting task using Eqs. 9-10 and least squares, and then applied to graph-degree and database tasks; the downstream errors in Tables 2a/2b are measured LLM output errors on held-out tasks, not the same values used to fit X_L. Relevance scores are produced by a separate helper model (Sections 3.1-3.2 and 5.3), and Table 1 evaluates ranking utility against true relevance under the fixed exposure 1/i, which is a standard ranking-quality metric rather than a self-fulfilling prediction. The only transfer assumptions, that exposure generalizes across symmetric tasks (Section 5.5) and that Eq. 9 is a valid error model, are unproven heuristics and are explicitly contested by the cited Machlab et al. [28] result discussed in Section 6.2, but an unsupported assumption is a correctness and robustness risk, not circularity. The paper contains no load-bearing self-citations: its cited external results (Sinkhorn's algorithm, the exposure-utility model of Singh and Joachims, Lost-in-the-Middle findings) are independent and are not author-owned. The Section 6.2 sentence claiming cross-task consistency of recall patterns asserts the missing experiment, but asserting that an experiment exists does not make the result equivalent to its input by definition. Accordingly, no fitted parameter is renamed as a prediction and no equation reduces to its own inputs; the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim depends on the empirical validity of the exposure model and its transferability, plus the helper bias model. These are stated as assumptions without independent verification; the standard math (Sinkhorn, least squares) is sound.

free parameters (3)
  • Exposure vector X_L(i) = not reported numerically; shown as curves in Fig 3
    Fitted by least squares to observed inverse errors on a calibration token-counting task (Eq 10).
  • Number of permutations p for exposure estimation = not specified
    Chosen by hand; increasing p reduces variance but costs API calls. The paper does not report exact p.
  • Number of shuffles sigma and partitions m for relevance estimation = not specified (only 'small value of sigma')
    Hyperparameters controlling helper LLM query budget and estimation quality; values not reported.
assumptions (5)
  • domain assumption Position-dependent exposure X_L(i) is stable across symmetric tasks for a given LLM
    Core to the transfer of exposure learned in Section 4 to graph and database tasks in Section 5.5; not directly validated by an ablation or analysis.
  • domain assumption Inverse error is proportional to the sum of exposure-weighted relevance (Eq 9)
    Heuristic linear model introduced in Section 4 without derivation; used as the basis for least-squares exposure estimation.
  • domain assumption Each helper evaluation E_j has a constant multiplicative bias beta_j for all elements in its partition
    Key identifiability assumption in Section 3.2 that allows the bipartite debiasing; not empirically verified.
  • standard math Sinkhorn-Knopp convergence for nonnegative matrix scaling
    Used in Theorem 1 proof to claim iterative debiasing converges; standard result.
  • standard math Least-squares normal equations have a solution (R R^T invertible)
    Used in Section 4.1 to compute exposure vector; requires p >= n and full row rank, which is plausible with random permutations.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks." pith.science (2026). https://pith.science/paper/MK3YKCS4

@misc{pith2026241200546,
  author       = {Pith},
  title        = {Pith review of: Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MK3YKCS4}},
  note         = {Machine review of arXiv:2412.00546}
}
abstract

Large language models (LLMs) have quickly emerged as practical and versatile tools that provide new solutions for a wide range of domains. In this paper, we consider the application of LLMs on symmetric tasks where a query is asked on an (unordered) bag of elements. Examples of such tasks include answering aggregate queries on a database table. In general, when the bag contains a large number of elements, LLMs tend to overlook some elements, leading to challenges in generating accurate responses to the query. LLMs receive their inputs as ordered sequences. However, in this problem, we leverage the fact that the symmetric input is not ordered, and reordering should not affect the LLM's response. Observing that LLMs are less likely to miss elements at certain positions of the input, we introduce the problem of LLM input reranking: to find a ranking of the input that maximizes the LLM's accuracy for the given query without making explicit assumptions about the query. Finding the optimal ranking requires identifying (i) the relevance of each input element for answering the query and (ii) the importance of each rank position for the LLM's attention. We develop algorithms for estimating these values efficiently utilizing a helper LLM. We conduct comprehensive experiments on different synthetic and real datasets to validate our proposal and to evaluate the effectiveness of our proposed algorithms. Our experiments confirm that our reranking approach improves the accuracy of the LLMs on symmetric tasks by up to $99\%$ proximity to the optimum upper bound.

Figures

Figures reproduced from arXiv: 2412.00546 by the authors.

Figure 1
Figure 1. Illustrating the average error of GPT-3.5 Turbo on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An example of the bipartite representation of eval [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Exposure values for ’GPT-3.5 Turbo’ and ’GPT-4o [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Verifying the effect of the exposure function [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

46 extracted references · 20 canonical work pages

  1. [28]

    Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics 12 (2024), 157–173

    2024

  2. [1]

    Barry Becker and Ronny Kohavi. 1996. Adult. UCI Machine Learning Repository. DOI: https://doi.org/10.24432/C5XW20

    doi:10.24432/c5xw20 1996

  3. [2]

    Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long- document transformer. arXiv preprint arXiv:2004.05150 (2020)

    arXiv 2020

  4. [3]

    Tom B Brown. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 (2020)

    arXiv 2020

  5. [4]

    Anujit Chakraborty, Jatin Jindal, and Swaprava Nath. 2018. Removing bias and incentivizing precision in peer-grading. arXiv preprint arXiv:1807.11657 (2018)

    work page Pith review arXiv 2018

  6. [5]

    Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. 2023. Walking down the memory maze: Beyond context limit through interactive reading. arXiv preprint arXiv:2310.05029 (2023)

    arXiv 2023

  7. [6]

    Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. 2023. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306.15595 (2023)

    arXiv 2023

  8. [7]

    Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. 2019. Generating long sequences with sparse transformers. arXiv preprint arXiv:1904.10509 (2019)

    arXiv 2019

Show all 46 references
  1. [8]

    Kwangsu Cho and Christian D Schunn. 2007. Scaffolded writing and rewriting in the discipline: A web-based reciprocal peer review system. Computers & Education 48, 3 (2007), 409–426

    2007

  2. [9]

    Luca De Alfaro, Michael Shavlovsky, and Vassilis Polychronopoulos. 2016. Incen- tives for truthful peer grading. arXiv preprint arXiv:1604.03178 (2016)

    2016 arXiv

  3. [10]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 (2024)

    2024 arXiv

  4. [11]

    Paul Erdos, Alfréd Rényi, et al. 1960. On the evolution of random graphs. Publ. math. inst. hung. acad. sci 5, 1 (1960), 17–60

    1960

  5. [12]

    Jianfei Gao, Yangze Zhou, Jincheng Zhou, and Bruno Ribeiro. 2023. Double equivariance for inductive link prediction for both new nodes and new relation types. NeurIPS (2023)

    2023

  6. [13]

    John Hamer, Kenneth TK Ma, and Hugh HF Kwong. 2005. A method of auto- matic grade calibration in peer assessment. In Proceedings of the 7th Australasian conference on Computing education-Volume 42. 67–72

    2005

  7. [14]

    Junqing He, Kunhao Pan, Xiaoqun Dong, Zhuoyang Song, LiuYiBo LiuYiBo, Qian- guosun Qianguosun, Yuxin Liang, Hao Wang, Enming Zhang, and Jiaxing Zhang

  8. [15]

    Cheng-Yu Hsieh, Yung-Sung Chuang, Chun-Liang Li, Zifeng Wang, Long Le, Ab- hishek Kumar, James Glass, Alexander Ratner, Chen-Yu Lee, Ranjay Krishna, et al

  9. [16]

    Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar Shah, Vincent Conitzer, and Fei Fang. 2020. Mitigating manipulation in peer review via randomized reviewer assignments. Advances in Neural Information Processing Systems 33 (2020), 12533– 12545

    2020

  10. [17]

    In Findings of the Association for Computational Linguistics ACL 2024

    Found in the middle: Calibrating Positional Attention Bias Improves Long Context Utilization. In Findings of the Association for Computational Linguistics ACL 2024. 14982–14995

    2024

  11. [18]

    Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. [n.d.]. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. In The 2023 Conference on Empirical Methods in Natural Language Pro- cessing

    2023

  12. [19]

    Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, De- vendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al . 2023. Mistral 7B. arXiv preprint arXiv:2310.06825 (2023)

    2023 arXiv

  13. [20]

    Jakub Kuzilek, Martin Hlosta, and Zdenek Zdrahal. 2017. Open university learning analytics dataset. Scientific data 4, 1 (2017), 1–8

    2017

  14. [21]

    Huiqiang Jiang, Qianhui Wu, Xufang Luo, Dongsheng Li, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. 2023. Longllmlingua: Accelerating and enhancing llms in long context scenarios via prompt compression. arXiv preprint arXiv:2310.06839 (2023)

    2023 arXiv

  15. [22]

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2024. Long-context llms struggle with long in-context learning. arXiv preprint arXiv:2404.02060 (2024)

    2024 arXiv

  16. [23]

    Rui Li, Jianlin Su, Chenxi Duan, and Shunyi Zheng. 2020. Linear attention mechanism: An efficient attention for semantic segmentation. arXiv preprint arXiv:2007.14902 (2020)

    2020 arXiv

  17. [24]

    Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michi- hiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al

  18. [25]

    Yucheng Li. 2023. Unlocking context constraints of llms: Enhancing context efficiency of llms with self-information-based content filtering. arXiv preprint arXiv:2304.12102 (2023)

    2023 arXiv

  19. [26]

    Yusha Liu, Yichong Xu, Nihar B Shah, and Aarti Singh. 2022. Integrating rankings into quantized scores in peer review. arXiv preprint arXiv:2204.03505 (2022)

    2022 arXiv

  20. [27]

    Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies . 142–150

    2011

  21. [29]

    Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713 (2020)

    2020 arXiv

  22. [30]

    Devendra Singh Sachan, Mike Lewis, Mandar Joshi, Armen Aghajanyan, Wen-tau Yih, Joelle Pineau, and Luke Zettlemoyer. 2022. Improving passage retrieval with zero-shot question generation. arXiv preprint arXiv:2204.07496 (2022)

    2022 arXiv

  23. [31]

    Daniel Machlab and Rick Battle. 2024. LLM In-Context Recall is Prompt Depen- dent. arXiv preprint arXiv:2404.08865 (2024)

    2024 arXiv

  24. [32]

    Ashudeep Singh and Thorsten Joachims. 2018. Fairness of exposure in rankings. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining . 2219–2228

    2018

  25. [33]

    Richard Sinkhorn. 1967. Diagonal equivalence to matrices with prescribed row and column sums. The American Mathematical Monthly 74, 4 (1967), 402–405

    1967

  26. [34]

    Nihar B Shah, Joseph K Bradley, Abhay Parekh, Martin Wainwright, and Kannan Ramchandran. 2013. A case for ordinal peer-evaluation in MOOCs. In NIPS workshop on data driven education , Vol. 15. 67

    2013

  27. [35]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118 (2024)

    2024 arXiv

  28. [36]

    Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2021. An explanation of in-context learning as implicit bayesian inference. arXiv preprint arXiv:2111.02080 (2021)

    2021 arXiv

  29. [37]

    Richard Sinkhorn and Paul Knopp. 1967. Concerning nonnegative matrices and doubly stochastic matrices. Pacific J. Math. 21, 2 (1967), 343–348. 12 Rank It, Then Ask It: Input Reranking for Maximizing the Performance of LLMs on Symmetric Tasks SIGMOD-Companion ’25, ,

    1967

  30. [38]

    Yue Yu, Wei Ping, Zihan Liu, Boxin Wang, Jiaxuan You, Chao Zhang, Mohammad Shoeybi, and Bryan Catanzaro. 2024. RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs. arXiv preprint arXiv:2407.02485 (2024)

    2024 arXiv

  31. [39]

    Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems 33 (2020), 17283–17297

    2020

  32. [40]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Cheng- peng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671 (2024)

    2024 arXiv

  33. [41]

    Shengyao Zhuang, Hang Li, and Guido Zuccon. 2021. Deep query likelihood model for information retrieval. In Advances in Information Retrieval: 43rd Euro- pean Conference on IR Research, ECIR 2021, Virtual Event, March 28–April 1, 2021, Proceedings, Part II 43 . Springer, 463–470

    2021

  34. [42]

    Shengyao Zhuang, Honglei Zhuang, Bevan Koopman, and Guido Zuccon. 2024. A setwise approach for effective and highly efficient zero-shot ranking with large language models. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information ...

    2024

  35. [43]

    Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, et al . 2024. DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence.arXiv preprint arXiv:2406.11931 (2024)

    2024 arXiv

  36. [46]

    Shengyao Zhuang and Guido Zuccon. 2021. TILDE: Term independent likelihood moDEl for passage re-ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval . 1483–1492. 13

    2021

  37. [2022]

    arXiv preprint arXiv:2211.09110 (2022)

    Holistic evaluation of language models. arXiv preprint arXiv:2211.09110 (2022)

    2022 arXiv

  38. [2024]

    In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Never Lost in the Middle: Mastering Long-Context Question Answering with Position-Agnostic Decompositional Training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 13628–13642

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.