Pith. sign in

REVIEW 2 major objections 5 minor 38 references

Post-Calibration Reliability Reranking of Relevance Decisions via Label-wise Monotone Projection

T0 review · 2 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that after probability calibration, the same calibrated confidence does not imply the same correctness reliability across predicted relevance labels, and that a label-wise monotone projection (MRP) improves…

desk verdict A well-executed post-calibration reranking method whose label-conditioned gains are real, but the load-bearing monotonicity assumption is asserted, not tested. read the letter →

arxiv 2608.10406 v1 pith:PDPJCY6J submitted 2026-08-11 cs.IR cs.LG

Inwoo Tae , Yongjae Lee This is my paper · ORCID
classification cs.IRcs.LG
keywords reliabilityrerankingprobabilitycalibrationselectivepredictionlabel-wisemonotoneprojectionrelevancefallbackroutingexpectederrorinformationaccesssystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Calibration aligns confidence with average correctness, but the paper argues this is not the end of the reliability problem: predictions that share the same calibrated confidence can still differ in how likely they are to be correct, depending on which relevance label was predicted. The proposed Label-wise Monotone Reliability Projection (MRP) learns one monotone function per predicted label that maps calibrated confidence into a correctness-reliability score, then reranks the fixed predictions by that score. This matters because deployed systems use confidence to decide which results to trust, route to fallback, or review; a better risk ordering among equally confident predictions directly improves such decisions. Across six information access datasets and several post-hoc calibrators, MRP improves correctness negative log-likelihood and area under the risk-coverage curve, improves average fallback utility, and improves error-detection AUPR when label-conditioned residual signal exists, while preserving full-coverage accuracy and expected calibration error by construction.

What carries the argument

The central object is the label-wise monotone reliability function $T_k: [0,1] \to [0,1]$ for each predicted label $k$, implemented as a one-dimensional monotone lattice over confidence knots. Each lattice is parameterized with logit-scale values built from nonnegative increments, so monotonicity $c \leq c' \Rightarrow T_k(c) \leq T_k(c')$ holds throughout optimization, and linear interpolation between knots gives smooth curves. The objective is binary cross-entropy of the correctness indicator $Z$ against $T_k(\hat{c}^A)$ plus a penalty on second differences that discourages oscillation. This map carries the argument: it converts calibrated confidence into a decision-conditioned reliability score used only for reranking, and the paper's ablations show that label conditioning, rather than a shared confidence remapping, is what produces the reranking gains.

What would settle it

On a validation split, bin predictions by predicted label and calibrated confidence and compute the empirical correctness rate per bin; if for any predicted label the empirical rate falls as calibrated confidence rises across a meaningful range of bins, the monotone constraint forces MRP to miss or invert that structure, and an unconstrained label-wise estimate would outperform MRP on reliability-reranking metrics in that setting.

Watch

Extended reading notes

Core claim

The paper's central claim is that post-hoc calibration only fixes the average meaning of confidence, not the label-conditioned correctness structure that remains inside each confidence level. Formally, the null hypothesis $P(Z=1 \mid \hat{C}^A=c, D=k) = P(Z=1 \mid \hat{C}^A=c)$ for all $c,k$ fails empirically, and MRP is designed for that failure. MRP estimates $\hat{q}_{\mathrm{MRP}}(x) = T_{d(x)}(\hat{c}^A(x))$, where each $T_k$ is a monotone confidence-to-reliability curve learned by binary cross-entropy on the correctness event with a smoothness regularizer. Because the projection leaves the calibrated probability vector, the predicted label, the calibrated confidence, and the correctness event unchanged, full-coverage accuracy and calibration error are preserved by design; only the ordering of fixed predictions by estimated error probability $1 - \hat{q}_{\mathrm{MRP}}$ changes. The paper reports broad improvements in reliability-reranking metrics and positive average selective-accuracy gains under a fallback budget, with gains concentrated where label-conditioned residual reliability is present.

Load-bearing premise

The load-bearing premise is that, within each fixed predicted relevance label, correctness reliability is monotone in calibrated confidence: a higher-confidence prediction is never less reliable than a lower-confidence one, so restricting each label-wise map to nondecreasing curves cannot throw away true structure.

Editorial extensions

If this is right

  • Systems that route predictions to fallback or review can keep their calibrated model unchanged and still make better use of a limited budget by replacing the ordering confidence with the MRP reliability score.
  • Because MRP is a layer on top of any probability-producing calibrator, it adds a reliability-reranking stage without forcing a choice among calibrators.
  • Calibration and reliability reranking measure different things: a calibrator can sharply lower ECE while leaving risk ordering nearly unchanged, so both should be reported for information access systems.
  • The label-wise 2D variant that adds the top-runner logit gap performs about the same as label-wise 1D, indicating that the decision label plus calibrated confidence captures most of the residual structure in these datasets.
  • The method's headroom shrinks when label-conditioned residual reliability is weak, as in SciDocs, where confidence already carries most of the useful signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A direct diagnostic the paper does not run is fitting an unconstrained, non-monotone label-wise reliability curve per label; if any such curve has a decreasing segment, the monotone constraint discards real structure and MRP's gains are a lower bound on what a fully flexible label-wise score could achieve.
  • Because MRP produces a separate score, it could plausibly be combined with other per-query signals, such as retrieval scores or query difficulty estimates, to build a richer fallback router; the paper only conditions on the predicted label and calibrated confidence.
  • The MRC simplex analysis suggests a general compatibility test for decision-level reliability scores: check whether the score can be realized as a top-label probability on the power-temperature path, which tells designers when a reliability score can be embedded back into class-probability geometry.
  • In a deployed setting, one could use the label-wise spread of residuals on held-out data as a cheap detector of whether a new domain needs MRP at all, since small spread predicts little reranking gain.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper studies the problem of reliability reranking after probability calibration. Under a fixed-decision protocol, where the top-label decision d(x) is fixed from the uncalibrated predictor and a post-hoc calibrator A only changes the assigned confidence c^A(x), the authors propose Label-wise Monotone Reliability Projection (MRP). MRP fits per-label monotone functions T_k(c) mapping calibrated confidence to the probability that the fixed decision is correct, using binary cross-entropy with a second-difference regularizer on a lattice parameterization. The calibrated probability vector, predicted label, full-coverage accuracy, and ECE are unchanged by construction; only the ranking for fallback/review is altered. Experiments over six information-access relevance datasets and six calibrators report improvements in correctness NLL, AUPR-Error, AURC, and selective accuracy, with structural ablations attributing the gain to label conditioning. A secondary analysis (MRC) explores when the reliability score can be embedded into top-label probability geometry via sample-wise power-temperature paths. The authors acknowledge the method's scope limitations and provide a robustness check with a stronger base predictor.

Significance. The paper addresses a genuine and under-studied gap: after top-label calibration, equal calibrated confidence can carry different correctness reliability across predicted labels, and reranking fixed decisions by residual risk is a distinct task from calibrating class probabilities. The fixed-decision protocol is cleanly defined, and the construction guarantees preservation of accuracy and ECE by design, which the paper correctly states. The ablations in Table 5 (Shared 1D, label-only intercept, per-label isotonic, 2D) are well chosen and support the claim that label conditioning, rather than a global confidence remap, drives the reranking gains. The use of six datasets, multiple calibrators, and a three-seed protocol indicates a careful empirical effort, and the authors honestly report where gains are weak (e.g., SciDocs). The main weakness is that the central monotonicity assumption in Section 3.3 is asserted without direct validation; Table 4's spread statistic is orthogonal to monotonicity. If the empirical label-conditional reliability curves are non-monotone, the proposed projection is misspecified. This tempers, but does not eliminate, the significance of the contribution.

major comments (2)
  1. [Section 3.3 (and Figure 2, Table 4, Section 3.5)] The monotonicity constraint c <= c' implies T_k(c) <= T_k(c') is the structural core of MRP, yet the manuscript never tests whether the empirical label-conditional reliability curves f_k(c) = P(Z=1 | C=c, D=k) are in fact non-decreasing. The marginal top-label calibration condition P(Z=1 | C=c) = c from Section 3.1 is compatible with non-monotone f_k, so the justification that higher calibrated confidence does not imply lower correctness reliability is not a derived property. Table 4 measures cross-label spread of Z - c within confidence groups, which is orthogonal to monotonicity, and Figure 2 plots only the fitted monotone curves, not binned empirical reliabilities. All fitted variants in Table 5 (Shared 1D, label-only intercept, per-label isotonic, MRP, Label-wise 2D) are monotone or constant in c, so the ablations cannot detect a violation. If f_k is non-monotone for some label, the monotone projection is misspecified and the reported reranking gains could be attenuated or reversed relative to an unconstrained per-label estimator. I recommend adding binned empirical reliability curves per label for representative datasets and calibrators, testing monotonicity directly (e.g., a non-parametric check or a comparison against an unconstrained per-label spline or logistic model), and reporting whether any non-monotone patterns appear. This validation is necessary to support the central claim.
  2. [Table 2 and Section 4.3] Many entries in Table 2 report a standard deviation of exactly 0.000 across the three protocol seeds, including MRP values that depend on the fitted projection (e.g., ESCI-Rerank-US NLLcorrect 0.647±0.000 for all calibrators; WANDS and Amazon ESCI rows). Section 4.3 states that for each protocol seed the validation data are re-split into calibrator-fit, projection-fit, and projection-selection subsets, so the fitted MRP curves should differ across seeds and the test metrics should show nonzero variance. The zero variance suggests either that the seed affects almost nothing in the reported metric, or that the reported numbers come from a single seed. Please report per-seed values or clarify exactly what the protocol seed changes and why the variance vanishes; if the three-seed protocol in fact averages over a deterministic procedure, this should be stated explicitly. This matters because the paper uses the mean±std presentation to support the stability and generality of the improvements.
minor comments (5)
  1. [Figure 1] The overview figure is visually dense and the arrows are difficult to follow; consider enlarging the figure and simplifying the text labels to make the reliability reranking concept clearer.
  2. [Table 4] The 'Random' column uses the 95th percentile of the shuffled-label distribution as a null comparison; a mean or median over shuffles would be a more standard choice. Please justify the percentile choice or report the full distribution.
  3. [Section 4.3] Only one base predictor per dataset (model seed 0) is used; the robustness check on MSLR-WEB10K with LightGBM is welcome, but reporting results for a second base predictor on at least one more dataset would strengthen the generality claims.
  4. [Section 3.4] The hyperparameters J=8 and rho=1e-4 are fixed without sensitivity analysis; a short sensitivity check over, for example, J in {4,12} and rho in {1e-3,1e-5} would help assess robustness of the reported gains.
  5. [Section 4.2] The description of the DIAG calibrator is brief; for reproducibility, please provide the exact objective or pseudocode used for the diagonal intra-order-preserving calibration in the fixed-decision evaluation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: MRP is a held-out fitted reliability model; its invariance claims are definitional but not presented as empirical predictions.

full rationale

The derivation chain is self-contained. MRP's label-wise monotone maps T_k are estimated on a dedicated projection-fit subset by minimizing a binary cross-entropy objective (Eq. 1) against the correctness indicators Z_i, with model selection on a separate projection-selection subset, and evaluation on a test set that is touched only after all parameters and hyperparameters are fixed (Section 4.3). The reported reliability-reranking gains (NLLcorrect, AUPR-Error, AURC, SelAcc) therefore compare held-out predictions against the confidence baseline q = c and are not forced by construction. The claim that MRP preserves pA, d, c, and Z is an explicit design invariant rather than a fitted result, and the paper does not present it as an empirical discovery. The monotonicity constraint in Section 3.3 is an untested structural assumption; if the true label-conditional reliability functions are non-monotone, the projection could be misspecified, but this is a correctness and robustness concern, not circularity. The ablations include a per-label isotonic variant and a label-only intercept, and the label-wise 2D variant is compared against the 1D method, so the label-conditioning conclusion is supported by relative comparisons rather than by reusing fitted values as evidence. No load-bearing self-citation or uniqueness theorem is invoked: all cited related work is external and none is required to justify the MRP objective. No step in the paper reduces a predicted quantity to its own input by definition.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim depends on hyperparameters (J knots, rho regularizer, subsampling cap) chosen by hand, on the domain assumption of monotonicity, and on the fixed-decision protocol. No invented entities are introduced. The method is an empirical fitting procedure, so free parameters are standard but the monotonicity assumption is the most fragile structural choice.

free parameters (3)
  • Number of confidence knots J = 8
    Hyperparameter for the monotone lattice; set to 8 across all datasets. Chosen by hand, not tuned per dataset.
  • Second-difference regularizer weight rho = 1e-4
    Regularization coefficient in Eq. (1) to stabilize label-wise curves; fixed across experiments.
  • Projection-fit and projection-selection cap = 8000 samples
    Subsampling cap for computational comparability; chosen by hand.
assumptions (3)
  • domain assumption For each predicted label k, the reliability function T_k is monotone nondecreasing in calibrated confidence c (Section 3.3).
    This is a modeling assumption, not derived from data or theory. If violated, the projection cannot represent the true reliability and the reranking gains could diminish. The paper does not validate this against non-monotone alternatives.
  • domain assumption The fixed-decision protocol: the calibrator and MRP only modify confidence, never the predicted label d(x) (Section 3.1).
    The paper evaluates only the reliability of already-made decisions; it cannot address errors that require changing the decision. This bounds the scope of the claim.
  • standard math Correctness of a prediction is Bernoulli distributed independently given confidence and label; BCE loss is the appropriate objective (Eq. 1).
    Standard statistical assumption for binary correctness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Post-Calibration Reliability Reranking of Relevance Decisions via Label-wise Monotone Projection." pith.science (2026). https://pith.science/paper/PDPJCY6J

@misc{pith2026260810406,
  author       = {Pith},
  title        = {Pith review of: Post-Calibration Reliability Reranking of Relevance Decisions via Label-wise Monotone Projection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PDPJCY6J}},
  note         = {Machine review of arXiv:2608.10406}
}
read the original abstract

Web search, product search, and question-answering retrieval systems often assign a relevance label and confidence score to each query-candidate pair. The relevance label describes how well a page, product, or passage matches the query, while the confidence often guides downstream use or fallback decisions. Post-hoc calibration is therefore needed because misaligned confidence can make systems over-trust wrong predictions or unnecessarily defer correct ones. However, calibration mainly aligns confidence with average correctness, and does not remove predicted-label-dependent reliability differences that remain within the same calibrated confidence level. We address this gap with Label-wise Monotone Reliability Projection (MRP), which learns label-wise monotone functions that map calibrated confidence to correctness reliability while preserving the original predicted labels and class probabilities. The resulting reliability score reranks fixed predictions according to residual risk. Across six information access relevance datasets and multiple post-hoc calibrators, MRP improves reliability reranking and average fallback utility while preserving full-coverage accuracy and ECE. Structural ablations show that the main gains come from label-wise residual reliability rather than from global confidence remapping. We further analyze when MRP reliability scores can be embedded back into top-label probability geometry, showing that this projection is useful as a compatibility analysis but is distinct from the main reliability-reranking objective. The implementation will be made publicly available.

Figures

Figures reproduced from arXiv: 2608.10406 by the authors.

Figure 1
Figure 1. Overview of MRP for post-calibration reliability reranking. Each row is a fixed query-candidate relevance prediction. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Learned label-wise reliability curves 𝑇𝑘 (𝑐), shown under SMART calibration as a representative base calibrator. The dashed line is the identity 𝑞 = 𝑐. The curves show that equal calibrated confidence can correspond to different correctness reliability depending on the predicted relevance label [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Schematic of MRC on the probability simplex. The [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

38 extracted references · 22 canonical work pages

  1. [1]

    Yan Chen, Shujian Liu, Zheng Liu, Weiyi Sun, Linas Baltrunas, and Benjamin Schroeder. 2022. Wands: Dataset for product search relevance assessment. In European Conference on Information Retrieval. Springer, 128–141

    work page 2022

  2. [2]

    Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld

  3. [3]

    Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord, and Patrick Pérez. 2019. Addressing failure prediction by learning model confidence.Advances in neural information processing systems32 (2019)

    work page 2019

  4. [4]

    Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. 2016. Learning with rejec- tion. InInternational conference on algorithmic learning theory. Springer, 67–82

    work page 2016

  5. [5]

    Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. 2021. Ms marco: Benchmarking ranking models in the large-data regime. InProceedings of the 44th international ACM SIGIR conference on research and development in information retrieval. 1566–1576

    work page 2021

  6. [6]

    Ran El-Yaniv et al. 2010. On the Foundations of Noise-free Selective Classification. Journal of Machine Learning Research11, 5 (2010)

    work page 2010

  7. [7]

    Yonatan Geifman and Ran El-Yaniv. 2017. Selective classification for deep neural networks.Advances in neural information processing systems30 (2017)

    work page 2017

  8. [8]

    Yonatan Geifman and Ran El-Yaniv. 2019. Selectivenet: A deep neural network with an integrated reject option. InInternational conference on machine learning. PMLR, 2151–2159

    2019

Show all 38 references
  1. [9]

    Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. InInternational conference on machine learning. PMLR, 1321–1330

    2017

  2. [10]

    Haolan Guo, Linwei Tao, Haoyang Luo, Minjing Dong, and Chang Xu. 2025. Sample Margin-Aware Recalibration of Temperature Scaling.arXiv preprint arXiv:2506.23492(2025)

    2025 arXiv

  3. [11]

    Chirag Gupta and Aaditya Ramdas. 2022. Top-label calibration and multiclass-to- binary reductions. InInternational Conference on Learning Representations

    2022

  4. [12]

    Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink, Cristian Sminchisescu, and Richard Hartley. 2020. Calibration of neural networks using splines.arXiv preprint arXiv:2006.12800(2020)

    2020 arXiv

  5. [13]

    Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. InInternational conference on machine learning. PMLR, 3929–3938

    2020

  6. [14]

    Jakob Heiss, Sören Lambrecht, Jakob Weissteiner, Hanna Wutte, Žan Žurič, Josef Teichmann, and Bin Yu. 2026. JUCAL: Jointly Calibrating Aleatoric and Epistemic Uncertainty in Classification Tasks.arXiv preprint arXiv:2602.20153(2026)

    2026

  7. [15]

    Dan Hendrycks and Kevin Gimpel. 2016. A baseline for detecting misclassified and out-of-distribution examples in neural networks.arXiv preprint arXiv:1610.02136 (2016)

    2016 arXiv

  8. [16]

    Wenjian Huang, Guiping Cao, Jiahao Xia, Jingkun Chen, Hao Wang, and Jianguo Zhang. 2025. h-calibration: Rethinking Classifier Recalibration with Probabilistic Error-Bounded Objective.IEEE Transactions on Pattern Analysis and Machine Intelligence(2025)

    2025

  9. [17]

    Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. InProceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume. 874–880

    2021

  10. [18]

    Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. InProceedings of the 2020 conference on empirical methods in natural language processing (EMNLP)...

    2020

  11. [19]

    Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach. 2019. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration.Advances in neural information processing systems32 (2019)

    2019

  12. [20]

    Ananya Kumar, Percy S Liang, and Tengyu Ma. 2019. Verified uncertainty calibration.Advances in neural information processing systems32 (2019)

    2019

  13. [21]

    Aviral Kumar, Sunita Sarawagi, and Ujjwal Jain. 2018. Trainable calibration measures for neural networks from kernel mean embeddings. InInternational Conference on Machine Learning. PMLR, 2805–2814

    2018

  14. [22]

    Antoine Lefebvre-Brossard, Stephane Gazaille, and Michel C Desmarais. 2023. Alloprof: a new french question-answer education dataset and its use in an information retrieval case study.arXiv preprint arXiv:2302.07738(2023)

    2023 arXiv

  15. [23]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing s...

    2020

  16. [24]

    Tie-Yan Liu. 2009. Learning to Rank for Information Retrieval.Foundations and Trends in Information Retrieval3, 3 (2009), 225–331. doi:10.1561/1500000016

    2009 doi

  17. [25]

    David Madras, Toni Pitassi, and Richard Zemel. 2018. Predict responsibly: improv- ing fairness and accuracy by learning to defer.Advances in neural information processing systems31 (2018)

    2018

  18. [26]

    Microsoft Research. 2010. Microsoft Learning to Rank Datasets. https://www. microsoft.com/en-us/research/project/mslr/. Accessed: 2026-05-19

    2010

  19. [27]

    Hussein Mozannar and David Sontag. 2020. Consistent estimators for learning to defer to an expert. InInternational conference on machine learning. PMLR, 7076–7087

    2020

  20. [28]

    Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. 2015. Obtaining well calibrated probabilities using bayesian binning. InProceedings of the AAAI conference on artificial intelligence, Vol. 29

    2015

  21. [29]

    Alexandru Niculescu-Mizil and Rich Caruana. 2005. Predicting good probabilities with supervised learning. InProceedings of the 22nd international conference on Machine learning. 625–632

    2005

  22. [30]

    Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv preprint arXiv:1901.04085(2019)

    2019 arXiv

  23. [31]

    John Platt et al. 1999. Probabilistic outputs for support vector machines and com- parisons to regularized likelihood methods.Advances in large margin classifiers 10, 3 (1999), 61–74

    1999

  24. [32]

    Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, and Byron Boots. 2020. Intra order-preserving functions for calibration of multi-class neural networks.Advances in neural information processing systems33 (2020), 13456– 13467

    2020

  25. [33]

    Chandan K Reddy, Lluís Màrquez, Fran Valero, Nikhil Rao, Hugo Zaragoza, Sambaran Bandyopadhyay, Arnab Biswas, Anlu Xing, and Karthik Subbian. 2022. Shopping queries dataset: A large-scale ESCI benchmark for improving product search.arXiv preprint arXiv:2206.06588(2022)

    2022 arXiv

  26. [34]

    2009.The probabilistic relevance frame- work: BM25 and beyond

    Stephen Robertson and Hugo Zaragoza. 2009.The probabilistic relevance frame- work: BM25 and beyond. Vol. 4. Now Publishers Inc

    2009

  27. [35]

    Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.arXiv preprint arXiv:2104.08663(2021)

    2021 arXiv

  28. [36]

    Juozas Vaicenavicius, David Widmann, Carl Andersson, Fredrik Lindsten, Jacob Roll, and Thomas Schön. 2019. Evaluating model calibration in classification. In The 22nd international conference on artificial intelligence and statistics. PMLR, 3459–3467

    2019

  29. [37]

    Bianca Zadrozny and Charles Elkan. 2002. Transforming classifier scores into ac- curate multiclass probability estimates. InProceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining. 694–699. 11

    2002

  30. [2020]

    InProceedings of the 58th annual meeting of the association for computational linguistics

    Specter: Document-level representation learning using citation-informed transformers. InProceedings of the 58th annual meeting of the association for computational linguistics. 2270–2282

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.