Bounded-Abstention Pairwise Learning to Rank

Andrea Pugnana; Antonio Ferrara; Francesco Bonchi; Salvatore Ruggieri

REVIEW 3 major objections 5 minor 34 references

Bounded-Abstention Pairwise Learning to Rank

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read For a fixed ranker and a symmetric loss, optimal bounded abstention in pairwise ranking is a threshold on the conditional risk: accept below the c-th quantile, reject above it.

desk verdict BALToR cleanly brings bounded abstention to pairwise ranking with ties and the core theorem holds up, but the algorithm-theory gap around plug-in risks and a sloppy appendix keep it from being more than a solid subfield contribution. read the letter →

arxiv 2505.23437 v2 pith:SPLAYFDF submitted 2025-05-29 cs.LG cs.AIcs.IR

Antonio Ferrara , Andrea Pugnana , Francesco Bonchi , Salvatore Ruggieri This is my paper

classification cs.LGcs.AIcs.IR

keywords abstentionselectivepredictionlearningtorankpairwisepreferencesconditionalriskcoverageconstrainttiesBradley-Terrymodel

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces bounded abstention to pairwise learning to rank: after a ranker is trained, a separate selection function decides which item pairs the ranker should decide and which should be deferred to a human or another model, subject to a pre-specified minimum coverage. Its central result is that, for a symmetric loss, the optimal selection function is a threshold rule: accept every pair whose conditional risk lies below the c-th risk quantile, randomize on pairs exactly at the quantile, and reject the rest. This matters because abstention mechanisms have mostly been built for classification, while ranking systems in hiring, medicine, and admissions make high-stakes comparisons that are natural candidates for deferral. The paper packages the theorem into BALToR, a model-agnostic algorithm that estimates the conditional risk as one minus the maximum predicted probability on the pair and thresholds it at the calibrated quantile, and shows on three learning-to-rank datasets that this rule raises accuracy on the accepted pairs while keeping actual coverage close to the target.

What carries the argument

The load-bearing object is the conditional risk $r(x,x') = \sum_{y} \ell(f(x,x'), y)\, p(y \mid x,x')$, the expected loss of the given ranker on a particular pair. The theorem turns abstention into a level-set rule for this quantity: $\beta$ is the $c$-th conditional-risk quantile, and the optimal selection function is $g^*(x,x') = 1$ if $r(x,x') < \beta$, $g^*(x,x') = 0$ if $r(x,x') > \beta$, and a Bernoulli draw at $r(x,x') = \beta$ chosen to satisfy coverage exactly. In the BALToR algorithm, $r$ is replaced by the plug-in estimate $1 - \max_y \hat{p}(y \mid x,x')$ for 0-1 loss, and $\beta$ is replaced by the empirical $c$-th quantile computed on a held-out calibration set. The pairwise probability estimates themselves come from tie-aware versions of the Bradley-Terry and Thurstone-Mosteller models, so the machinery connects the distribution-free optimality theorem to a concrete calibration procedure.

What would settle it

On a synthetic dataset where the true label distribution is known, train a ranker whose predicted probabilities are intentionally misordered (for example, confident on pairs that are actually ties), run BALToR with decreasing $c$, and check whether accuracy on accepted pairs falls rather than rises as coverage drops. A single such dataset where the plug-in threshold selects low-risk pairs for rejection while accepting high-risk pairs would show that the estimated-risk ordering cannot be relied on in general.

Watch

Extended reading notes

Core claim

The paper's claim is that the bounded-abstention pairwise ranking problem—minimize the selective risk $R_l(f,g) = \mathbb{E}[\ell(f(X,X'),Y)\,g(X,X')]/\mathbb{E}[g(X,X')]$ subject to coverage at least $c$, with the ranker $f$ fixed—has a complete optimal solution. For a symmetric loss, an optimal $g^*$ accepts, almost surely, every pair with conditional risk $r(x,x') < \beta$; rejects every pair with $r(x,x') > \beta$; and accepts exactly enough of the probability mass at $r(x,x') = \beta$, by randomization if needed, to make coverage exactly $c$, where $\beta$ is the $c$-th quantile of the conditional risk distribution. For the 0-1 loss, the conditional risk of the Bayes optimal ranker is $1 - \max_y p(y \mid x,x')$, which is the quantity the plug-in algorithm thresholds. This characterization extends the known optimal reject-option result from classification to pairwise ranking with ties, and the paper shows it remains usable because the symmetric-loss assumption makes the decision on $(x,x')$ and on $(x',x)$ consistent.

Load-bearing premise

The math says the perfect rule is to refuse exactly the pairs the ranker is most likely to get wrong, but the algorithm has to guess which pairs those are from the ranker's own confidence scores; the whole approach depends on those guesses being ordered correctly and on the calibration data matching the test data.

Editorial extensions

If this is right

When the ranker and loss are fixed, no abstention strategy can beat the conditional-risk threshold rule; any feasible selection function that violates one of the three conditions in Theorem 3.1 can be modified to reduce selective risk.
For the Bayes optimal ranker under 0-1 loss, the optimal abstention rule depends only on the top predicted probability, not on the full probability vector or on entropy.
The same plug-in procedure works with any ranker that supplies pairwise probabilities, since BALToR is agnostic to how the ranker was trained.
Empirically, under the tested LambdaMART implementations, accuracy on accepted pairs increases as the target coverage $c$ decreases, actual test coverage stays within about 0.002 of $c$, and rejection rates stay balanced across the three preference classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because Theorem 3.1 is stated for the true conditional risk, a natural extension is a finite-sample guarantee on the calibrated quantile; without one, the algorithm's worst-case behavior at small calibration sizes is open.
The same threshold characterization might transfer to listwise ranking if list-level risk is used as the selection score, though the pairwise symmetry argument would not carry over unchanged.
A testable consequence of relying on estimated probabilities is that deliberately miscalibrated rankers should degrade BALToR's accuracy-on-accepted-pairs curve; running the experiments on rankers with corrupted probability estimates would isolate that dependence.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: BALToR is a clean, model-agnostic way to add a bounded abstention budget to a pairwise ranker that outputs class probabilities. The main theorem is a faithful extension of Franc et al.'s selective classification result to pairwise ranking with ties, and the algorithm is simple. It is not a landmark, but it is a solid, useful contribution for the selective learning-to-rank community.

What's actually new: the bounded-abstention (coverage-constrained) formulation for pairwise ranking with ties, as opposed to Mao et al.'s cost-based rejection. The characterization of the optimal selection function for a fixed ranker under symmetric loss is correct in spirit, and the plug-in algorithm using Bradley-Terry/Thurstone-Mosteller probabilities is natural. The experiments cover three standard datasets and three GBDT rankers, and the code is public. Coverage is well calibrated: empirical coverage is within about 0.002 of target. That is real evidence.

The soft spots, in rough order of seriousness. First, there is an unstated assumption connecting theory to algorithm. Theorem 3.1 concerns the conditional risk of the fixed ranker f, i.e., 1 - p(f(x,x') | x,x'). Proposition 3.3 derives 1 - max_y p(y|x,x') only for the Bayes-optimal ranker. The algorithm thresholds 1 - max \hat p. That is the right quantity only if f is the argmax of the same probabilities used in the plug-in estimator, which is exactly what the experiments do, but the paper never states that as a condition of the algorithm. A user with a ranker that does not produce calibrated probabilities, or where f is not the argmax of \hat p, would be thresholding the wrong quantity. This should be stated and given some finite-sample or calibration justification.

Second, the appendix is sloppy. In Theorem 3.2, the definition of pr has a typo: the denominator should be the mass on r=β, not r<β. Case 2 and Case 3 of the proof of Theorem 3.1 are not fully rigorous, though the result itself is standard and almost certainly correct.

Third, the empirical gains are modest. On Web-30k the accuracy improvement from full coverage to c=0.7 is about 0.005; on OHSUMED it is about 0.016 but with standard deviations around 0.06-0.08, so it is within noise. No significance tests are reported. The coverage and class-balance results are solid, but the headline accuracy claim is weaker than the text implies.

Who is this for? People working on selective prediction, learning to rank, or human-in-the-loop pipelines. It gives them a simple baseline and a clean problem formulation. The paper deserves a serious referee, but it needs a revision that fixes the appendix, states the plug-in assumption explicitly, and adds significance testing.

Referee Report

3 major / 5 minor

Summary. The paper introduces BALToR, a method for bounded abstention in pairwise learning-to-rank with ties. The main theoretical result (Theorem 3.1) characterizes the optimal selection function for a given ranker as thresholding the ranker's conditional risk at the c-th quantile, with randomization on the risk-equal-to-threshold boundary. The authors propose a plug-in algorithm that estimates the risk as 1 - max_y \hat p(y | x, x'), calibrates the threshold on a held-out set, and applies it at test time. Experiments on Web-30k, OHSUMED, and MQ2007 compare BALToR against entropy-based and random abstention baselines for LambdaMART rankers paired with Bradley-Terry and Thurstone-Mosteller probability models.

Significance. If the theory is made rigorous and the algorithm is aligned with it, the paper would provide a useful extension of selective classification to pairwise ranking with ties, with a clean plug-in procedure and publicly released code. The bounded-abstention formulation for pairwise ranking with ties appears to be new, and the connection to the selective-classification results of Franc et al. is clearly drawn. The paper is appropriately modest in claiming an extension rather than a wholly new theory. The empirical evidence is mixed: gains are clear on MQ2007, small on Web-30k, and within one standard deviation on OHSUMED, so the experimental claims need stronger support.

major comments (3)

[§3, Algorithm 1, and Proposition 3.3] The selection score used in Algorithm 1, 1 - max_y \hat p(y | x, x'), is the plug-in estimate of the conditional risk of the Bayes-optimal ranker f*(x, x') = argmax_y p(y | x, x') (Proposition 3.3), not of an arbitrary fixed ranker f. For a fixed f, the 0-1 conditional risk is r_f(x, x') = 1 - p(f(x, x') | x, x'). Unless f equals the Bayes-optimal ranker with respect to the true distribution P, or \hat p is a calibrated estimate and f is its argmax, thresholding 1 - max_y \hat p(y | x, x') is not estimating r_f. The paper never states this f = f* assumption; Theorem 3.1 therefore does not justify BALToR for the 'given ranker f' setting claimed in the introduction. The experiments sidestep the issue because f is defined as the argmax of the same parametric probability model used for risk estimation, but no consistency or calibration result is provided for those models. The algorithm can be repaired by using 1 - \hat p(f(x, x') | x, x') as the risk estimate, or by explicitly restricting the scope to Bayes-optimal rankers.
[Appendix A, Proof of Theorem 3.1, Case 2 and Case 3] The proof of Theorem 3.1 is not rigorous as written. In Case 2 (the 'violation p2' paragraph), the assumed violation has ∫_{r=β} g p < c - ∫_{r<β} p, which means coverage is below c, so g is infeasible and cannot be compared as a candidate solution. The definition of g' is also inconsistent: equation (10) sets g' = 0 on {r > β}, while equation (11) requires the mass previously assigned to {r > β} to be reassigned to {r = β}; the proof does not show that such a g' ∈ [0,1] exists or that coverage is preserved in general. The final inequality chain contains a strict '>' followed by '= 0' with no justification. Case 3 similarly invokes Case 1 after scaling, but Case 1 was only proved under the assumption that p2 and p3 hold, which is not established for the scaled g'. Since Theorem 3.1 is the central theoretical contribution, this proof needs a complete rewrite with a proper exchange argument.
[§4.2, Figure 2c and 2f] The empirical support for the main claim is weak on OHSUMED. For the Bradley-Terry model, the reported accuracy at c = 0.70 is 0.578 ± 0.07 versus 0.562 ± 0.06 at full coverage; for the Thurstone-Mosteller model, the corresponding numbers are 0.594 ± 0.076 versus 0.573 ± 0.07. These differences are well within one standard deviation across the five folds, so the result is compatible with no improvement. Even on Web-30k, the absolute gain is only 0.005-0.007. The paper should report paired significance tests across the five folds (for example, a Wilcoxon signed-rank test) or confidence intervals, and discuss effect sizes; without this, the claim that the experiments 'demonstrate the effectiveness' of BALToR is overstated.

minor comments (5)

[Theorem 3.2] In the displayed definition of pr, the denominator is printed as ∫∫_{r(X×X)<β} p(x, x') dxdx'; it should be ∫∫_{r(X×X)=β} p(x, x') dxdx'. The proof of the theorem uses the correct denominator.
[Appendix A, Eq. (5) and surrounding text] Equation (5) is dimensionally inconsistent: it writes β + (1/c)∫_{r<β} p - (β/c)∫_{r<β} p, omitting the factor r(x, x') in the first integral. The correct expression is β + (1/c)∫_{r<β} p(x, x')(r(x, x') - β) dxdx'. There are also multiple notational typos in the proof: l(f(x), f(x'), y) should be l(f(x, x'), y); l(f(x'), f(x'), y) should be l(f(x, x'), y); and the sentence after condition p3 says 'r(X × X)<β' where it should say 'r(X × X)>β'.
[Eq. (1) and Eq. (2)] There is a typo in the definition of selective risk: the loss is written as l((f(X, X'), Y) g(X, X')), which is missing the intended composition; it should be l(f(X, X'), Y) g(X, X') (with the expectation over X, X', Y in the numerator).
[Theorem 3.1 statement] The theorem states 'Let us consider a symmetric loss' but the proof never uses symmetry. If the result holds for arbitrary losses, the assumption should be removed; if symmetry is needed only for the consistency of abstaining on both (x, x') and (x', x), that role should be stated and proved explicitly.
[Abstract] There are minor grammar issues in the abstract ('an algorithmic decision-making system to defer', 'while abstention have been predominantly explored'), which should be corrected.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the optimal-abstention theorem is proved from the selective-risk objective, and BALToR's threshold is a coverage quantile rather than a fitted accuracy parameter.

full rationale

The paper's central theoretical claim, Theorem 3.1, is a self-contained characterization of the bounded-abstention pairwise ranking problem for a fixed ranker and a symmetric loss. The proof in Appendix A uses a mass-shifting argument on the conditional risk r(x,x') and does not depend on the authors' own prior results; the citation to Franc et al. (2023) is an external source for the proof strategy, not a load-bearing self-citation. Algorithm 1 implements a standard plug-in approach: the conditional probabilities are estimated, the risk is estimated as 1 - max_y p_hat, and the threshold beta_hat is the c-th quantile of the estimated risks on a calibration set. This threshold is chosen to satisfy the coverage constraint, not to maximize the accuracy metric reported in the experiments, so the empirical accuracy gains are not forced by construction. The self-citations that appear in the paper (Pugnana and Ruggieri 2023, 2025; Pugnana et al. 2024) are contextual references in related work and limitations and are not load-bearing for the main derivation. One can identify a correctness gap: the plug-in score 1 - max_y p_hat is the conditional risk of the Bayes-optimal ranker, and it coincides with the risk of the deployed ranker only when that ranker is the argmax of the estimated probabilities. That is a modeling assumption about the ranker and the probability estimates, not a circularity in the derivation. The paper's own limitations section acknowledges dependence on the given ranker's quality. Overall, the derivation is not circular.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claim rests on a standard selective-classification optimality result, on distributional assumptions about pairwise data, and on an unverified plug-in step. No new physical or model entities are introduced; the abstainer is a standard selection function.

free parameters (2)

Bradley-Terry tie parameter theta = theta = 2 * npairs / n_no_ties - 1, per dataset
Used to convert ranker scores into pairwise probabilities for ties in the BT model; affects risk estimates and thus the abstention threshold. Chosen heuristically as suggested by Rao and Kupper, not optimized for accuracy.
Abstention threshold beta_c = c-th quantile of estimated conditional risks on the calibration set
Data-dependent threshold that sets coverage. It is not fitted to maximize test accuracy, but it is a quantity estimated from a held-out set on which the method relies.

assumptions (4)

domain assumption Pairwise instances (x,x') are i.i.d. from an unknown distribution P over X x X x Y
The risk, coverage, and Theorem 3.1 are formulated in terms of expectations over P; ranking datasets actually have per-query item dependencies, so this is an idealization (Sections 2.1 and 3).
domain assumption The loss is symmetric: l(f(x,x'), y) = l(f(x',x), -y)
Used in Theorem 3.1 and Theorem 3.2 to ensure consistent rejection of both orientations of a pair (Section 3, before Theorem 3.1).
domain assumption Ranker predicted probabilities p_hat accurately approximate true posteriors p(y|x,x')
Algorithm 1 uses 1 - max p_hat as conditional risk; no calibration guarantee is given, and the optimality theorem assumes true risk (Section 3, Algorithm 1).
domain assumption Calibration set and test set are drawn from the same distribution
The quantile beta_hat estimated on the calibration set is applied at test time to satisfy coverage; stated in the experimental setup (Section 4.1).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bounded-Abstention Pairwise Learning to Rank." pith.science (2026). https://pith.science/paper/SPLAYFDF

@misc{pith2026250523437,
  author       = {Pith},
  title        = {Pith review of: Bounded-Abstention Pairwise Learning to Rank},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SPLAYFDF}},
  note         = {Machine review of arXiv:2505.23437}
}

read the original abstract

Ranking systems influence decision-making in high-stakes domains like health, education, and employment, where they can have substantial economic and social impacts. This makes the integration of safety mechanisms essential. One such mechanism is abstention, which enables algorithmic decision-making systems to defer uncertain or low-confidence decisions to human experts. While abstention has been predominantly explored in the context of classification tasks, its application to other machine learning paradigms remains underexplored. In this paper, we introduce a novel method for abstention in pairwise learning-to-rank tasks. Our approach is based on thresholding the ranker's conditional risk: the system abstains from making a decision when the estimated risk exceeds a predefined threshold. Our contributions are threefold: a theoretical characterization of the optimal abstention strategy, a model-agnostic, plug-in algorithm for constructing abstaining ranking models, and a comprehensive empirical evaluation across multiple datasets, demonstrating the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2505.23437 by the authors.

**Figure 2.** Accuracy Acc on the selected pairs (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line). For smaller target coverages c (i.e., more abstention), the accuracy increases for BALToR, remains stable for the random abstainer, has erratic performance for the entropy-based abstainer. (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

**Figure 3.** Actual (empirical) coverage Cov on the test set (mean ± std over five folds) for the BT model . The actual coverage remains very close to the target coverage c. OHSUMED [Hersh et al., 1994] is a MEDLINE database subset containing 106 queries with 16,140 query-documents pairs. Each query-document pair has a 25-dimensional feature vector that contains popular information-retrieval features, such as tf-idf and BM25 sco… view at source ↗

Figures from the paper (13 more)

**Figure 4.** Figure 4: Distribution of classes SelRate on the selected pairs (mean ± std over five folds) for the BT model . While the target coverage c varies, BALToR maintains stable the proportions of the classes in Y = {−1, 0, 1}. predicted probabilities rather than the conditional risk …

**Figure 5.** Figure 5: Estimated density functions over MQ2007 Fold 1 calibration set for BT (Figure 5a) and TM (Figure 5b) when using an XGBRanker. The shapes of the density functions differ (are similar) when using BT (TM) model. (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p016_5.png]

**Figure 6.** Figure 6: Actual (empirical) coverage Cov on the test set (mean ± std over five folds) for the TM model . The actual coverage remains very close to the target coverage c. (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

**Figure 7.** Figure 7: Distribution of classes SelRate on the selected pairs (mean ± std over five folds) for the TM model . While the target coverage c varies, BALToR maintains stable the proportions of the classes in Y = {−1, 0, 1}. (a) (b) (c) [PITH_FULL_IMAGE:figures/full_fig_p016_7.png]

**Figure 8.** Figure 8: Distribution of classes SelRate on the selected pairs (mean ± std over five folds) for the TM model . While the target coverage c varies, entropy maintains stable the proportions of the classes in Y = {−1, 0, 1}. 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]

**Figure 9.** Figure 9: Acc results (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line) when using LightGBM. (a) (b) (c) (d) (e) (f) [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]

**Figure 10.** Figure 10: Cov results (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line) when using LightGBM. 17 [PITH_FULL_IMAGE:figures/full_fig_p017_10.png]

**Figure 11.** Figure 11: SelRate results (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line) when using LightGBM and BALToR. (a) (b) (c) (d) (e) (f) [PITH_FULL_IMAGE:figures/full_fig_p018_11.png]

**Figure 12.** Figure 12: SelRate results (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line) when using LightGBM and entropy. 18 [PITH_FULL_IMAGE:figures/full_fig_p018_12.png]

**Figure 13.** Figure 13: Acc results (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line) when using CatBoost. (a) (b) (c) (d) (e) (f) [PITH_FULL_IMAGE:figures/full_fig_p019_13.png]

**Figure 14.** Figure 14: Cov results (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line) when using CatBoost. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_14.png]

**Figure 15.** Figure 15: SelRate results (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line) when using CatBoost and BALToR. (a) (b) (c) (d) (e) (f) [PITH_FULL_IMAGE:figures/full_fig_p020_15.png]

**Figure 16.** Figure 16: SelRate results (mean ± std over five folds) for the BT model (top line) and the TM model (bottom line) when using CatBoost and entropy. 20 [PITH_FULL_IMAGE:figures/full_fig_p020_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

34 extracted references · 30 canonical work pages

[1]

Rank analysis of incomplete block designs: I

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. T he method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

1952
[2]

From ranknet to lambdarank to lambdamart: An overview

Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 11 0 (23-581): 0 81, 2010

2010
[3]

Xgboost: A scalable tree boosting system

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In KDD , pages 785--794. ACM , 2016

work page 2016
[4]

e l Rademaker, Bernard De Baets, and Eyke H \

Weiwei Cheng, Micha \"e l Rademaker, Bernard De Baets, and Eyke H \"u llermeier. Predicting partial orders: ranking with abstention. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part I 21, pages 215--230. Springer, 2010

work page 2010
[5]

Label ranking with partial abstention based on thresholded probabilistic models

Weiwei Cheng, Eyke H \" u llermeier, Willem Waegeman, and Volkmar Welker. Label ranking with partial abstention based on thresholded probabilistic models. In NIPS , pages 2510--2518, 2012

work page 2012
[6]

C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory , 16 0 (1): 0 41--46, 1970

work page 1970
[7]

Bioucas - Dias, Carlos A

Filipe Condessa, Jos \' e M. Bioucas - Dias, Carlos A. Castro, John A. Ozolek, and Jelena Kovacevic. Classification with reject option using contextual information. In ISBI , pages 1340--1343. IEEE , 2013

work page 2013
[8]

Cordella, Claudio De Stefano, Carlo Sansone, and Mario Vento

Luigi P. Cordella, Claudio De Stefano, Carlo Sansone, and Mario Vento. An adaptive reject option for LVQ classifiers. In ICIAP , volume 974 of Lecture Notes in Computer Science, pages 68--73. Springer, 1995

work page 1995

Show all 34 references

[9]

Boosting with abstention

Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Boosting with abstention. In NIPS , pages 1660--1668, 2016

2016
[10]

Theory and algorithms for learning with rejection in binary classification

Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Theory and algorithms for learning with rejection in binary classification. Ann. Math. Artif. Intell., 92 0 (2): 0 277--315, 2024

2024
[11]

Consistency of plug-in confidence sets for classification in semi-supervised learning

Christophe Denis and Mohamed Hebiri. Consistency of plug-in confidence sets for classification in semi-supervised learning. J. of Nonpar. Statistics, 32 0 (1): 0 42--72, 2020

2020
[12]

On the foundations of noise-free selective classification

Ran El - Yaniv and Yair Wiener. On the foundations of noise-free selective classification. J. Mach. Learn. Res., 11: 0 1605--1641, 2010

2010
[13]

Leo Feng, Mohamed Osama Ahmed, Hossein Hajimirsadeghi, and Amir H. Abdi. Towards better selective classification. In ICLR . OpenReview.net, 2023

2023
[14]

Optimal strategies for reject option classifiers

Vojtech Franc, Daniel Pr u s a, and V \' a clav Vor \' a cek. Optimal strategies for reject option classifiers. J. Mach. Learn. Res., 24: 0 11:1--11:49, 2023

2023
[15]

Selective classification for deep neural networks

Yonatan Geifman and Ran El - Yaniv. Selective classification for deep neural networks. In NIPS , pages 4878--4887, 2017

2017
[16]

Selectivenet: A deep neural network with an integrated reject option

Yonatan Geifman and Ran El - Yaniv. Selectivenet: A deep neural network with an integrated reject option. In ICML , volume 97, pages 2151--2159. PMLR , 2019

2019
[17]

Machine learning with a reject option: a survey

Kilian Hendrickx, Lorenzo Perini, Dries Van der Plas, Wannes Meert, and Jesse Davis. Machine learning with a reject option: a survey. Mach. Learn., 113 0 (5): 0 3073--3110, 2024

2024
[18]

Radu Herbei and Maten H. Wegkamp. Classification with reject option. Can. J. Stat., 34 0 (4): 0 709–--721, 2006

2006
[19]

Ohsumed: An interactive retrieval evaluation and new large test collection for research

William Hersh, Chris Buckley, TJ Leone, and David Hickam. Ohsumed: An interactive retrieval evaluation and new large test collection for research. In SIGIR’94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information R...

1994
[20]

Lightgbm: A highly efficient gradient boosting decision tree

Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. Lightgbm: A highly efficient gradient boosting decision tree. Advances in neural information processing systems, 30, 2017

2017
[21]

David Madras, Toniann Pitassi, and Richard S. Zemel. Predict responsibly: Improving fairness and accuracy by learning to defer. In NeurIPS, pages 6150--6160, 2018

2018
[22]

Ranking with abstention

Anqi Mao, Mehryar Mohri, and Yutao Zhong. Ranking with abstention. CoRR, abs/2307.02035, 2023

2023 arXiv
[23]

Catboost: unbiased boosting with categorical features

Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018

2018
[24]

Constrained binary decision making

Daniel Prusa and Vojtech Franc. Constrained binary decision making. In NeurIPS , 2025

2025
[25]

Auc-based selective classification

Andrea Pugnana and Salvatore Ruggieri. Auc-based selective classification. In AISTATS , volume 206 of Proceedings of Machine Learning Research, pages 2494--2514. PMLR , 2023

2023
[26]

Deep neural network benchmarks for selective classification

Andrea Pugnana, Lorenzo Perini, Jesse Davis, and Salvatore Ruggieri. Deep neural network benchmarks for selective classification. Journal of Data-centric Machine Learning Research (DMLR), 1 0 (17): 0 1--58, 2024

2024
[27]

Introducing LETOR 4.0 datasets

Tao Qin and Tie - Yan Liu. Introducing LETOR 4.0 datasets. CoRR, abs/1306.2597, 2013. URL http://arxiv.org/abs/1306.2597

2013 arXiv
[28]

Are neural rankers still outperformed by gradient boosted decision trees? In ICLR

Zhen Qin, Le Yan, Honglei Zhuang, Yi Tay, Rama Kumar Pasumarthi, Xuanhui Wang, Michael Bendersky, and Marc Najork. Are neural rankers still outperformed by gradient boosted decision trees? In ICLR . OpenReview.net, 2021

2021
[29]

Ties in paired-comparison experiments: A generalization of the bradley-terry model

PV Rao and Lawrence L Kupper. Ties in paired-comparison experiments: A generalization of the bradley-terry model. Journal of the American Statistical Association, 62 0 (317): 0 194--204, 1967

1967
[30]

Things machine learning models know that they don't know

Salvatore Ruggieri and Andrea Pugnana. Things machine learning models know that they don't know. In AAAI , pages 28684--28693. AAAI Press, 2025

2025
[31]

A law of comparative judgment

Louis L Thurstone. A law of comparative judgment. Psychological review, 101 0 (2): 0 266, 1994

1994
[32]

A ROC -based reject rule for dichotomizers

Francesco Tortorella. A ROC -based reject rule for dichotomizers. Pattern Recognit. Lett., 26 0 (2): 0 167--180, 2005

2005
[33]

Regression with reject option and application to knn

Ahmed Zaoui, Christophe Denis, and Mohamed Hebiri. Regression with reject option and application to knn. In NeurIPS, 2020

2020
[34]

Learning to rank with ties

Ke Zhou, Gui-Rong Xue, Hongyuan Zha, and Yong Yu. Learning to rank with ties. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pages 275--282, 2008

2008

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

Rank analysis of incomplete block designs: I

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. T he method of paired comparisons. Biometrika, 39 0 (3/4): 0 324--345, 1952

1952

[2] [2]

From ranknet to lambdarank to lambdamart: An overview

Christopher JC Burges. From ranknet to lambdarank to lambdamart: An overview. Learning, 11 0 (23-581): 0 81, 2010

2010

[3] [3]

Xgboost: A scalable tree boosting system

Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In KDD , pages 785--794. ACM , 2016

work page 2016

[4] [4]

e l Rademaker, Bernard De Baets, and Eyke H \

Weiwei Cheng, Micha \"e l Rademaker, Bernard De Baets, and Eyke H \"u llermeier. Predicting partial orders: ranking with abstention. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2010, Barcelona, Spain, September 20-24, 2010, Proceedings, Part I 21, pages 215--230. Springer, 2010

work page 2010

[5] [5]

Label ranking with partial abstention based on thresholded probabilistic models

Weiwei Cheng, Eyke H \" u llermeier, Willem Waegeman, and Volkmar Welker. Label ranking with partial abstention based on thresholded probabilistic models. In NIPS , pages 2510--2518, 2012

work page 2012

[6] [6]

C. K. Chow. On optimum recognition error and reject tradeoff. IEEE Trans. Inf. Theory , 16 0 (1): 0 41--46, 1970

work page 1970

[7] [7]

Bioucas - Dias, Carlos A

Filipe Condessa, Jos \' e M. Bioucas - Dias, Carlos A. Castro, John A. Ozolek, and Jelena Kovacevic. Classification with reject option using contextual information. In ISBI , pages 1340--1343. IEEE , 2013

work page 2013

[8] [8]

Cordella, Claudio De Stefano, Carlo Sansone, and Mario Vento

Luigi P. Cordella, Claudio De Stefano, Carlo Sansone, and Mario Vento. An adaptive reject option for LVQ classifiers. In ICIAP , volume 974 of Lecture Notes in Computer Science, pages 68--73. Springer, 1995

work page 1995