Pith. sign in

REVIEW 3 major objections 4 minor 48 references

SubROC: AUC-Based Discovery of Exceptional Subgroup Performance for Binary Classifiers

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new framework, SubROC, identifies interpretable subgroups where a binary classifier performs exceptionally well or poorly by scoring subgroups with ROC and PR AUC, with tight bounds that make exhaustive search practical.

desk verdict Solid, reproducible methods contribution for AUC-based subgroup discovery; ROC/ARL theory is sound, but the PR-AUC lower-bound proof has a gap that should be fixed before the exhaustive-search guarantee is taken on faith. read the letter →

arxiv 2505.11283 v2 pith:ULYAOYJW submitted 2025-05-16 cs.LG

Tom Siegl , Kutalm{i}ş Coşkun , Bjarne C. Hiller , Amin Mirzaei , Florian Lemmerich , Martin Becker This is my paper · ORCID
classification cs.LG
keywords subgroupdiscoveryexceptionalmodelminingROCAUCPRoptimisticestimatesclassimbalancesignificancetestingbinaryclassifiers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SubROC seeks to make it routine to find interpretable population subgroups on which a binary classifier performs exceptionally well or badly. It defines exceptionality as the difference between the dataset-wide ROC AUC or PR AUC and the subgroup's own AUC, and wraps this score with optional cover-size and class-balance weights, redundancy reduction, and a significance filter. The paper's central contribution is a set of tight optimistic estimates—provably reachable upper bounds on the score of any specialized subgroup—that allow exhaustive search to prune the exponential pattern space without losing the best results. If the claims hold, practitioners can obtain guaranteed top-k subgroups (e.g., sex=female and housing=rural) with large speedups, and the resulting subgroups are more likely to reflect real model weaknesses rather than small-sample artifacts.

What carries the argument

The carrying object is the optimistic estimate: for a scoring function $\varphi$, an upper bound $\mathrm{oe}_{\varphi}(p)$ such that every specialization of $p$ scores at most $\mathrm{oe}_{\varphi}(p)$, and "tight" means some specialization attains the bound exactly. SubROC constructs these from lower bounds on the AUC performance measures: $b_{\mathrm{ROCAUC}}$ returns $1$ for perfectly separated labels, $1/2$ when only ties permit separation, and $0$ when an inverted pair exists; $b_{\mathrm{PRAUC}}$ returns the PR AUC of the set containing the lowest-scoring positive instance and all negatives. A separate weighting bound $b_w(I) = (2\cdot\min\{|P_I|, |N_I|\})^\alpha$ bounds the product of cover size and class-balance terms when $\alpha \le \beta$, extending pruning to weighted scores. These bounds let the best-first exhaustive search discard entire specialization branches and guarantee that no top-k subgroup is missed; the same lower bounds are also used to define the optimistic estimates for the relative scores $\varphi^{\mathrm{rROCAUC}}$ and $\varphi^{\mathrm{rPRAUC}}$.

What would settle it

Enumerate all subsets of a small labeled score multiset (or sample many random multisets), compute the linearly interpolated PR AUC of each subset, and check whether any value lies strictly below $\mathrm{PRAUC}(\{\arg\min_{c\in P_I} \hat{Y}(c)\}\cup N_I)$. One such subset falsifies Lemma 4, and with it the claim that the PR AUC optimistic estimate still yields an exhaustive search.

Watch

Extended reading notes

Core claim

The central discovery is that AUC-based subgroup scoring can be paired with tight optimistic estimates so that an exhaustive search stays complete while becoming fast. For a pattern $p$, the relative scores $\varphi^{\mathrm{rROCAUC}}(p)$ and $\varphi^{\mathrm{rPRAUC}}(p)$ measure how much a subgroup's AUC falls short of the whole dataset's AUC; large shortfalls mark underperformance, and the negative of the same quantity marks overperformance. The paper derives a tight optimistic estimate for the existing ARL score (Theorem 1), for ROC AUC via a three-case lower bound $b_{\mathrm{ROCAUC}}$ (Theorem 3 and Lemma 2), and for PR AUC via a worst-case subset consisting of the lowest-scoring positive instance plus all negatives (Theorem 5 and Lemma 4), plus a bound for the cover-size/class-balance weight product (Lemma 6). The PR AUC proof rests on Lemma 4, whose appendix proof contains an asserted step—points in PR space below the constructed worst-case curve "have no impact on the AUC"—that the paper itself flags in Section 3.2.3 as not verified. With these bounds, the full framework reports larger, more balanced, and mostly significant subgroups on standard datasets and recovers an injected weakness at the top ranks for ROC AUC.

Load-bearing premise

The load-bearing premise is Lemma 4's claim that no subset of a subgroup's instances can have a PR AUC below the PR AUC of the worst-case set built from the lowest-scoring positive instance and all negatives, because the appendix proof justifies this with an asserted statement that such below-curve points "have no impact on the AUC."

Editorial extensions

If this is right

  • For a trained classifier, the framework returns ranked, interpretable subgroups—such as sex=female and housing=rural—where the model is unusually bad or good, directly targeting where retraining or additional data collection is needed.
  • Because the optimistic estimates are tight and the search is exhaustive, the top-k result set is guaranteed to contain the highest-scoring subgroups under the chosen score and constraints, not a heuristic approximation.
  • Weighted scoring with $\alpha=\beta=1$ shifts results toward larger, more balanced subgroups, many of which pass holdout significance tests; on the Census and Adult datasets, 5 of 5 filtered ROC subgroups were significant versus 0 or 1 of 5 in the unweighted baseline.
  • Reported speedups of 300–700× for ARL, PR AUC, and ROC AUC on several datasets imply that exhaustive AUC-based subgroup search is practical at dataset scales previously considered out of reach.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the ROC AUC lower bound is coarse (only three values $0$, $1/2$, $1$), the ROC search may visit more nodes than necessary; a natural next step, not tested in the paper, is a tighter bound that uses the actual minimum AUC over admissible subsets, which could further cut runtimes without changing the exhaustive guarantee.
  • The injected-subgroup experiment showed ROC AUC recovering the injected weakness while ARL and PR AUC did not; this suggests the class-imbalance and size skews are not fully removed by weighting, so adopting normalized PR AUC (as the paper itself suggests) might make PR-based discovery as reliable as ROC-based discovery for small, skewed subgroups.
  • The significance test samples random subsets of equal size and class balance; a stronger test would condition on the same attribute distribution as the subgroup, which would reduce false positives for patterns that merely correlate with label imbalance.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SubROC, an Exceptional Model Mining framework for discovering interpretable subgroups where a binary soft classifier performs exceptionally well or poorly, measured by relative ROC AUC, PR AUC, and average ranking loss (ARL). The main technical contributions are new scoring functions defined as differences from the overall dataset AUC, tight optimistic estimates that enable pruning in exhaustive subgroup search, a class-balance and cover-size weighting scheme, generalization-aware redundancy reduction, and a significance filtering procedure. The framework is implemented in the open-source pysubgroup package, and experiments on eight datasets demonstrate large runtime speedups from optimistic-estimate pruning and improved quality of discovered subgroups compared to a baseline setting.

Significance. If the technical results are correct, this is a substantial contribution to subgroup discovery and model evaluation: it provides the first tight optimistic estimates for ROC AUC and PR AUC in the SCaPE model class, enabling exhaustive search with pruning, and it packages these with practical controls for class imbalance, redundancy, and statistical significance. The paper ships open-source code and reproducible experiments, which strengthens its practical value. The empirical evaluation is broad, covering multiple datasets and performance measures, and includes an injected-subgroup recovery study. The main caveat is that the PR AUC optimistic estimate rests on a proof with a gap in a key lemma; the overall framework is plausible and likely correct, but the PR AUC exhaustive-search guarantee is not yet rigorously established.

major comments (3)
  1. [Appendix A.3, Lemma 4, Case 28] The proof concludes that points of a sub-multiset's PR curve "either [do] not lie below the PR curve of I_worst or have no influence on the AUC," but no argument is supplied for the second alternative. Since the trapezoidal AUC used in Definition 6 integrates over all horizontal intervals, any point strictly below the constructed curve could reduce the area. This assertion is load-bearing for Theorem 5 and the exhaustive-search guarantee for PR AUC; the authors must either prove that such points occur only on zero-width vertical segments or provide a different argument.
  2. [Appendix A.3, Lemma 4, Case 29] The proof invokes the minimum PR curve result of [16] and then analyzes only the two-point set P'_worst. It does not establish that for an arbitrary sub-multiset C' with positive class ratio not lower than that of C_worst, every point of the linearly interpolated PR curve of C' (as defined in Definition 6) lies at or above the corresponding linearly interpolated point of C_worst's curve. The transfer from the "correct interpolation" of [16] to the trapezoidal linear-interpolation definition is nontrivial and must be proved explicitly.
  3. [Section 3.2.3 and Theorem 5] Because Lemma 4 is not fully proved, the claim that oe_{φ^r_PRAUC}(p) = PRAUC(I) - b_PRAUC(sg(p)) is a tight optimistic estimate is not established. This directly affects the central claim that SubROC allows exhaustive search for PR AUC with pruning. The authors should complete the proof of Lemma 4, or alternatively state the lemma as a conjecture with strong empirical verification, and clearly mark the exhaustive-search guarantee as conditional on that conjecture.
minor comments (4)
  1. [Section 4.1, Table 1] The ARL rows for Bank and Credit report "-" and 0/0, but the text does not explain why no ARL subgroups were found; please clarify whether this is due to the minimum support constraint, undefined ARL, or another reason.
  2. [Section 4.2, Table 3] The speedup for ROC AUC on Adult at α=β=0 is only 1.1x, indicating negligible pruning benefit; this deserves a brief comment, as it contrasts with the large speedups on Bank and Mushroom.
  3. [Appendix A, Definition 5 and Definition 6] The ROC and PR AUC definitions divide by P(C)·N(C) and P(C), respectively; the paper should state explicitly that these values are undefined when the denominator is zero, and that the scoring functions are only applied when the required positive/negative instances exist.
  4. [Section 3.1.1] The sentence "For PR AUC, we employ the widely used approximation via a linearly interpolated PR curve" cites [14], but the reference appears later; please ensure the citation is included at the point of definition.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: SubROC's scoring functions and optimistic estimates are derived from standard AUC definitions; the flagged PR-AUC proof gap is a correctness risk, not a circular reduction.

full rationale

SubROC's load-bearing derivations are self-contained. The scoring functions in Section 3.1.1 are defined as differences between the whole-dataset AUC and the subgroup AUC; this is a design choice that does not presuppose the discovered subgroups. The optimistic estimates in Section 3.2 are proved via the generic transitions in Lemmas 7-9 from performance-measure lower/upper bounds to interestingness-measure optimistic estimates, and the bounds (Lemma 2, Lemma 4, Lemma 6) are proved from the formal AUC definitions (Definitions 5-7) and the subset relation between a pattern and its specializations. No parameter is fitted to the data and then reported as a prediction; alpha and beta are user-set weights, and the injected-subgroup experiment uses an externally constructed target. The only flagged issue is a proof gap in Lemma 4, Appendix A.3: in case 28 the proof asserts that some PR-curve points 'have no influence on the AUC' without giving an argument, and case 29 imports the minimum-PR-curve statement of [16] without fully transferring it to the linearly interpolated PR AUC of Definition 6. That is a correctness risk in an otherwise plausible bound, not a circular reduction: the lower bound is not defined in terms of the quantity it is used to bound. Self-citations such as [12] and [20] supply standard definitions and prior generalization-aware scoring; they are not load-bearing uniqueness claims that force the paper's conclusions. Consequently, no step qualifies under the hard rules.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

The central method introduces no new physical entities. The mathematical derivation relies on standard ROC/PR AUC definitions and the EMM framework. The main user-chosen constants are the cover size weight α and class balance weight β; Lemma 6 restricts the optimistic estimate to α≤β. The PR AUC lower bound proof in Appendix A.3 assumes without formal proof that certain edge-case interpolation points do not affect the computed AUC. The significance testing assumes that 1000 random permutations with fixed size and class balance offer a valid conditional null distribution.

free parameters (2)
  • Cover size weight α = 0, 0.1, 0.3, 1 in experiments
    User-chosen exponent that trades off subgroup cover size against exceptionality in φα(p)=|sg(p)|^α·φ'(p).
  • Class balance weight β = 0, 0.1, 0.3, 1 in experiments
    User-chosen exponent that trades off class balance against exceptionality in φβ(p)=cb(sg(p))^β·φ'(p); Lemma 6 requires α≤β for the optimistic estimate.
assumptions (5)
  • standard math Trapezoidal definition of AUC and linear interpolation for PR curves
    Used in Definitions 5 to 7 of Appendix A; this is the standard numerical integration convention for ROC and PR curves.
  • standard math Standard multiset notation and subset properties
    Appendix A formalizes instances and subgroups as multisets and uses subset monotonicity to prove the optimistic estimates.
  • domain assumption Permutation test with 1000 random subsets and fixed size/class balance approximates the null distribution
    Section 3.3 uses this to compute empirical p-values; the approximation is standard but not exact.
  • domain assumption Linearly interpolated PR AUC approximation has no practical consequences
    Section 3.1.1 states this without a dedicated experiment, relying on prior work.
  • ad hoc to paper PR-space points below the constructed lower curve in the PR AUC proof have no impact on the AUC
    Section 3.2.3 and Appendix A.3 assert this without a formal proof; the soundness of the PR AUC optimistic estimate depends on it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SubROC: AUC-Based Discovery of Exceptional Subgroup Performance for Binary Classifiers." pith.science (2026). https://pith.science/paper/ULYAOYJW

@misc{pith2026250511283,
  author       = {Pith},
  title        = {Pith review of: SubROC: AUC-Based Discovery of Exceptional Subgroup Performance for Binary Classifiers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ULYAOYJW}},
  note         = {Machine review of arXiv:2505.11283}
}
read the original abstract

Machine learning (ML) is increasingly employed in real-world applications like medicine or economics, thus, potentially affecting large populations. However, ML models often do not perform homogeneously, leading to underperformance or, conversely, unusually high performance in certain subgroups (e.g., sex=female AND marital_status=married). Identifying such subgroups can support practical decisions on which subpopulation a model is safe to deploy or where more training data is required. However, an efficient and coherent framework for effective search is missing. Consequently, we introduce SubROC, an open-source, easy-to-use framework based on Exceptional Model Mining for reliably and efficiently finding strengths and weaknesses of classification models in the form of interpretable population subgroups. SubROC incorporates common evaluation measures (ROC and PR AUC), efficient search space pruning for fast exhaustive subgroup search, control for class imbalance, adjustment for redundant patterns, and significance testing. We illustrate the practical benefits of SubROC in case studies as well as in comparative analyses across multiple datasets.

Figures

Figures reproduced from arXiv: 2505.11283 by the authors.

Figure 1
Figure 1. Illustrative example. Predictions of a binary soft classifier produce a ranking of patients, from healthy to sick. The clearer this separation, the better the classifier. Here, the density distributions of healthy and diseased patients (red and blue curves, middle left) are clearly distinguishable but there is no perfect separation. SubROC automatically discovers interpretable subgroups where the model predictions c… view at source ↗
Figure 2
Figure 2. Intersection over Union (IoU) of subgroup covers, comparing an injected subgroup with [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗
Figure 3
Figure 3. Skewed response of cover size and class balance weighted scoring functions depend [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Skewed response of cover size and class balance weighted scoring functions depending [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]
Figure 5
Figure 5. Figure 5: Lower border of the achievable area in PR space and approximations for a positive class ratio 1 2 . The curve named “border” is the actual border of the achievable area. The curves named “approximation 1” and “approximation 2” are the (lin￾early interpolated) PR curves…
Figure 6
Figure 6. Figure 6: IoU of an injected subgroup with subgroups in a top-10 ARL result set, ordered [PITH_FULL_IMAGE:figures/full_fig_p042_6.png]
Figure 7
Figure 7. Figure 7: IoU of an injected subgroup with subgroups in a top-10 PR AUC result set, ordered [PITH_FULL_IMAGE:figures/full_fig_p042_7.png]
Figure 8
Figure 8. Figure 8: Histogram of standard deviations divided by medians for all runtimes of Tables 14 [PITH_FULL_IMAGE:figures/full_fig_p045_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 29 canonical work pages

  1. [16]

    Unachievable region in precision-recall space and its effect on empir- ical evaluation

    Kendrick Boyd et al. “Unachievable region in precision-recall space and its effect on empir- ical evaluation”. In:Proceedings of the 29th International Coference on International Con- ference on Machine Learning. ICML’12. Edinburgh, Scotland: Omnipress, 2012, pp. 1619–

    work page 2012

  2. [1]

    A Survey on Bias and Fairness in Machine Learning

    Ninareh Mehrabi et al. “A Survey on Bias and Fairness in Machine Learning”. In:ACM Comput. Surv.54.6 (2021), 115:1–115:35.issn: 0360-0300. doi: 10.1145/3457607

    doi:10.1145/3457607 2021

  3. [2]

    Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization

    Shiori Sagawa et al. “Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization”. In:2020 International Con- ference on Learning Representations. 2020. url: https://openreview.net/forum?id= ryxGuJrFvS

    work page 2020

  4. [3]

    No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems

    Nimit Sohoni et al. “No Subclass Left Behind: Fine-Grained Robustness in Coarse-Grained Classification Problems”. In:Advances in Neural Information Processing Systems. Vol. 33. Curran Associates, Inc., 2020, pp. 19339–19352.url: https://proceedings.neurips. cc/paper/2020/hash/e0688d13958a19e087e123148555e4b4-Abstract.html

    work page 2020

  5. [4]

    TFX: A TensorFlow-Based Production-Scale Machine Learning Plat- form

    Denis Baylor et al. “TFX: A TensorFlow-Based Production-Scale Machine Learning Plat- form”. In:Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’17. Halifax, NS, Canada: Association for Comput- ing Machinery, 2017, pp. 1387–1395. isbn: 978-1-4503-4887-4. doi: 10 . 1145 / 3097983 . 3098021

    work page 2017

  6. [5]

    Visual exploration of machine learning results using data cube analysis

    Minsuk Kahng, Dezhi Fang, and Duen Horng (Polo) Chau. “Visual exploration of machine learning results using data cube analysis”. In:Proceedings of the Workshop on Human-In- the-Loop Data Analytics. HILDA ’16. New York, NY, USA: Association for Computing Machinery, 2016, pp. 1–6.isbn: 978-1-4503-4207-0. doi: 10.1145/2939502.2939503

    arXiv 2016

  7. [6]

    Automated Data Slicing for Model Validation: A Big Data - AI Integration Approach

    Yeounoh Chung et al. “Automated Data Slicing for Model Validation: A Big Data - AI Integration Approach”. In:IEEE Transactions on Knowledge and Data Engineering32.12 (2020), pp. 2284–2296.issn: 1558-2191. doi: 10.1109/TKDE.2019.2916074

    arXiv 2020

  8. [7]

    FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning

    Ángel Alexander Cabrera et al. “FAIRVIS: Visual Analytics for Discovering Intersectional Bias in Machine Learning”. In:2019 IEEE Conference on Visual Analytics Science and Technology (VAST). 2019, pp. 46–56.doi: 10.1109/VAST47406.2019.8986948

    arXiv 2019

Show all 48 references
  1. [8]

    Understanding Where Your Classifier Does (Not) Work – The SCaPE Model Class for EMM

    Wouter Duivesteijn and Julia Thaele. “Understanding Where Your Classifier Does (Not) Work – The SCaPE Model Class for EMM”. In:2014 IEEE International Conference on Data Mining. 2014, pp. 809–814.doi: 10.1109/ICDM.2014.10. 13

    2014 doi

  2. [9]

    SubROC: AUC-Based Discovery of Exceptional Subgroup Perfor- mance for Binary Classifiers

    Tom Siegl et al.Code for “SubROC: AUC-Based Discovery of Exceptional Subgroup Perfor- mance for Binary Classifiers”. Zenodo. Available athttps://doi.org/10.5281/zenodo. 16952343. 2025

    2025 doi

  3. [10]

    Explora: a multipattern and multistrategy discovery assistant

    Willi Klösgen. “Explora: a multipattern and multistrategy discovery assistant”. In: Ad- vances in knowledge discovery and data mining. USA: American Association for Artificial Intelligence, 1996, pp. 249–271.isbn: 978-0-262-56097-9

    1996

  4. [11]

    Exceptional Model Mining

    Dennis Leman, Ad Feelders, and Arno Knobbe. “Exceptional Model Mining”. en. In:Ma- chine Learning and Knowledge Discovery in Databases. Ed. by Walter Daelemans, Bart Goethals, and Katharina Morik. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelber...

    2008 doi

  5. [12]

    Novel Techniques for Efficient and Effective Subgroup Discovery

    Florian Lemmerich. “Novel Techniques for Efficient and Effective Subgroup Discovery”. eng. PhD thesis. Universität Würzburg, 2014.url: https://nbn- resolving.org/urn: nbn:de:bvb:20-opus-97812

    2014

  6. [13]

    Tight Optimistic Estimates for FastSubgroupDiscovery

    Henrik Grosskreutz, Stefan Rüping, and Stefan Wrobel. “Tight Optimistic Estimates for FastSubgroupDiscovery”.en.In: Machine Learning and Knowledge Discovery in Databases. Ed.byWalterDaelemans,BartGoethals,andKatharinaMorik.Berlin,Heidelberg:Springer Berlin Heidelberg, 2008, pp...

    2008 doi

  7. [14]

    The relationship between Precision-Recall and ROC curves

    Jesse Davis and Mark Goadrich. “The relationship between Precision-Recall and ROC curves”. In: Proceedings of the 23rd international conference on Machine learning. ICML ’06. Pittsburgh, Pennsylvania, USA: Association for Computing Machinery, 2006, pp. 233–

    2006

  8. [15]

    Whentoconsultprecision-recallcurves

    JonathanCookandVikramRamadas.“Whentoconsultprecision-recallcurves”.en.In: The Stata Journal20.1(2020),pp.131–148. issn:1536-867X. doi: 10.1177/1536867X20909693

    2020 doi

  9. [17]

    Binary classification performance measures/metrics: A comprehensive visualized roadmap to gain new insights

    Gürol Canbek et al. “Binary classification performance measures/metrics: A comprehensive visualized roadmap to gain new insights”. In:2017 International Conference on Computer Science and Engineering (UBMK). Antalya, 2017, pp. 821–826.isbn: 978-1-5386-0930-9. doi: 10.1109/UBMK...

    2017

  10. [18]

    Multiobjective Support Vector Machines: Handling Class Imbalance With Pareto Optimality

    Shounak Datta and Swagatam Das. “Multiobjective Support Vector Machines: Handling Class Imbalance With Pareto Optimality”. In:IEEE Transactions on Neural Networks and Learning Systems30.5 (2019), pp. 1602–1608.issn: 2162-2388.doi: 10.1109/TNNLS.2018. 2869298

    2019 doi

  11. [19]

    The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets

    Takaya Saito and Marc Rehmsmeier. “The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets”. In:PLoS ONE 10.3 (2015), e0118432.doi: 10.1371/journal.pone.0118432

    2015 doi

  12. [20]

    Difference-Based Estimates for Generalization-Aware Subgroup Discovery

    Florian Lemmerich, Martin Becker, and Frank Puppe. “Difference-Based Estimates for Generalization-Aware Subgroup Discovery”. en. In:Machine Learning and Knowledge Dis- covery in Databases. Ed. by Hendrik Blockeel et al. Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp....

    2013 doi

  13. [21]

    A Concise Representation of Association Rules Us- ing Minimal Predictive Rules

    Iyad Batal and Milos Hauskrecht. “A Concise Representation of Association Rules Us- ing Minimal Predictive Rules”. en. In: Machine Learning and Knowledge Discovery in Databases. Ed. by José Luis Balcázar et al. Berlin, Heidelberg: Springer Berlin Heidel- berg, 2010, pp. 87–102...

    2010 doi

  14. [22]

    Local Models for Expectation-Driven Subgroup Discovery

    Florian Lemmerich and Frank Puppe. “Local Models for Expectation-Driven Subgroup Discovery”. In: 2011 IEEE 11th International Conference on Data Mining. ISSN: 2374-

    2011

  15. [23]

    Subgroup Discovery for ElectionAnalysis:ACaseStudyinDescriptiveDataMining

    Henrik Grosskreutz, Mario Boley, and Maike Krause-Traudes. “Subgroup Discovery for ElectionAnalysis:ACaseStudyinDescriptiveDataMining”.en.In: Discovery Science.Ed. by Bernhard Pfahringer, Geoff Holmes, and Achim Hoffmann. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010, p...

    2010 doi

  16. [24]

    Discovering Significant Patterns

    Geoffrey I. Webb. “Discovering Significant Patterns”. en. In:Machine Learning68.1 (2007), pp. 1–33. issn: 1573-0565. doi: 10.1007/s10994-007-5006-x

    2007 doi

  17. [25]

    Assessing data mining results via swap randomization

    Aristides Gionis et al. “Assessing data mining results via swap randomization”. In:ACM Trans. Knowl. Discov. Data1.3 (2007), 14–es. issn: 1556-4681. doi: 10.1145/1297332. 1297338

    2007 doi

  18. [26]

    The Control of the False Discovery Rate in Multiple Testing under Dependency

    Yoav Benjamini and Daniel Yekutieli. “The Control of the False Discovery Rate in Multiple Testing under Dependency”. In:The Annals of Statistics29.4 (2001), pp. 1165–1188.issn: 0090-5364. url: https://www.jstor.org/stable/2674075

    2001

  19. [27]

    OpenML:NetworkedScienceinMachineLearning

    JoaquinVanschorenetal.“OpenML:NetworkedScienceinMachineLearning”.In: SIGKDD Explor. Newsl.15.2 (2014), pp. 49–60.doi: 10.1145/2641190.2641198

    2014

  20. [28]

    The UCI Machine Learning Repository

    Markelle Kelly, Rachel Longjohn, and Kolby Nottingham. The UCI Machine Learning Repository. url: https://archive.ics.uci.edu (visited on 02/27/2025)

    2025

  21. [29]

    XGBoost: A Scalable Tree Boosting System

    Tianqi Chen and Carlos Guestrin. “XGBoost: A Scalable Tree Boosting System”. In:Pro- ceedings of the 22nd ACM SIGKDD. 2016, pp. 785–794

    2016

  22. [30]

    SliceLine: Fast, Linear-Algebra-based Slice Finding for ML Model Debugging

    Svetlana Sagadeeva and Matthias Boehm. “SliceLine: Fast, Linear-Algebra-based Slice Finding for ML Model Debugging”. In:Proceedings of the 2021 International Conference on Management of Data. SIGMOD ’21. New York, NY, USA: Association for Computing Ma- chinery, 2021, pp. 2290–...

    2021

  23. [31]

    Neill.Identifying Significant Predictive Bias in Classifiers

    Zhe Zhang and Daniel B. Neill.Identifying Significant Predictive Bias in Classifiers. Pre- sented as a poster at the 2017 Workshop on Fairness, Accountability, and Transparency in Machine Learning (FAT/ML 2017). 2017.doi: 10.48550/arXiv.1611.08292

    2017 doi arXiv

  24. [32]

    SliceTeller: A Data Slice-Driven Approach for Machine Learning Model Validation

    Xiaoyu Zhang et al. “SliceTeller: A Data Slice-Driven Approach for Machine Learning Model Validation”. In:IEEE Transactions on Visualization and Computer Graphics29.1 (2023), pp. 842–852.issn: 1941-0506. doi: 10.1109/TVCG.2022.3209465

    2023

  25. [33]

    Evaluating the Fairness of Predic- tive Student Models Through Slicing Analysis

    Josh Gardner, Christopher Brooks, and Ryan Baker. “Evaluating the Fairness of Predic- tive Student Models Through Slicing Analysis”. In:Proceedings of the 9th International Conference on Learning Analytics & Knowledge. LAK19. New York, NY, USA: Associa- tion for Computing Mach...

    2019

  26. [34]

    Deep ROC Analysis and AUC as Balanced Average Accu- racy, for Improved Classifier Selection, Audit and Explanation

    André M. Carrington et al. “Deep ROC Analysis and AUC as Balanced Average Accu- racy, for Improved Classifier Selection, Audit and Explanation”. In:IEEE Transactions on Pattern Analysis and Machine Intelligence45.1 (2023), pp. 329–341.issn: 1939-3539.doi: 10.1109/TPAMI.2022.3145392

    2023

  27. [35]

    Arya Tafvizi, Besim Avci, and Mukund Sundararajan.Attributing AUC-ROC to Analyze Binary Classifier Performance. en. Preprint. 2022.doi: 10.48550/arXiv.2205.11781. 15

    2022 doi arXiv

  28. [36]

    Henrik Grosskreutz and Daniel Paurat.Fast discovery of relevant subgroups using a reduced search space. en. Tech. rep. Fraunhofer IAIS, 2010.doi: 10.24406/publica-fhg-295070

    2010 doi

  29. [37]

    Fast exhaustive subgroup dis- covery with numerical target concepts

    Florian Lemmerich, Martin Atzmueller, and Frank Puppe. “Fast exhaustive subgroup dis- covery with numerical target concepts”. en. In:Data Mining and Knowledge Discovery30.3 (2016), pp. 711–762.issn: 1573-756X. doi: 10.1007/s10618-015-0436-8

    2016 doi

  30. [38]

    Discovering Robustly Connected Sub- graphs with Simple Descriptions

    Janis Kalofolias, Mario Boley, and Jilles Vreeken. “Discovering Robustly Connected Sub- graphs with Simple Descriptions”. In:2019 IEEE International Conference on Data Mining (ICDM). ISSN: 2374-8486. 2019, pp. 1150–1155.doi: 10.1109/ICDM.2019.00139

    2019

  31. [39]

    Identifying exceptional (dis)agreement between groups

    Adnene Belfodil et al. “Identifying exceptional (dis)agreement between groups”. en. In: Data Mining and Knowledge Discovery 34.2 (2020), pp. 394–442. issn: 1384-5810. doi: 10.1007/s10618-019-00665-9

    2020 doi

  32. [40]

    Mathematics of Multisets

    Apostolos Syropoulos. “Mathematics of Multisets”. en. In:Multiset Processing. Ed. by Cris- tian S. Calude et al. Berlin, Heidelberg: Springer Berlin Heidelberg, 2001, pp. 347–358. isbn: 978-3-540-45523-3. doi: 10.1007/3-540-45523-X_17

    2001 doi

  33. [41]

    An introduction to ROC analysis

    Tom Fawcett. “An introduction to ROC analysis”. In:Pattern Recognition Letters. ROC Analysis in Pattern Recognition 27.8 (2006), pp. 861–874.issn: 0167-8655. doi: 10.1016/ j.patrec.2005.10.010. 16 A Proofs of Tight Optimistic Estimates Firstly this section generalizes and form...

    2006

  34. [45]

    It follows ∀p′⊃p :sg(p′)⊆sg(p)

    states that for all two rulespgen and pspec the following statement holds pgen⊂pspec⇒sg(pgen)⊇sg(pspec). It follows ∀p′⊃p :sg(p′)⊆sg(p). From the definition of databases follows ∀p′⊃p :sg(p′){y, ˆy}⊆sg(p){y, ˆy}. (4) Because bml is defined to be a soft classifier performance m...

  35. [46]

    From the definition ofbROCAUC we know that in this case ∀(y, ˆy), (y′, ˆy′)∈C : (y <y′→ ˆy≤ ˆy′)∧ (y >y′→ ˆy≥ ˆy′) (24) holds

    Therefore the tightness criterion is fulfilled in this case. From the definition ofbROCAUC we know that in this case ∀(y, ˆy), (y′, ˆy′)∈C : (y <y′→ ˆy≤ ˆy′)∧ (y >y′→ ˆy≥ ˆy′) (24) holds. This condition also holds for any sub-multiset ofC, that contains at least one instance w...

  36. [47]

    Therefore ROCAUC (C′) < 1 2 implies that at least one supporting point of the ROC curve ofC′ lies below the diagonal ROC curve

    Clearly any ROC curve with only supporting points(0, 0), (1, 1) and additional points above or on the diagonal ROC curve has an AUC of1 2 +x with x∈ R≥0. Therefore ROCAUC (C′) < 1 2 implies that at least one supporting point of the ROC curve ofC′ lies below the diagonal ROC cu...

  37. [48]

    border” is the actual border of the achievable area. The curves named “approximation 1

    The curve named “border” is the actual border of the achievable area. The curves named “approximation 1” and “approximation 2” are the (lin- early interpolated) PR curves for the multisets of instance values[(1, 0), (1, 0), (0, 1), (0, 1)] and [(1, 0), (1, 0.5), (0, 1), (0, 1)...

    2021

  38. [240]

    doi: 10.1145/1143844.1143874

    isbn: 978-1-59593-383-6. doi: 10.1145/1143844.1143874

  39. [1626]

    url: https://icml.cc/2012/papers/349.pdf

    isbn: 9781450312851. url: https://icml.cc/2012/papers/349.pdf

    2012

  40. [8486]

    360–369.doi: 10.1109/ICDM.2011.94

    2011, pp. 360–369.doi: 10.1109/ICDM.2011.94

    2011 doi

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.