Pith. sign in

REVIEW 3 major objections 5 minor 116 references

Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read By averaging net benefit over a user-specified prevalence interval, a pointwise clipped log score yields one number that ranks clinical classifiers on calibration, label-shift robustness, and asymmetric error costs.

desk verdict A useful closed-form evaluation metric, but the main theorem has a factor-of-two error that needs fixing before citation. read the letter →

arxiv 2506.14540 v3 pith:HESM7PAA submitted 2025-06-17 cs.LG cs.AI

Gerardo A. Flores , Alyssa H. Smith , Julia A. Fukuyama , Ashia C. Wilson This is my paper · ORCID
classification cs.LGcs.AI MSC 62H3062C05
keywords modelevaluationcalibrationlabelshiftasymmetricerrorcostsproperscoringrulesclippedcrossentropyclinicaldecisionsupportAUC-ROC
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to give clinical machine learning a single evaluation score that means something at the bedside: the expected net benefit of a thresholded classifier averaged over the range of disease prevalences a clinic expects, with the relative cost of false positives and false negatives stated explicitly. The authors derive a clipped cross-entropy (bounded DCA log score) whose per-patient average equals that expected net benefit up to a constant, so no binning or ROC curves are needed. If the derivation is right, model selection in medicine could move from accuracy and AUC-ROC to a pointwise metric that rewards calibration exactly where decisions are made and that is trivial to bootstrap for confidence intervals.

What carries the argument

The carrying object is the Schervish representation, the integral representation showing that any proper scoring rule is a weighted average of cost-weighted binary losses; designing a scoring rule is therefore choosing the mixture over class balances. The paper takes a logit-uniform prior on prevalence within user-supplied bounds $[a,b]$, adjusts scores for label shift with odds multiplication $a\otimes b$, and restricts attention to the clinically relevant score band with the clipping operator $\mathrm{clip}_{[1-b,1-a]}$. The pivotal Lemma C.6 converts an average over prevalence of a cost-weighted correctness indicator into an integral of a log-density over the clipped interval, which is exactly what turns the average net benefit into a closed-form, pointwise log-score difference.

What would settle it

Take a deliberately miscalibrated score function (for example, a calibrated score passed through a monotone but non-identity transform), fix cost $c$ and prevalence bounds $a$, $b$, and compare the closed-form bounded DCA log score of Theorem 5.1 with the prior-adjusted maximum net benefit simulated directly at each prevalence in $[a,b]$. If the two disagree by more than sampling error, or if the metric ranks two classifiers opposite to the simulated expected net benefit, the coherence assumption is violated and the metric's clinical-interpretation claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that one decision-theoretic metric can simultaneously handle calibration, label-shift uncertainty, and asymmetric error costs. Theorems 5.1 and D.3 state that the prior-adjusted maximum net benefit averaged over a logit-uniform distribution of deployment prevalence between bounds $a$ and $b$ equals, up to a constant, a clipped log-score difference: $$\mathbb{E}[\mathrm{PAMNB}(D_\pi,s,\tau,c)] = \gamma\left(\mathbb{E}\left[\log|1-y-\mathrm{clip}_{[1-b,1-a]}(1-c\otimes s_{1/2}(x))| - \log|1-y-\mathrm{clip}_{[1-b,1-a]}(1-y)|\right]\right),$$ with the left expectation over prevalence and the right over cost-shifted data. This makes the metric a pointwise sum of per-patient losses in true-positive net-benefit units, so confidence intervals can be bootstrapped directly. The paper argues this corrects the two standard alternatives: accuracy fixes one operating point and ignores costs, while AUC-ROC ignores calibration and averages over a prevalence distribution induced by the model's own scores rather than one set by the clinical context.

Load-bearing premise

The expected-net-benefit interpretation assumes the model's original probability score is already calibrated on the development data (the paper calls this coherence); for a miscalibrated score the formula is an average over decisions that trust unreliable probabilities.

Editorial extensions

If this is right

  • A single scalar, the bounded DCA log score, ranks classifiers by expected clinical net benefit over a user-specified prevalence interval, with units of a true positive.
  • Because the score is an average of per-patient clipped log losses, bootstrap confidence intervals for model comparisons follow immediately from the same pointwise losses.
  • AUC-ROC is reinterpreted as accuracy averaged over a prevalence distribution induced by the model's score distribution on balanced data, making explicit why it is calibration-blind and model-dependent in its weighting.
  • The same clipped log-score construction yields a weighted-accuracy variant (Theorem D.2) whose perfect classifier always scores 1, giving a normalized companion metric.
  • On eICU in-hospital mortality predictions, accuracy and AUC-ROC can both favor one subgroup while the cost-weighted expected benefit favors the other; decompositions trace the gap to label shift and calibration rather than to the mechanism of prediction.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same clipped log score could be used as a differentiable training objective, turning clinical net benefit into a loss to optimize rather than only a metric to evaluate; the paper does not make this move.
  • The clinical meaning depends on the prevalence bounds $[a,b]$ being honest: if the true deployment prevalence falls outside the bound, the clipped term can dominate the score and distort rankings, so reporting sensitivity to the bounds is a natural safeguard.
  • The logit-uniform prior is a modeling choice; comparing model rankings under logit-uniform versus uniform-in-probability prevalence weighting would show whether clinical selections are robust to the shape of the prior.
  • The causal decomposition used for eICU subgroups could be repurposed as a fairness audit tool, separating base-rate differences from mechanism differences in any setting where label shift is plausible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a framework for evaluating binary clinical classifiers with three desiderata: calibration, robustness to label shift, and asymmetric error costs. It critiques accuracy and AUC-ROC, develops prior-adjusted cost-weighted metrics (PAMA, PAMNB), and derives a clipped log-score identity (Theorem 5.1) that expresses the net benefit averaged over a logit-uniform interval of prevalences as a pointwise-computable loss. The paper also gives an eICU subgroup analysis illustrating how the proposed metric and its decompositions can expose label-shift and calibration effects that accuracy and AUC-ROC miss.

Significance. If the central identity is correct, the paper provides a practical, pointwise-computable evaluation metric that jointly accounts for cost asymmetry, label-shift uncertainty, and calibration, and it gives a useful decision-theoretic interpretation of AUC-ROC as averaging accuracy over label shift. The eICU decomposition is a concrete illustration of the metric's diagnostic value, and the connection to the Schervish representation is conceptually valuable. The paper is clearly written and the appendices make a serious effort to provide derivations. However, the central theorem contains a normalization error and a sign inconsistency in the supporting theorems, and the proof of the key lemma has a gap; these issues must be fixed before the main claims can be accepted.

major comments (3)
  1. [§5.2, Theorem 5.1; Appendix D.3] The normalization constant γ in Theorem 5.1 is off by a factor of 2. In the proof of Theorem D.3, the step 'and we use importance sampling' replaces E_{D_{1/2}}[c^{1-y}(1-c)^y f] by E_{D_{(1-c)}}[f]; the correct identity is E_{D_{1/2}}[c^{1-y}(1-c)^y f] = (1/2) E_{D_{(1-c)}}[f], because the importance weight from D_{1/2} to D_{(1-c)} is 2c^{1-y}(1-c)^y. The published prefactor is therefore 2(1-c)^{-1}/Δ instead of (1-c)^{-1}/Δ. At c=1/2 this contradicts Theorem D.1, and a two-point check confirms the smaller constant. Since γ is model-independent, rankings for fixed (a,b,c) are unaffected, but the stated equality and the claim that the metric is denominated in units of true-positive net benefit are invalid as written. The sign of the two log terms also differs between Theorem 5.1 and Theorems D.1/D.3; this should be reconciled.
  2. [Appendix C, Lemma C.6] The proof of Lemma C.6 drops absolute values: after the substitution p=1-π, the integrand is displayed as 2(p-y)C(1-p;y) dp, while the previous line contains 2|p-y|C(1-p;y) dp. For y=1 this is not cosmetic, because the integration limits are reversed and the orientation matters. The manuscript needs a complete case analysis on y (or a formal verification) showing that the absolute value and the integration limits produce the stated clipped integral; as written, this central lemma is not fully established.
  3. [§3, Definition 3.3; §5] The decision-theoretic interpretation of PAMNB and of Theorem 5.1 requires the score function s to be coherent, so that π⊗s_{1/2}(x) is the true posterior under label shift and threshold c is optimal. The paper's stated goal, however, is to evaluate possibly miscalibrated models. For miscalibrated s, the left-hand side of Theorem 5.1 is the expected net benefit of a decision rule that trusts the uncalibrated score, not the maximum net benefit under an optimal threshold. The manuscript should either define the metric for arbitrary s as the net benefit of that fixed score-based rule (and adjust the name and interpretation accordingly), or state explicitly and prove what the clipped log score measures for miscalibrated s.
minor comments (5)
  1. [Appendix B, Definition B.9] The first factor in the PAMNB formula is written with exponent 1-y; it should be y to match Definition 3.4 and the surrounding text.
  2. [Appendix D.3] The notation s_{(1-c)}(x) is used without definition; it should be defined as (1-c)⊗s_{1/2}(x) to match Theorem 5.1.
  3. [Appendix D, Theorems D.1 and D.2] The order of the two log terms in these theorems is the reverse of that in Theorem 5.1; since these theorems are used to justify the main result, the sign convention should be made consistent.
  4. [§7.1] The Cost Uncertainty paragraph refers to an extension of the DCA log score to uncertain cost ratios with dilogarithmic expressions, but no such extension appears in the body of the paper; either add the derivation or remove the claim.
  5. [Appendix G] The figure captions are duplicated (both panels labeled (C)), and the text refers to panels (A) and (B) that are not labeled in the captions; relabel the figures for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the DCA log-score identity is a derived change-of-measure result; a, b, and c are user inputs and no fitted parameter is relabeled as a prediction.

full rationale

The derivation chain is self-contained. The central Theorem 5.1 and its appendix proofs (Lemma C.6, Theorems D.1 and D.3) begin from the explicit definition of PAMNB and proceed by importance sampling, set-membership recoding, and antidifferentiation; the log-score expression on the right-hand side is a derived quantity, not the definition of PAMNB. The prevalence bounds a and b and the cost ratio c are user-specified inputs, and the empirical prevalence π0 is used only through standard importance-sampling weights, not as a fitted parameter subsequently called a prediction. The Schervish representation and related scoring-rule results are cited from external literature, and no load-bearing claim rests on a self-citation or on a uniqueness theorem imported from the authors. The coherence/calibration assumption that connects PAMNB to clinical expected net benefit is an explicit interpretative premise, not a circular redefinition. A separate mathematical concern is that the proof of Theorem D.3 appears to drop a factor of 1/2 when replacing E_{D1/2}[c^{1-y}(1-c)^y f] with E_{D(1-c)}[f], which would change the prefactor in Theorem 5.1 by a factor of 2; this is a correctness issue concerning the constant, not a circularity in the derivation structure.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central formulas rest on standard probability theory plus three domain assumptions: label shift, coherence of the score, logit-uniform prevalence prior, and a single-parameter cost model. No entities are invented.

free parameters (3)
  • a
    User-specified lower bound on deployment class prevalence π; defines the averaging interval. Chosen post hoc in the eICU example.
  • b
    User-specified upper bound on deployment class prevalence π; defines the averaging interval. Chosen post hoc in the eICU example.
  • c
    Cost ratio encoding relative cost of false positives to false negatives; user-specified, not fitted. The DCA log score averages over prevalences at fixed c.
assumptions (4)
  • domain assumption Label shift structure Dπ → Y → X, with P(X|Y,Dπ)=P(X|Y).
    Invoked in Section 3 (Label Shift) and used in Appendices A.3 and C to justify importance sampling and prior adjustment. If deployment differs by covariate shift instead of label shift, the formulas do not apply.
  • domain assumption The original score s is coherent/calibrated on the development distribution, so the Bayes-adjusted score π⊗s1/2(x) is the optimal threshold.
    Explicitly required in Section 3 under Definition 3.3 ('requires that our original score be probabilistically meaningful'). Without it, the DCA log score is not an expected net benefit but a formal average of a decision rule that trusts uncalibrated probabilities.
  • ad hoc to paper Prevalence uncertainty is modeled as σ^{-1}(π) ~ Uniform(σ^{-1}(a), σ^{-1}(b)).
    Chosen in Section 5.2 for interpretability ('average uniformly over the log odds'). This is a modeling choice, not derived from clinical practice, and different priors change the metric.
  • domain assumption Asymmetric error costs are fully captured by a single ratio c ∈ (0,1).
    Used throughout Definitions 3.2, B.6, and Theorem 5.1. More complex cost structures (e.g., costs depending on patient subgroup) are out of scope.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs." pith.science (2026). https://pith.science/paper/HESM7PAA

@misc{pith2026250614540,
  author       = {Pith},
  title        = {Pith review of: Aligning Evaluation with Clinical Priorities: Calibration, Label Shift, and Error Costs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HESM7PAA}},
  note         = {Machine review of arXiv:2506.14540}
}
read the original abstract

Machine learning-based decision support systems are increasingly deployed in clinical settings, where probabilistic scoring functions are used to inform and prioritize patient management decisions. However, widely used scoring rules, such as accuracy and AUC-ROC, fail to adequately reflect key clinical priorities, including calibration, robustness to distributional shifts, and sensitivity to asymmetric error costs. In this work, we propose a principled yet practical evaluation framework for selecting calibrated thresholded classifiers that explicitly accounts for the uncertainty in class prevalences and domain-specific cost asymmetries often found in clinical settings. Building on the theory of proper scoring rules, particularly the Schervish representation, we derive an adjusted variant of cross-entropy (log score) that averages cost-weighted performance over clinically relevant ranges of class balance. The resulting evaluation is simple to apply, sensitive to clinical deployment conditions, and designed to prioritize models that are both calibrated and robust to real-world variations.

Figures

Figures reproduced from arXiv: 2506.14540 by the authors.

Figure 1
Figure 1. Causal diagrams: (1) shows differences in performance are based both on label shift and [PITH_FULL_IMAGE:figures/full_fig_p042_1.png] view at source ↗
Figure 2
Figure 2. The label shift effect is quite dramatic on the public subsample of EICU. In fact a great [PITH_FULL_IMAGE:figures/full_fig_p042_2.png] view at source ↗
Figure 3
Figure 3. The difference in sharpness is fairly dramatic on the public subsample of EICU, but [PITH_FULL_IMAGE:figures/full_fig_p043_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The difference in sharpness for clipped cross entropy is not exactly the same as the difference [PITH_FULL_IMAGE:figures/full_fig_p044_4.png]
Figure 5
Figure 5. Figure 5: There’s a significant gap between the performance on the public dataset for black and [PITH_FULL_IMAGE:figures/full_fig_p045_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

116 extracted references · 33 canonical work pages

  1. [1]

    Adams and D

    N. Adams and D. Hand. Comparing classifiers when the misallocation costs are uncer- tain. Pattern Recognition, 32(7):1139–1147, 1999. ISSN 0031-3203. doi: https://doi.org/ 10.1016/S0031-3203(98)00154-X. URL https://www.sciencedirect.com/science/article/ pii/S003132039800154X

    doi:10.1016/s0031-3203(98)00154-x 1999

  2. [2]

    Angstrom

    A. Angstrom. On the effectivity of weather warnings.Nordisk Statistisk Tidskrift, 1:394–408, 1922

    1922

  3. [3]

    Assel, D

    M. Assel, D. D. Sjoberg, and A. J. Vickers. The brier score does not evaluate the clinical utility of diagnostic tests or prediction models.Diagnostic and Prognostic Research, 1(1):19, 2017. doi: 10.1186/s41512-017-0020-3. URL https://doi.org/10.1186/s41512-017-0020-3

    doi:10.1186/s41512-017-0020-3 2017

  4. [4]

    M. Ayer, H. D. Brunk, G. M. Ewing, W. T. Reid, and E. Silverman. An empirical distribution function for sampling with incomplete information.The Annals of Mathematical Statistics, 26 (4):641–647, 1955. ISSN 00034851. URLhttp://www.jstor.org/stable/2236377

    work page arXiv 1955

  5. [5]

    Azizzadenesheli, A

    K. Azizzadenesheli, A. Liu, F. Yang, and A. Anandkumar. Regularized learning for domain adaptation under label shifts. InInternational Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=rJl0r3R9KX

    2019

  6. [6]

    D. Bamber. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. Journal of Mathematical Psychology, 12(4):387–415, 1975. ISSN 0022-2496. doi: https://doi.org/10.1016/0022-2496(75)90001-2. URL https://www. sciencedirect.com/science/article/pii/0022249675900012

    arXiv 1975

  7. [7]

    Ben-David, J

    S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of learning from different domains. Machine Learning, 79(1):151–175, 2010. doi: 10.1007/ s10994-009-5152-4. URL https://doi.org/10.1007/s10994-009-5152-4

    doi:10.1007/s10994-009-5152-4 2010

  8. [8]

    Blockeel, K

    H. Blockeel, K. Kersting, S. Nijssen, and F. Železný, editors.Area under the Precision-Recall Curve: Point Estimates and Confidence Intervals, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-40994-3

    2013

Show all 116 references
  1. [9]

    A. P. Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Pattern Recognition, 30(7):1145–1159, 1997. ISSN 0031-3203

    1997

  2. [10]

    G. W. Brier. Verification of forecasts expressed in terms of probability.Monthly Weather Review, 78:1–3, 1950. URL https://api.semanticscholar.org/CorpusID:122906757. 13

    1950

  3. [11]

    A. M. Carrington, D. G. Manuel, P. W. Fieguth, T. Ramsay, V. Osmani, B. Wernly, C. Bennett, S. Hawken, O. Magwood, Y. Sheikh, M. McInnes, and A. Holzinger. Deep roc analysis and auc as balanced average accuracy, for improved classifier selection, audit and explanation.IEEE Tra...

    2023

  4. [12]

    Corbett-Davies, E

    S. Corbett-Davies, E. Pierson, A. Feller, S. Goel, and A. Huq. Algorithmic decision making and the cost of fairness. InProceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’17, pages 797–806, New York, NY, USA, 2017. Associati...

    2017

  5. [13]

    Cruz Rivera, X

    S. Cruz Rivera, X. Liu, A.-W. Chan, A. K. Denniston, M. J. Calvert, H. Ashrafian, A. L. Beam, G. S. Collins, A. Darzi, J. J. Deeks, M. K. ElZarrad, C. Espinoza, A. Esteva, L. Faes, L. Ferrante di Ruffano, J. Fletcher, R. Golub, H. Harvey, C. Haug, C. Holmes, A. Jonas, P. A. Ke...

    2025

  6. [14]

    Davis and M

    J. Davis and M. Goadrich. The relationship between precision-recall and roc curves. In Proceedings of the 23rd International Conference on Machine Learning, ICML ’06, pages 233– 240, New York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1145/11...

    2006

  7. [15]

    de Finetti

    B. de Finetti. La prévision : ses lois logiques, ses sources subjectives.Annales de l’institut Henri Poincaré, 7(1):1–68, 1937. URLhttp://eudml.org/doc/79004

    1937

  8. [16]

    de Finetti.Foresight: Its Logical Laws, Its Subjective Sources, pages 134–174

    B. de Finetti.Foresight: Its Logical Laws, Its Subjective Sources, pages 134–174. Springer New York, New York, NY, 1992. ISBN 978-1-4612-0919-5. doi: 10.1007/978-1-4612-0919-5_10. URL https://doi.org/10.1007/978-1-4612-0919-5_10

    1992 doi

  9. [17]

    Dimitriadis, T

    T. Dimitriadis, T. Gneiting, A. I. Jordan, and P. Vogel. Evaluating probabilistic classifiers: The triptych. International Journal of Forecasting, 40(3):1101–1122, 2024. ISSN 0169-2070. doi: https://doi.org/10.1016/j.ijforecast.2023.09.007. URL https://www.sciencedirect.com/ s...

    2024 doi

  10. [18]

    Domingos

    P. Domingos. Metacost: a general method for making classifiers cost-sensitive. InProceedings of the Fifth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’99, pages 155–164, New York, NY, USA, 1999. Association for Computing Machinery. ISBN 1581...

    1999

  11. [19]

    Drummond and R

    C. Drummond and R. C. Holte. Explicitly representing expected cost: an alternative to roc representation. In Proceedings of the Sixth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’00, pages 198–207, New York, NY, USA, 2000. Association for Co...

  12. [20]

    Drummond and R

    C. Drummond and R. C. Holte. Cost curves: An improved method for visualizing classifier performance. Machine Learning, 65(1):95–130, 2006. doi: 10.1007/s10994-006-8199-5. URL https://doi.org/10.1007/s10994-006-8199-5

    2006 doi

  13. [21]

    J. C. Duchi and H. Namkoong. Learning models with uniform performance via distributionally robust optimization.The Annals of Statistics, 49(3):1378 – 1406, 2021. doi: 10.1214/20-AOS2004. URL https://doi.org/10.1214/20-AOS2004

    2021 doi

  14. [22]

    W. Ehm, T. Gneiting, A. Jordan, and F. Krüger. Of quantiles and expectiles: Consistent scoring functions, choquet representations and forecast rankings.Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(3):505–562, 05 2016. ISSN 1369-7412. doi: 10.1...

    2016 doi

  15. [23]

    C. Elkan. The foundations of cost-sensitive learning. InProceedings of the 17th International Joint Conference on Artificial Intelligence - Volume 2, IJCAI’01, pages 973–978, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608125

    2001

  16. [24]

    Fawcett and A

    T. Fawcett and A. Niculescu-Mizil. Pav and the roc convex hull.Machine Learning, 68(1):97–106,

  17. [25]

    Flach, J

    P. Flach, J. Hernández-Orallo, and C. Ferri. A coherent interpretation of auc as a measure of aggregated classification performance. InProceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, pages 657–664, Madison, WI, USA,

  18. [26]

    Ganin, E

    Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. March, and V. Lempitsky. Domain-adversarial training of neural networks.Journal of Machine Learning Research, 17(59):1–35, 2016. URLhttp://jmlr.org/papers/v17/15-239.html

    2016

  19. [27]

    S. Garg, Y. Wu, S. Balakrishnan, and Z. Lipton. A unified view of label shift estima- tion. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 3290–3300. Curran Asso- ciates, Inc., 202...

    2020

  20. [28]

    Gneiting and A

    T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378, 2007. doi: 10.1198/ 016214506000001437. URL https://doi.org/10.1198/016214506000001437

    2007 doi

  21. [29]

    Goldberger, L

    A. Goldberger, L. Amaral, L. Glass, J. Hausdorff, P. C. Ivanov, R. Mark, J. E. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. Physiobank, physiotoolkit, and physionet: Components of a new research resource for complex physiologic signals.Circulation, 101(23):e215–e220, 20...

    2000

  22. [30]

    I. J. Good. Rational decisions.Journal of the Royal Statistical Society. Series B (Methodological), 14(1):107–114, 1952. ISSN 00359246. URLhttp://www.jstor.org/stable/2984087

    1952

  23. [31]

    C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, pages 1321–1330. JMLR.org, 2017. 15

    2017

  24. [32]

    Hand and C

    D. Hand and C. Anagnostopoulos. A better beta for the h measure of classification perfor- mance. Pattern Recognition Letters, 40:41–46, 2014. ISSN 0167-8655. doi: https://doi.org/ 10.1016/j.patrec.2013.12.011. URL https://www.sciencedirect.com/science/article/pii/ S0167865513004984

    2014 doi

  25. [33]

    D. J. Hand. Measuring classifier performance: a coherent alternative to the area under the roc curve. Machine Learning, 77(1):103–123, 2009. doi: 10.1007/s10994-009-5119-5. URL https://doi.org/10.1007/s10994-009-5119-5

    2009 doi

  26. [34]

    D. J. Hand and C. Anagnostopoulos. Notes on the h-measure of classifier performance.Advances in Data Analysis and Classification, 17(1):109–124, 2023. doi: 10.1007/s11634-021-00490-3. URL https://doi.org/10.1007/s11634-021-00490-3

    2023 doi

  27. [35]

    J. A. Hanley and B. J. McNeil. The meaning and use of the area under a receiver operating characteristic (roc) curve.Radiology, 143(1):29–36, 1982. ISSN 0033-8419

    1982

  28. [36]

    Hebert-Johnson, M

    U. Hebert-Johnson, M. Kim, O. Reingold, and G. Rothblum. Multicalibration: Calibration for the (Computationally-identifiable) masses. In J. Dy and A. Krause, editors,Proceedings of the 35th International Conference on Machine Learning, volume 80 ofProceedings of Machine Learni...

    1939

  29. [37]

    J. Heckman. Shadow prices, market wages, and labor supply.Econometrica, 42(4):679–694,

  30. [38]

    J. J. Heckman. Sample selection bias as a specification error.Econometrica, 47(1):153–161,

  31. [39]

    Hernández-Orallo, P

    J. Hernández-Orallo, P. Flach, and C. Ferri. Brier curves: a new cost-based visualisation of classifier performance. InProceedings of the 28th International Conference on International Conference on Machine Learning, ICML’11, pages 585–592, Madison, WI, USA, 2011. Omnipress. I...

    2011

  32. [40]

    Hernandez-Orallo, P

    J. Hernandez-Orallo, P. Flach, and C. Ferri. Threshold choice methods: the missing link. 12 2011

    2011

  33. [41]

    Hernández-Orallo, P

    J. Hernández-Orallo, P. Flach, and C. Ferri. A unified view of performance metrics: translating threshold choice into expected classification loss.J. Mach. Learn. Res., 13(1):2813–2869, 10 2012

    2012

  34. [42]

    Hernández-Orallo, P

    J. Hernández-Orallo, P. Flach, and C. Ferri. Roc curves in cost space. Machine Learn- ing, 93(1):71–91, 2013. doi: 10.1007/s10994-013-5328-9. URL https://doi.org/10.1007/ s10994-013-5328-9

    2013 doi

  35. [43]

    Huang and C

    J. Huang and C. Ling. Using auc and accuracy in evaluating learning algorithms.IEEE Transactions on Knowledge and Data Engineering, 17:299–310, 2005. doi: 10.1109/TKDE.2005. 50

    2005 doi

  36. [44]

    Johnson, T

    A. Johnson, T. Pollard, O. Badawi, and J. Raffa. eicu collaborative research database demo (version 2.0.1).PhysioNet, 2021. doi: 10.13026/4mxk-na84. URLhttps://doi.org/10.13026/ 4mxk-na84. 16

    2021 doi

  37. [45]

    Kallus and A

    N. Kallus and A. Zhou.The fairness of risk scores beyond classification: bipartite ranking and the xAUC metric. Curran Associates Inc., Red Hook, NY, USA, 2019

    2019

  38. [46]

    E. M. Keen. Measures and averaging methods used in performance testing of indexing systems. Technical report, The College of Aeronautics, Cranfield, England, 1966. URLhttps://sigir. org/resources/museum/. Available in the SIGIR Museum resources collection

    1966

  39. [47]

    E. M. Keen. Evaluation parameters. Scientific Report ISR-13, Department of Computer Science, Cornell University, Ithaca, New York, 1968. Information Storage and Retrieval: Scientific Report No. ISR-13 to the National Science Foundation

    1968

  40. [48]

    Kleinberg, S

    J. Kleinberg, S. Mullainathan, and M. Raghavan. Inherent trade-offs in the fair determination of risk scores. In C. H. Papadimitriou, editor,8th Innovations in Theoretical Computer Science Conference (ITCS 2017), volume 67 ofLeibniz International Proceedings in Informatics (LI...

    2017 doi

  41. [49]

    M. Kull, T. S. Filho, and P. Flach. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In A. Singh and J. Zhu, editors,Pro- ceedings of the 20th International Conference on Artificial Intelligence and Statistics,...

    2017

  42. [50]

    Lipton, Y.-X

    Z. Lipton, Y.-X. Wang, and A. Smola. Detecting and correcting for label shift with black box predictors. 02 2018. doi: 10.48550/arXiv.1802.03916

    2018 doi arXiv

  43. [51]

    X. Liu, S. Cruz Rivera, D. Moher, M. J. Calvert, A. K. Denniston, A.-W. Chan, A. Darzi, C. Holmes, C. Yau, H. Ashrafian, J. J. Deeks, L. Ferrante di Ruffano, L. Faes, P. A. Keane, S. J. Vollmer, A. Y. Lee, A. Jonas, A. Esteva, A. L. Beam, M. B. Panico, C. S. Lee, C. Haug, C. J...

    2020

  44. [52]

    M. Long, Y. Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In F. Bach and D. Blei, editors,Proceedings of the 32nd International Conference on Machine Learning, volume 37 ofProceedings of Machine Learning Research, pages 97–105, Lill...

    2015

  45. [53]

    McCarthy

    J. McCarthy. Measures of the value of information.Proceedings of the National Academy of Sciences, 42(9):654–655, 1956. doi: 10.1073/pnas.42.9.654. URLhttps://www.pnas.org/doi/ abs/10.1073/pnas.42.9.654

    1956 doi

  46. [54]

    D. K. McClish. Analyzing a portion of the roc curve.Med Decis Making, 9(3):190–195, 1989. ISSN 0272-989X (Print); 0272-989X (Linking). doi: 10.1177/0272989X8900900307

    1989 doi

  47. [55]

    D. K. McClish. Evaluation of the accuracy of medical tests in a region around the optimal point. Academic Radiology, 19(12):1484–1490, 2025/05/05 2012. doi: 10.1016/j.acra.2012.09.004. URL https://doi.org/10.1016/j.acra.2012.09.004. 17

    2025 doi

  48. [56]

    M. B. McDermott, H. Zhang, L. H. Hansen, G. Angelotti, and J. Gallifant. A closer look at AUROC and AUPRC under class imbalance. InThe Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URLhttps://openreview.net/forum?id=S3HvA808gk

    2024

  49. [57]

    P. E. Meehl and A. Rosen. Antecedent probability and the efficiency of psychometric signs, patterns, or cutting scores.Psychological Bulletin, 52(3):194–216, 1955. doi: 10.1037/h0048070

    1955 doi

  50. [58]

    C. E. Metz. Basic principles of roc analysis.Semin Nucl Med, 8(4):283–98, Oct 1978. doi: 10.1016/s0001-2998(78)80014-2

    1978 doi

  51. [59]

    C. E. Metz. Roc methodology in radiologic imaging.Invest Radiol, 21(9):720–733, Sep 1986. ISSN 0020-9996 (Print); 0020-9996 (Linking). doi: 10.1097/00004424-198609000-00009

    1986 doi

  52. [60]

    C. E. Metz. Some practical issues of experimental design and data analysis in radiological roc studies. Invest Radiol, 24(3):234–245, Mar 1989. ISSN 0020-9996 (Print); 0020-9996 (Linking). doi: 10.1097/00004424-198903000-00012

    1989 doi

  53. [61]

    J. G. Moreno-Torres, T. Raeder, R. Alaiz-Rodríguez, N. V. Chawla, and F. Herrera. A unifying view on dataset shift in classification.Pattern Recognition, 45(1):521–530, 2012. ISSN 0031-3203. doi: https://doi.org/10.1016/j.patcog.2011.06.019. URL https://www.sciencedirect.com/ ...

    2012 doi

  54. [62]

    Muandet, D

    K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. InProceedings of the 30th International Conference on International Conference on Machine Learning - Volume 28, ICML’13, pages I–10–I–18. JMLR.org, 2013

    2013

  55. [63]

    A. H. Murphy. A note on the utility of probabilistic predictions and the probability score in the cost-loss ratio decision situation.Journal of Applied Meteorology and Climatology, 5(4):534 – 537,

  56. [64]

    A. H. Murphy. A new vector partition of the probability score.Journal of Applied Meteorology (1962-1982), 12(4):595–600, 1973. ISSN 00218952, 2163534X. URLhttp://www.jstor.org/ stable/26176769

    1962

  57. [65]

    A. H. Murphy. The value of climatological, categorical and probabilistic forecasts in the cost-loss ratio situation. Monthly Weather Review, 105(7):803 – 816, 1977. doi: 10.1175/1520-0493(1977) 105<0803:TVOCCA>2.0.CO;2. URL https://journals.ametsoc.org/view/journals/mwre/ 105/...

    1977 doi

  58. [66]

    A. H. Murphy and R. L. Winkler. A general framework for forecast verification.Monthly Weather Review, 115(7):1330 – 1338, 1987. doi: 10.1175/1520-0493(1987)115<1330:AGFFFV>2.0. CO;2. URL https://journals.ametsoc.org/view/journals/mwre/115/7/1520-0493_1987_ 115_1330_agfffv_2_0_co_2.xml

    1987 doi

  59. [67]

    Pakdaman Naeini, G

    M. Pakdaman Naeini, G. Cooper, and M. Hauskrecht. Obtaining well calibrated probabilities using bayesian binning. Proceedings of the AAAI Conference on Artificial Intelligence, 29 (1), Feb. 2015. doi: 10.1609/aaai.v29i1.9602. URLhttps://ojs.aaai.org/index.php/AAAI/ article/view/9602

    2015 doi

  60. [68]

    S. G. Pauker and J. P. Kassirer. Therapeutic decision making: A cost-benefit analysis.New England Journal of Medicine, 293(5):229–234, 1975. doi: 10.1056/NEJM197507312930505. URL https://www.nejm.org/doi/full/10.1056/NEJM197507312930505. 18

    1975 doi

  61. [69]

    J. Pearl. Causality : models, reasoning, and inference. Cambridge University Press, Cambridge [u.a.], repr. with corrections edition, 2001. ISBN 0521773628

    2001

  62. [70]

    Wesley and T

    Peterson, W. Wesley and T. G. Birdsall. The theory of signal detectability. Michigan. University. Department of Electrical Engineering. Electronic Defense Group. Technical report; no. 13. Engineering Research Institute, Ann Arbor, 1953

    1953

  63. [71]

    J. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. 1999. URL https://api.semanticscholar.org/CorpusID:56563878

    1999

  64. [72]

    Pollard, A

    T. Pollard, A. Johnson, J. Raffa, L. A. Celi, O. Badawi, and R. Mark. eicu collaborative research database (version 2.0). PhysioNet, 2019. doi: 10.13026/C2WM1R. URL https: //doi.org/10.13026/C2WM1R

    2019 doi

  65. [73]

    T. J. Pollard, A. E. W. Johnson, J. D. Raffa, L. A. Celi, R. G. Mark, and O. Badawi. The eicu collaborative research database, a freely available multi-center database for critical care research. Scientific Data, 2018. doi: 10.1038/sdata.2018.178. URL http://dx.doi.org/10. 103...

    2018 doi

  66. [74]

    F. J. Provost and T. Fawcett. Analysis and visualization of classifier performance: Comparison under imprecise class and cost distributions. InKnowledge Discovery and Data Mining, 1997. URL https://api.semanticscholar.org/CorpusID:157595

    1997

  67. [75]

    F. P. Ramsey. Truth and probability. In R. B. Braithwaite, editor, The Foundations of Mathematics and other Logical Essays, chapter 7, pages 156–198. McMaster University Archive for the History of Economic Thought, 1926. URLhttps://EconPapers.repec.org/RePEc: hay:hetcha:ramsey1926

    1926

  68. [76]

    Saerens, P

    M. Saerens, P. Latinne, and C. Decaestecker. Adjusting the outputs of a classifier to new a priori probabilities: a simple procedure.Neural Comput, 14(1):21–41, Jan 2002. ISSN 0899-7667 (Print); 0899-7667 (Linking). doi: 10.1162/089976602753284446

    2002 doi

  69. [77]

    Sagawa*, P

    S. Sagawa*, P. W. Koh*, T. B. Hashimoto, and P. Liang. Distributionally robust neural networks. In International Conference on Learning Representations, 2020. URL https://openreview. net/forum?id=ryxGuJrFvS

    2020

  70. [78]

    L. J. Savage. Elicitation of personal probabilities and expectations.Journal of the American Statistical Association, 66(336):783–801, 1971. ISSN 01621459, 1537274X. URLhttp://www. jstor.org/stable/2284229

    1971

  71. [79]

    M. J. Schervish. A general method for comparing probability assessors.The Annals of Statistics, 17(4):1856–1879, 1989. ISSN 00905364, 21688966. URL http://www.jstor.org/stable/ 2241668

    1989

  72. [80]

    Y. Shen. Loss functions for binary classification and class probability estima- tion. PhD thesis, 2005. URL https://www.proquest.com/dissertations-theses/ loss-functions-binary-classification-class/docview/305411117/se-2 . Copyright - Database copyright ProQuest LLC; ProQuest ...

    2005

  73. [81]

    E. H. Shuford, A. Albert, and H. Edward Massengill. Admissible probability measurement procedures. Psychometrika, 31(2):125–145, 1966. doi: 10.1007/BF02289503. URL https: //doi.org/10.1007/BF02289503. 19

    1966 doi

  74. [82]

    K. A. Spackman. Signal detection theory: valuable tools for evaluating inductive learning. In Proceedings of the Sixth International Workshop on Machine Learning, pages 160–163, San Francisco, CA, USA, 1989. Morgan Kaufmann Publishers Inc. ISBN 1558600361

    1989

  75. [83]

    E. W. Steyerberg and A. J. Vickers. Decision curve analysis: a discussion. Med Decis Making, 28(1):146–149, 2008. ISSN 0272-989X (Print); 0272-989X (Linking). doi: 10.1177/ 0272989X07312725

    2008

  76. [84]

    Subbaswamy, P

    A. Subbaswamy, P. Schulam, and S. Saria. Preventing failures due to dataset shift: Learn- ing predictive models that transport. 2018. URLhttp://arxiv.org/abs/1812.04597. cite arxiv:1812.04597Comment: In Proceedings of the 22nd International Conference on Artificial Intelligenc...

    2018 arXiv

  77. [85]

    Sugiyama, N

    M. Sugiyama, N. Rubens, and K.-R. Müller. A conditional expectation approach to model selection and active learning under covariate shift. InDataset Shift in Machine Learning. The MIT Press, 12 2008. ISBN 9780262255103. doi: 10.7551/mitpress/7921.003.0012. URL https://doi.org/...

    2008 doi

  78. [86]

    Swets and T

    J. Swets and T. Birdsall. The human use of information–iii: Decision-making in signal detection and recognition situations involving multiple alternatives.IRE Transactions on Information Theory, 2(3):138–165, 1956. doi: 10.1109/TIT.1956.1056799

    1956

  79. [87]

    W. P. Tanner, J. A. Swets, and H. W. Welch. A new theory of visual detection. Technical Report UMR3825, University of Michigan, 1953. URLhttps://hdl.handle.net/2027.42/

    1953

  80. [88]

    J. M. C. Thompson and G. W. Brier. The economic utility of weather forecasts.Monthly Weather Review, 83:249–253, 1955. URLhttps://api.semanticscholar.org/CorpusID:122117332

    1955

  81. [89]

    D. G. Turakhia. Thirteen ways of looking: a theoretical inquiry in computational creative thinking. Master’s thesis, Massachusetts Institute of Technology, Cambridge, MA, 2017. URL http://hdl.handle.net/1721.1/113918. S.M. Thesis, Department of Architecture and De- partment of...

    2017

  82. [90]

    Vaicenavicius, D

    J. Vaicenavicius, D. Widmann, C. Andersson, F. Lindsten, J. Roll, and T. Schön. Evaluating model calibration in classification. In K. Chaudhuri and M. Sugiyama, editors,Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume ...

    2019

  83. [91]

    A. J. Vickers and E. B. Elkin. Decision curve analysis: A novel method for evaluating prediction models. Medical Decision Making, 26(6):565–574, 2006. doi: 10.1177/0272989X06295361. URL https://doi.org/10.1177/0272989X06295361. PMID: 17099194

    2006 doi

  84. [92]

    A. J. Vickers and F. Holland. Decision curve analysis to evaluate the clinical benefit of prediction models. The Spine Journal, 21(10):1643–1648, 2021. ISSN 1529-9430. doi: https://doi.org/ 10.1016/j.spinee.2021.02.024. URL https://www.sciencedirect.com/science/article/pii/ S1...

    2021 doi

  85. [93]

    A. J. Vickers and S. Woo. Decision curve analysis in the evaluation of radiology research. European Radiology, 32(9):5787–5789, 2022. doi: 10.1007/s00330-022-08685-8. URL https: //doi.org/10.1007/s00330-022-08685-8. 20

    2022 doi

  86. [94]

    A. J. Vickers, B. van Calster, and E. W. Steyerberg. A simple, step-by-step guide to interpreting decision curve analysis. Diagnostic and Prognostic Research, 3(1):18, 2019. doi: 10.1186/ s41512-019-0064-7. URL https://doi.org/10.1186/s41512-019-0064-7

    2019 doi

  87. [95]

    Widmann, F

    D. Widmann, F. Lindsten, and D. Zachariah. Calibration tests in multi-class classification: A unifying framework. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors,Advances in Neural Information Processing Systems, volume 32. Curran ...

    2019

  88. [96]

    X.-H. Zhou, N. Obuchowski, and D. McClish.Statistical Methods in Diagnostic Medicine, Second Edition. 01 2002. ISBN 9780470183144. doi: 10.1002/9780470906514

    2002 doi

  89. [97]

    K. Zhu, Y. Zheng, and K. C. G. Chan. Weighted brier score – an overall summary measure for risk prediction models with clinical utility consideration, 2024. URLhttps://arxiv.org/abs/ 2408.01626. A Calibration The weather forecasting literature focuses on what are known as stri...

    2024 arXiv

  90. [104]

    Carrington et al.[11] reports this as the most common interpretation, a bit tongue in cheek

    0.5 when the classifier is random, and 1.0 when the classifier is perfect. Carrington et al.[11] reports this as the most common interpretation, a bit tongue in cheek. It is, unfortunately, also the current authors’ experience that this is the most commonly given interpretatio...

  91. [105]

    This only makes sense in the original psychometric setting where an experimenter in fact guarantees that there is one positive and one negative case [11]

    The 2 alternative forced choice accuracy rate [86]. This only makes sense in the original psychometric setting where an experimenter in fact guarantees that there is one positive and one negative case [11]

  92. [106]

    This is actually the same as the statement above, but it sounds more impressive

    A rescaled version of the Mann-WhitneyU statistic [6, 35]. This is actually the same as the statement above, but it sounds more impressive. Note that AUC-ROC is never reported as a p-value based on this statistic, which suggests that the interpretation is not practically very useful

  93. [107]

    It is technically true that the AUC-ROC is a pairwise permutation distance between the ideal ranking and the actual ranking

    A rescaled version of the Kendall’sτ correlation coefficient [42]. It is technically true that the AUC-ROC is a pairwise permutation distance between the ideal ranking and the actual ranking. But there are only 2 ranks! This makes the exercise meaningless

  94. [108]

    Average Precision

    An average of precision (though not "Average Precision" which refers to something else) There is an occasional attempt to rescue the paradigm by arguing that AUC-ROC shows an average of T P = K × Precision@K over a range ofK. The trouble is twofold: • This gives a uniform aver...

  95. [109]

    The trouble is that in the Neyman-Pearson paradigm we’re meant to pick a power, and then find out what the size of the test is

    An average of power over a range of sizes (in the Neyman-Pearson sense) This one is very popular with practitioners and virtually absent from the literature, aside from [54], which was later criticized by [55, 11]. The trouble is that in the Neyman-Pearson paradigm we’re meant...

  96. [110]

    It is the area under a curve if FPR is plotted against TPR. [35]

  97. [111]

    False Positive Rate

    Average accuracy on the positive class across a uniform distribution of accuracy on the negative class, or vice versa [59, 60, 96]. This is actually the same as the statement above, but more useful-sounding, and slightly less mysterious since it doesn’t use the words "False Po...

  98. [112]

    This is a bit more useful, but it’s not clear why we’re using the exact class balance between the thresholds

    Given two thresholds a < b, the average accuracy on the positive class across a uniform distribution of accuracy on the negative class between those two thresholds, plus the average accuracy on the negative class across a uniform distribution of accuracy on the positive class ...

  99. [113]

    This interpretation is the only one from this set of authors that directly addresses the problem that data is empirical and discrete

    An average of accuracy as we set the threshold at each data point, leaving operating conditions the same [25]. This interpretation is the only one from this set of authors that directly addresses the problem that data is empirical and discrete

  100. [114]

    Hand[33] shows that if a score is calibrated, then the AUC-ROC is an average of the cost-weighted error over a range of prevalences

    An average of cost-weighted error over a range of cost ratios. Hand[33] shows that if a score is calibrated, then the AUC-ROC is an average of the cost-weighted error over a range of prevalences. The trouble is that: • Calibration is a really important property of a score! Wit...

  101. [115]

    An average of skew-weighted cost, for skewz = c ⊗ (1 − π). [42] proposes this interpretation, although again the costs are set arbitrarily and distinctly by each model, and class-conditional distributions are assumed to be continuous and fully known. Furthermore, it is not cle...

  102. [116]

    E (x,y)∈D1/2:s(x)=t 1 − y # y′ h 1(s(x′)≥t) i = E t∈s[D1/2]

    An average of accuracy under label shift, where the distribution of positive class prevalences is derived by sampling from the model scores on the training data. This is syntactically similar, but specifically derived in the case of the sampling problems that arise from label ...

  103. [1966]

    URL https://journals

    doi: 10.1175/1520-0450(1966)005<0534:ANOTUO>2.0.CO;2. URL https://journals. ametsoc.org/view/journals/apme/5/4/1520-0450_1966_005_0534_anotuo_2_0_co_2.xml

    1966 doi

  104. [1974]

    URLhttp://www.jstor.org/stable/1913937

    ISSN 00129682, 14680262. URLhttp://www.jstor.org/stable/1913937

  105. [1979]

    URLhttp://www.jstor.org/stable/1912352

    ISSN 00129682, 14680262. URLhttp://www.jstor.org/stable/1912352

  106. [2007]

    URL https://doi.org/10.1007/s10994-007-5011-0

    doi: 10.1007/s10994-007-5011-0. URL https://doi.org/10.1007/s10994-007-5011-0

    doi

  107. [2011]

    ISBN 9781450306195

    Omnipress. ISBN 9781450306195

  108. [7893]

    Engineering Technical Report

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.