Pith. sign in

REVIEW 2 major objections 4 minor 22 references

Machine-generated labels overstate cervical-spine segmentation performance by ~8 Dice points and manufacture statistical significance by collapsing score variance.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-10 16:40 UTC pith:IA3YRZA5

load-bearing objection Clean isolation of a pure ruler effect: same predictions, gold vs reconstructed silver, ~8 Dice inflation and an age-verdict flip via ~4× variance collapse (false confidence), not gap inflation. the 2 major comments →

arxiv 2607.07852 v1 pith:IA3YRZA5 submitted 2026-07-08 eess.IV cs.CVcs.CYcs.LG

False Confidence: Automated Labels Confound Fairness Audits in Cervical Spine Segmentation

Linus Juni , Aasa Feragen , Aditya Parikh This is my paper
classification eess.IV cs.CVcs.CYcs.LG
keywords FairnessSegmentationSpine MRIBiasLabel NoiseSilver labelsFalse confidenceCervical spine
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern medical segmentation datasets mix scarce expert (gold) labels with abundant machine-generated (silver) labels to control annotation cost. This paper shows that the silver labels are not neutral rulers: because they are produced by a model trained on the same gold labels, any new model trained on those gold labels agrees with the silver labels far more than with expert truth. Scoring the identical predictions against silver rather than gold therefore inflates Dice by roughly 8 points and, more critically, shrinks within-group variance by a factor of four. That variance collapse turns a non-significant age difference into a statistically significant one without enlarging the actual gap—an effect the authors call false confidence. The practical message is that both performance numbers and fairness verdicts must be reported against expert labels, and every fairness claim must name the provenance of its reference.

Core claim

On the CSpineSeg cervical-spine MRI dataset the deployed model is demographically fair across sex, race and age when judged by expert labels. The same predictions scored against a silver ruler generated from the gold labels, however, overestimate macro Dice by ~8 points (0.897 → 0.973) and flip the age fairness verdict from non-significant to significant solely by collapsing within-group standard deviation ~4× (0.058 → 0.014). This is the opposite of gap inflation: the correlated ruler manufactures false confidence rather than false magnitude.

What carries the argument

The correlated biased-ruler effect (false confidence): a silver reference that is itself a near-clone of the model under test, produced by training on the same gold labels, so that inter-model agreement far exceeds agreement with expert truth and within-group score variance collapses.

Load-bearing premise

The reconstructed silver ruler built from a gold-only model’s predictions on the held-out gold cases faithfully reproduces the statistical properties of the dataset’s original unpublished silver-label generator.

What would settle it

Re-run the exact E2 comparison using the dataset’s original unpublished silver generator (if released) on the same 76 gold-test images; if the ~4× variance collapse and the significance flip disappear, the false-confidence claim fails for this dataset.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any validation that uses a mixed-provenance dataset’s silver labels will systematically overestimate segmentation performance.
  • Fairness audits that rely on silver references can flip significance verdicts without any change in the underlying patient-level predictions.
  • Datasets must explicitly mark which masks are expert versus machine-generated so users can restrict evaluation to gold labels.
  • Fairness claims should always be accompanied by a statement of reference provenance and by magnitude, not only p-values.
  • A ruler that yields both much higher scores and much tighter variance than experts is too correlated to serve as an independent benchmark.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same leakage mechanism is likely to appear in any mixed gold/silver segmentation dataset whose silver labels were generated by a model trained on its own gold subset, not only cervical-spine MRI.
  • High-performance regimes (Dice > 0.9) are especially vulnerable to false confidence because success-rate metrics saturate while continuous tests gain power from the variance collapse.
  • A simple diagnostic—ratio of within-group variance under silver versus gold—could be added to standard evaluation pipelines to flag correlated rulers before fairness claims are published.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper presents the first demographic fairness audit of cervical-spine MRI segmentation (vertebral bodies and discs) on CSpineSeg, across sex, race, and age. Using nnU-Net ensembles trained under mixed (gold+silver) and gold-only regimes, the authors find the deployed model fair by Dice/nDSC/HD95, DIR/DPD, rank tests, and OLS. The central methodological claim is that silver labels, generated by a model trained on the same gold labels, act as a correlated ruler: scoring identical M_mix predictions on the 76 gold-test cases against M_gold outputs rather than expert labels inflates macro Dice by ~8 points (0.897 o0.973) and flips the age Kruskal–Wallis verdict from non-significant (p_fdr=0.27) to significant (p_fdr=0.026) solely by collapsing within-group SD ~4 imes (0.058 o0.014). They term this “false confidence,” complementary to Parikh et al.’s “false magnitude,” and recommend reporting performance/fairness only against expert labels with explicit provenance.

Significance. If the result holds, it is a practically important methodological warning for the large class of medical segmentation datasets that mix scarce expert labels with abundant machine-generated ones. The E2 design cleanly isolates a pure ruler effect on identical predictions, the variance-collapse mechanism is directly visible in Table 2 and Fig. 1, and the performance-overestimation claim is independent of fairness and therefore relevant to any user who validates on such data. The complementary mode to Parikh et al. is a genuine conceptual contribution, and the practitioner recommendations (report against expert labels, state provenance, check ruler–model similarity, report magnitude) are actionable. Strengths include volume-aware nDSC, FDR-corrected continuous tests, and transparent limitations.

major comments (2)
  1. [§3.2 Experimental design (E2) and §5 Limitations] §3.2 E2 / §5 Limitations: the silver ruler is a reconstruction (M_gold predictions) rather than the original unpublished generator used by Zhou et al. The paper correctly flags this, yet the central claim that the observed variance collapse is a property of CSpineSeg’s labeling process (rather than of the reconstruction) rests on that untested fidelity. A sensitivity check—e.g., training an alternative generator with different architecture or seed and re-scoring the same 76 cases—would strengthen the claim that false confidence generalizes beyond the authors’ own M_gold.
  2. [§4.2 The Biased Ruler and Table 2] §4.2 / Table 2: the age-bin boundaries and the Dice success threshold τ=0.8 are free parameters that affect both the continuous Kruskal–Wallis and the rate-based DIR/DPD. While a τ-sweep is mentioned, the manuscript does not show that the significance flip survives alternative clinically motivated age cuts (e.g., <50/50–70/≥70) or continuous age regression. Given that the magnitude of the age gradient is already described as clinically negligible, confirming robustness of the flip itself would make the false-confidence demonstration more load-bearing.
minor comments (4)
  1. [Abstract / §1] Abstract and §1: the phrase “turns the fairness verdict for age from non-significant to significant” should be immediately qualified by the magnitude (≤2.7 Dice points, all groups >0.89) so readers do not over-read clinical unfairness.
  2. [Fig. 1] Fig. 1 caption: the per-group SD ranges (0.03–0.10 gold vs 0.01–0.02 silver) are helpful; adding the exact Kruskal–Wallis H and p_fdr values already given in the text would make the figure self-contained.
  3. [§2.1 The CSpineSeg Cohort] §2.1: clarify whether the single post-doctoral drafter of the gold labels introduces any systematic style that both M_mix and M_gold could have absorbed, beyond the radiologist review step.
  4. [Bibliography] Bibliography: several arXiv preprints are cited with 2025/2026 dates; ensure final DOIs or stable versions are supplied at camera-ready.

Circularity Check

1 steps flagged

Minor non-load-bearing self-citation of the authors' prior biased-ruler framework; the empirical false-confidence result (identical predictions, gold vs reconstructed silver) is independently measured and does not reduce by construction.

specific steps
  1. self citation load bearing [§1 Introduction, Our Contributions (i–iii) and Abstract]
    "Building on the biased-ruler framework of Parikh et al. [15, 14], we contribute: ... iii A complementary mode of the biased-ruler effect. From that same leakage, our correlated ruler flips age from non-significant to significant by collapsing within-group variance (false confidence) – the opposite mechanism to Parikh et al.’s false magnitude[15]."

    The interpretive premise that a silver reference can act as a 'biased ruler' (and the contrast term 'false magnitude') is imported from prior work by overlapping authors rather than re-derived. The citation is not load-bearing for the numerical result itself—the variance collapse and Dice inflation are measured directly on identical M_mix predictions—but it supplies the sole conceptual warrant for framing the experiment as a 'complementary mode' of that earlier effect.

full rationale

The paper's central claims are experimental observations on CSpineSeg: M_mix predictions scored against expert gold labels versus a reconstructed silver ruler (M_gold outputs on the same 76 images) produce ~8 Dice inflation and a Kruskal-Wallis age-verdict flip driven by ~4 imes within-group SD collapse. These numbers are obtained by direct scoring of identical outputs under two references; no equation, fit, or definition forces the variance collapse or the performance gap. The only self-reference is the conceptual scaffolding of the 'biased ruler' (Parikh et al. [15,14], co-authored by Parikh and Feragen). That citation supplies terminology and the contrast with 'false magnitude,' but the new 'false confidence' mode is demonstrated on fresh data and does not inherit its numerical content from the prior papers. The intentional correlation of the silver ruler is the object of study, not a hidden circular support. No self-definitional loop, fitted-parameter-as-prediction, uniqueness import, or ansatz smuggling appears. Score 2 therefore reflects only the minor, non-load-bearing self-citation of framework; the derivation chain itself is self-contained against the reported experiments.

Axiom & Free-Parameter Ledger

2 free parameters · 3 axioms · 1 invented entities

The central claim rests on standard statistical tests, the public CSpineSeg provenance split, and the modeling assumption that a gold-only nnU-Net adequately reconstructs the original silver-label generator. No free parameters are fitted to produce the fairness flip; the only free choices are conventional thresholds (Dice τ=0.8, HD95=5 mm) and the three age bins.

free parameters (2)
  • Dice success threshold τ = 0.8
    Fixed at 0.8 for rate-based DIR/DPD; conventional but arbitrary; continuous tests are primary.
  • Age bin boundaries = <40 / 40–60 / ≥60
    Three bins (<40, 40–60, ≥60) chosen by the authors; Kruskal–Wallis result depends on this discretization.
axioms (3)
  • standard math Mann–Whitney / Kruskal–Wallis rank tests with Benjamini–Hochberg FDR control correctly assess demographic fairness on continuous Dice/nDSC/HD95.
    Invoked throughout §3.2 and §4 for all significance claims.
  • domain assumption A model trained only on the gold subset (M_gold) produces silver labels whose statistical relationship to a mixed-trained model mirrors the original unpublished CSpineSeg silver generator.
    Stated in §3.2 Experimental design (E2) and acknowledged as a reconstruction limitation in §5.
  • domain assumption Structure-size differences across sex/age/race are adequately controlled by reporting vertebral body and disc separately plus volume-normalized nDSC.
    §2.2 Anatomical Confounders; used to argue that residual gaps are not pure size bias.
invented entities (1)
  • false confidence (biased-ruler mode) independent evidence
    purpose: Name the complementary failure mode in which a correlated silver ruler collapses within-group variance and manufactures statistical significance without inflating the gap.
    Introduced in abstract and §1; operationalized by the E2 variance-collapse numbers. Independent evidence is the empirical SD drop and p_fdr flip on the same predictions.

pith-pipeline@v1.1.0-grok45 · 13580 in / 2707 out tokens · 25510 ms · 2026-07-10T16:40:25.142850+00:00 · methodology

0 comments
read the original abstract

Automated segmentation of cervical-spine MRI is increasingly used in clinical workflows, yet no fairness audit exists for this anatomy. We show that auditing these segmentation tasks is complicated by a common property of modern segmentation datasets: expert-annotated gold labels are expensive, so abundant machine-generated (silver) labels are added to limit annotation cost. This matters because the reference used to judge a model can itself be biased. In this study, we present the first fairness audit of cervical-spine MRI segmentation across sex, age, and race using the CSpineSeg dataset. We observe that the deployed model is demographically fair, but the choice of reference label, however, is not neutral. Because a dataset's silver labels are generated by a model trained on its gold labels, any new model trained on those same gold labels agrees more with the silver labels than with expert truth: scoring identical predictions against silver rather than gold overestimates performance by ~8 Dice points and turns the fairness verdict for age from non-significant to significant - not by the gap inflation Parikh et al. report (which we term false magnitude) but by collapsing within-group variance (which we term false confidence). Reference-label provenance is thus a first-order confounder in segmentation evaluation: performance and fairness should be reported against expert labels, and any fairness claim stated together with the provenance of its reference.

Figures

Figures reproduced from arXiv: 2607.07852 by Aasa Feragen, Aditya Parikh, Linus Juni.

Figure 1
Figure 1. Figure 1: The variance collapse behind the flip. Macro Dice by age bin for the same 76 predictions, scored against the gold ruler (left) and silver ruler (right). The gradient runs the same direction on both (60+ worst), but gold’s per-group SDs (0.03–0.10) bury it below FDR correction while silver’s collapsed variance (0.01–0.02) exposes it. The ruler changed the test’s power, not the patient. and much tighter agai… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · 2 internal anchors

  1. [1]

    Journal of the Royal Statistical Society: Series B (Methodological)57(1), 289–300 (1995)

    Benjamini, Y.,Hochberg, Y.:Controllingthe false discovery rate:a practical and powerful approach to multiple testing. Journal of the Royal Statistical Society: Series B (Methodological)57(1), 289–300 (1995). https://doi.org/ 10.1111/j.2517-6161.1995.tb02031.x

    work page doi:10.1111/j.2517-6161.1995.tb02031.x 1995

  2. [2]

    Bird, S., Dudík, M., Edgar, R., Horn, B., Lutz, R., Milan, V., Sameki, M., Wallach, H., Walker, K.: Fairlearn: A toolkit for assessing and improving fairness in AI. Tech. Rep. MSR-TR-2020-32, Microsoft (2020)

    work page 2020

  3. [3]

    Machine Learning for Biomedical Imaging (2026), arXiv preprint arXiv:2510.17999

    Danaee, G., Niethammer, M., Rushmore, J., Bouix, S.: Investigating de- mographic bias in brain MRI segmentation: A comparative study of deep- learning and non-deep-learning methods. Machine Learning for Biomedical Imaging (2026), arXiv preprint arXiv:2510.17999

    work page arXiv 2026

  4. [4]

    https://ens.dk/en/ analyses-and-statistics/key-figures (2023), observed (physical) 99 g/kWh; adjusted (trade-corrected) 138 g/kWh

    Danish Energy Agency (Energistyrelsen): Key figures for Den- mark 2023: CO2 emissions per kwh electricity. https://ens.dk/en/ analyses-and-statistics/key-figures (2023), observed (physical) 99 g/kWh; adjusted (trade-corrected) 138 g/kWh. Accessed 2026-06-11

    work page 2023

  5. [5]

    four-fifths rule

    Equal Employment Opportunity Commission, et al.: Uniform guidelines on employeeselectionprocedures(1978).29C.F.R.§1607.4(D)(the“four-fifths rule”);adoptedjointlybytheEEOC,CivilServiceCommission,Department of Labor, and Department of Justice (1978), accessed 2026-06-12

    work page 1978

  6. [6]

    Official Journal of the European Union, L series, 12 July 2024 (2024)

    European Parliament and Council of the European Union: Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (AI Act). Official Journal of the European Union, L series, 12 July 2024 (2024)

    work page 2024

  7. [7]

    Nature Methods18(2), 203–211 (2021)

    Isensee, F., Jaeger, P.F., Kohl, S.A.A., Petersen, J., Maier-Hein, K.H.: nnU- Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods18(2), 203–211 (2021). https://doi.org/10. 1038/s41592-020-01008-z

    work page 2021

  8. [8]

    In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024

    Isensee, F., Wald, T., Ulrich, C., Baumgartner, M., Roy, S., Maier-Hein, K.H., Jäger, P.F.: nnU-Net revisited: A call for rigorous validation in 3D medical image segmentation. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2024. Lecture Notes in Computer Science, vol. 15009, pp. 488–498. Springer Nature Switzerland, Cham (2024). h...

    work page doi:10.1007/978-3-031-72114-4_47 2024

  9. [9]

    In: Ad- vances in Neural Information Processing Systems (NeurIPS) – Datasets and Benchmarks Track (2024)

    Jin, R., Xu,Z., Zhong, Y.,Yao, Q., Dou,Q., Zhou, S.K., Li,X.: FairMedFM: Fairness benchmarking for medical imaging foundation models. In: Ad- vances in Neural Information Processing Systems (NeurIPS) – Datasets and Benchmarks Track (2024)

    work page 2024

  10. [10]

    Part II: magnetic resonance imaging of over 1,200 asymptomatic subjects

    Kato, F., Yukawa, Y., Suda, K., Yamagata, M., Ueta, T.: Normal mor- phology, age-related changes and abnormal findings of the cervical spine. Part II: magnetic resonance imaging of over 1,200 asymptomatic subjects. European Spine Journal21(8), 1499–1507 (2012) 10 L. Juni et al

    work page 2012

  11. [11]

    Journal of the American Statistical Association47(260), 583–621 (1952)

    Kruskal, W.H., Wallis, W.A.: Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association47(260), 583–621 (1952). https://doi.org/10.1080/01621459.1952.10483441

    work page doi:10.1080/01621459.1952.10483441 1952

  12. [12]

    Quantifying the Carbon Emissions of Machine Learning

    Lacoste, A., Luccioni, A., Schmidt, V., Dandres, T.: Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700 (2019)

    work page internal anchor Pith review Pith/arXiv arXiv 1910

  13. [13]

    The Annals of Mathematical Statistics18(1), 50–60 (1947)

    Mann, H.B., Whitney, D.R.: On a test of whether one of two random vari- ables is stochastically larger than the other. The Annals of Mathematical Statistics18(1), 50–60 (1947). https://doi.org/10.1214/aoms/1177730491

    work page doi:10.1214/aoms/1177730491 1947

  14. [14]

    Parikh,A.,Das,S.,Feragen,A.:Whodoesyouralgorithmfail?Investigating ageandethnicbiasintheMAMA-MIAdataset.In:NeurIPS2025Workshop on Medical Imaging (MedEurIPS) (2025)

    work page 2025

  15. [15]

    In: 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI)

    Parikh, A., Das, S., Feragen, A.: Investigating label bias and representa- tional sources of age-related disparities in medical segmentation. In: 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI). pp. 1–5 (2026). https://doi.org/10.1109/ISBI61048.2026.11515928

    work page doi:10.1109/isbi61048.2026.11515928 2026

  16. [16]

    Parikh, A., Frank, S., Das, S., Feragen, A.: Towards fairness under label bias in image segmentation: Impact, measurement and mitigation (2026), https://arxiv.org/abs/2605.06891

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  17. [17]

    In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2021

    Puyol-Antón, E., Ruijsink, B., Piechnik, S.K., Neubauer, S., Petersen, S.E., Razavi, R., King, A.P.: Fairness in cardiac MR image analysis: An inves- tigation of bias due to data imbalance in deep learning based segmenta- tion. In: Medical Image Computing and Computer Assisted Intervention – MICCAI 2021. Lecture Notes in Computer Science, Springer (2021)....

    work page doi:10.1007/978-3-030-87199-4_39 2021

  18. [18]

    In: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI)

    Raina, V., Molchanova, N., Graziani, M., Malinin, A., Müller, H., Bach Cuadra, M., Gales, M.: Tackling bias in the Dice similarity coeffi- cient: Introducing nDSC for white matter lesion segmentation. In: 2023 IEEE 20th International Symposium on Biomedical Imaging (ISBI). IEEE (2023). https://doi.org/10.1109/ISBI53787.2023.10230755

    work page doi:10.1109/isbi53787.2023.10230755 2023

  19. [19]

    BMC medical imaging15(1), 29 (2015)

    Taha, A.A., Hanbury, A.: Metrics for evaluating 3d medical image segmen- tation: analysis, selection, and tool. BMC medical imaging15(1), 29 (2015)

    work page 2015

  20. [20]

    Medical Image Analysis63, 101693 (2020)

    Tajbakhsh, N., Jeyaseelan, L., Li, Q., Chiang, J.N., Wu, Z., Ding, X.: Embracing imperfect datasets: A review of deep learning solutions for medical image segmentation. Medical Image Analysis63, 101693 (2020). https://doi.org/10.1016/j.media.2020.101693

    work page doi:10.1016/j.media.2020.101693 2020

  21. [21]

    In: International Conference on Learning Representations (ICLR) (2024)

    Tian, Y., Shi, M., Luo, Y., Kouhana, A., Elze, T., Wang, M.: FairSeg: A large-scale medical image segmentation dataset for fairness learning using segment anything model with fair error-bound scaling. In: International Conference on Learning Representations (ICLR) (2024)

    work page 2024

  22. [22]

    Scientific Data12(1), 1695 (2025)

    Zhou, L., Wiggins, W., Zhang, J., Colglazier, R., Willhite, J., Dixon, A., Malinzak, M., Gu, H., Mazurowski, M.A., Calabrese, E.: The Duke Uni- versity Cervical Spine MRI Segmentation Dataset (CSpineSeg). Scientific Data12(1), 1695 (2025). https://doi.org/10.1038/s41597-025-05975-w

    work page doi:10.1038/s41597-025-05975-w 2025