{"id":"43597423-1aad-4ab3-8a42-9ca4cd4d42b4","arxiv_id":"2507.14176","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new fairness audit metric, Predictive Representativity, reveals that five common skin-cancer classifiers trained on HAM10000 have much lower precision for darker Fitzpatrick skin types when tested on an independent Colombian dataset.","lead":"This paper proposes Predictive Representativity, a fairness metric that compares how well an AI model performs on a skin-tone subgroup versus the population as a whole. Testing five image classifiers trained on HAM10000 against a new Colombian dataset, it finds large precision drops for darker skin types, suggesting the widely used training data encodes racial bias.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Formal PR in Eq. (3) is never instantiated in the case study: reported values are metric-level PRMi with the opposite sign convention, and the point-mass estimator is undefined for KL or trivial for total variation.","rationale":"The reader's weakest assumption is on target: the estimator in Eq. (3) is underspecified. I would go further and flag that Eq. (3) is not used in the empirical section; the reported PR values come from metric-level PR, and the sign convention flips between Definition 1 and Table 2. This means the central formal contribution is currently unsupported by the case study. That is a genuine correctness risk for the paper's novelty, not just a missing detail. However, the empirical bias finding (large precision and F1 gaps, external BOSQUE benchmark, p-values) is credible and clinically important, so the correct response is to require the formalization and sign convention be fixed before the PR claim is accepted. The reader's CONDITIONAL verdict remains appropriate; no change is needed.","tokens_in":19680,"tokens_out":5283,"duration_ms":62678,"concrete_test":"Specify a divergence D (e.g., KL) and the model's softmax predictive distribution \\hat P(Y|X=x), then recompute cPR from Eq. (3) on the BOSQUE Test set, with bootstrap confidence intervals. Check whether the resulting cPR values reproduce the signs and subgroup ordering of the PRMi values in Table 2. If KL yields infinite values, or total variation reduces cPR to a simple error-rate gap, or the ordering changes, the formal PR definition is not what the case study measures.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central methodological claim is that Predictive Representativity, defined in Eq. (1) via the divergence D(P(Y|X) || \\hat P(Y|X)), is a new outcome-level fairness measure. The empirical estimator in Eq. (3) replaces both distributions with point masses, δ_{y_i} and δ_{f_A(x_i)}, and then averages D between them. For KL divergence, D(δ_y || δ_{\\hat y}) is 0 when y = \\hat y and +∞ when y ≠ \\hat y; for total variation it is the 0/1 mismatch indicator (up to scaling). The paper never states which D is used, nor proves consistency of the estimator, and the point-mass substitution is not justified from Eq. (1), where \\hat P(Y|X) should be the model's predictive distribution. More importantly, the case study does not actually report cPR from Eq. (3). Table 2 reports metric-level PRMi(S) = M_i(S) - M_i(P) for precision, sensitivity, AUC-PR, specificity, accuracy, AUC-ROC, and F1. These are performance-metric differences, not divergence estimates, and their sign convention is inverted relative to Table 1: a positive formal PR means underperformance, while a positive PRMi for light skin in Table 2 means better-than-average precision. Thus the formal quantity the paper introduces is disconnected from the empirical evidence that supports the headline. If the divergence-based PR is the contribution, it has not been operationalized; if only PRMi is used, the paper is reporting standard subgroup performance gaps. The raw disparities themselves remain credible, but the paper's new formal measure does not yet bear the weight placed on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Predictive Representativity (PR), defined in Eq. (1) as the difference between a subpopulation's average divergence from true labels to model predictions and the full population's average divergence, together with an External Transportability Criterion based on a tolerance epsilon. The empirical estimator in Eq. (3) replaces true and predicted labels by point masses. The authors then introduce metric-level PR, PRMi(S)=M_i(S)-M_i(P), and apply it to five CNN classifiers trained on HAM10000 and evaluated on the BOSQUE Test set (n=165) from Colombia, grouped by Fitzpatrick phototype. They report large, statistically significant precision gaps for darker skin (e.g., ResNet50 precision 0.897 light vs 0.484 dark, p<0.001) and conclude that proportional dataset composition does not ensure equitable predictive performance.","tokens_in":20076,"tokens_out":14775,"duration_ms":160682,"significance":"The raw empirical finding is meaningful and clinically important: an independently collected, Fitzpatrick-annotated dataset is used to show a consistent dark-skin precision deficit across five architectures, with small p-values and publicly available code and data. If the formal PR framework were valid, it would offer a practical outcome-level auditing tool. However, as presented, the formal divergence-based PR is not operationalized: Eq. (3) is undefined or trivial for standard divergences, the case study actually reports metric-level differences, and the interpretation of signs is inconsistent with the formal definition. The paper's contribution at present is therefore a conventional subgroup disparity audit plus a proposed but unvalidated formal metric. The empirical evidence supports a call for external validation and dataset documentation, but does not yet establish PR as a new fairness measure.","major_comments":[{"comment":"The formal definition of PR requires S⊆X, a measurable subset of the input space, and defines P_S as the law of (X,Y) given X∈S. In the case study, subgroups are defined by Fitzpatrick phototype, a demographic label that is not part of the dermoscopic image input X; the model does not receive this attribute. The framework therefore does not formally cover the case study's subgroups. The authors should either define a sensitive attribute Z and condition on Z, or justify treating skin phototype as a function of X.","section":"Modelling set-up, Eq. (1)"},{"comment":"Equation (3) estimates cPR by averaging D(δ_{y_i}∥δ_{f_A(x_i)}). For KL divergence this term is 0 when predictions match and +∞ when they differ; for total variation it is a 0/1 mismatch indicator. The paper does not state which divergence D is used, does not prove consistency of this point-mass estimator for the conditional-divergence quantity in Eq. (1), and does not explain why the model's predictive distribution \\hat P(Y|X) should be replaced by a point mass at the hard label. As written, cPR is an error-rate gap, not an estimate of the proposed PR.","section":"Empirical Estimation, Eq. (3)"},{"comment":"Table 1 states that PR>0 indicates underperformance and PR<0 indicates overfitting or disproportionate optimization for S. But PRMi(S)=M_i(S)-M_i(P), so a positive PRMi means the subgroup outperforms the population. Table 2 reports PRMi values that are systematically positive for light skin and negative for dark skin, and the text interprets the dark-skin negative values as reduced ability and underperformance (e.g., the F1-Score paragraph). This is the opposite of the formal sign convention. The paper must either redefine PRMi with the opposite sign or correct Table 1; the current inconsistency drives the interpretation of every result.","section":"Table 1 and Case Study, Table 2"},{"comment":"The metric-level quantity PRMi(S)=M_i(S)-M_i(P) is a standard subgroup performance gap. The paper does not show that PRMi estimates, bounds, or approximates the divergence-based PR of Eq. (1), nor does it connect the thresholds epsilon_i to the divergence scale. Thus the empirical case study does not instantiate the formal PR; it reports a conventional disparity audit under a new name. If the contribution is metric-level PR, the paper should position it as such and relate it to existing fairness metrics.","section":"Operationalising Predictive Representativity"},{"comment":"The criterion |PR(P',S',A)|≤epsilon is never instantiated: no epsilon value is specified, no source-population PR is computed (HAM10000 lacks skin-tone labels), and the paper does not evaluate |PR| on BOSQUE. The Discussion's claim that models failed to meet a standard PR tolerance is therefore unsupported. Without a specified tolerance and an operational estimator, the criterion is not falsifiable.","section":"External Transportability Criterion and Discussion"},{"comment":"The paper states that lower precision in darker skin means models are more likely to misclassify cancerous lesions as benign in this group. This is incorrect: precision is TP/(TP+FP), so low precision means more false positives among predicted malignant cases, i.e., benign lesions flagged as malignant. The delay-of-diagnosis harm corresponds to low sensitivity, which the paper finds less affected. The clinical narrative built on the precision result should be corrected.","section":"Introduction and Case Study precision paragraph"}],"minor_comments":[{"comment":"The paper reports AUC-PR and AUC-ROC but does not state whether these are computed from softmax scores or from hard predictions; specify the scoring protocol.","section":"Case Study, Table 2"},{"comment":"The Z-statistic's null hypothesis is not stated; clarify whether p-values test PRMi=0 or PRMi(Light)=PRMi(Dark), and whether multiple-comparison corrections were applied across 5 models × 7 metrics.","section":"Case Study, Table 2"},{"comment":"The BOSQUE sample size is stated as 167 in the Abstract and 165 in the Case Study and Table 2; reconcile the discrepancy.","section":"Abstract and Case Study"},{"comment":"The Fitzpatrick grouping is inconsistent: the Introduction refers to lighter types II–III and darker IV–VI, while the Case Study groups I–III as lighter; state which types are actually present in the data.","section":"Introduction and Case Study"},{"comment":"The claim that all five models exhibited reduced precision and recall rates is contradicted by Table 2 for MobileNetV2 sensitivity (0.899 light vs 0.947 dark); revise to refer to precision only or qualify the recall claim.","section":"Introduction"},{"comment":"The subscript I in S_I is undefined.","section":"Empirical Estimation, Eq. (3)"},{"comment":"The text alternates between representativity and representativeness and contains typos (e.g., benckmark, indentifying meaningfull examples, emploing effective estimators); copyediting is needed.","section":"General"},{"comment":"The claim that the finding holds despite proportional sampling in the source data is confusing because the paper also describes HAM10000's darker-skin share as around 5% and disproportionately light; clarify whether proportional refers to source-population prevalence or to the deployment population.","section":"Abstract and Introduction"}],"recommendation":"major_revision","confidential_remarks":"The raw empirical result is likely publishable after a substantial reframing. The formal PR as presented should either be removed or replaced with a properly defined and instantiated estimator; otherwise the paper's central methodological claim will not survive review. The BOSQUE dataset and code availability are assets, and the precision finding itself is credible and clinically relevant. I recommend major revision rather than rejection because the empirical contribution is salvageable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this paper. The empirical case study is worth taking seriously: an independent test set (BOSQUE, 165 images with Fitzpatrick labels) evaluated across five standard CNNs trained on HAM10000, with consistent, statistically strong precision/AUC-PR/F1 gaps for darker skin. The formal PR framework, by contrast, is a repackaging of ordinary subgroup performance gaps, and the paper's own equations don't connect the formal divergence definition to the metric-level numbers in Table 2.\n\nWhat's genuinely new and good: the BOSQUE dataset is public, the code is public, and the evaluation gives concrete evidence that HAM10000-trained models transfer poorly to darker skin in a real clinical population. That is a solid contribution for dermatology AI deployment. The choice to avoid synthetic skin darkening is defensible, and the paper is honest about the dataset's limitations.\n\nSoft spots, in proportion. First, the formal disconnect: Eq. (1) defines PR via an expected divergence between true and predicted conditional distributions; Eq. (3) estimates it with point masses, but for KL that quantity is undefined whenever prediction and label differ, and for total variation it collapses to a 0/1 error-rate gap. The paper never specifies which divergence is used. What Table 2 actually reports is metric-level PR, M_i(S) - M_i(P), which is just a performance gap. So the headline formal quantity is never instantiated. Second, the sign convention is internally inconsistent: Table 1 says positive PR means the model underperforms in S, but Table 2 shows lighter-skin groups with positive PR on precision and darker-skin groups with negative, and the text interprets negative as underperformance. That is a direct contradiction. Third, the External Transportability Criterion is a threshold on that same gap with unspecified epsilon; it adds no formal content. Fourth, the paper claims bootstrap confidence intervals but reports only Z-statistics, not CIs.\n\nThese are addressable problems, not fatal to the empirical finding. The raw numbers stand, and the disparity is clinically meaningful. But as a contribution to fairness metrics, the paper doesn't deliver a new measure; it's standard subgroup performance auditing with a new name. The authors should either repair the formal estimator (use a proper divergence on predictive distributions, reconcile Eq. (1) with Eq. (3), fix the sign) or drop the formal PR claim and present this as an empirical audit.\n\nWho this is for: people working on medical AI fairness, dataset documentation, and deployment validation. It deserves a serious referee, but with major revisions clearly needed. I'd send it to review, not desk reject.","headline":"Useful external-dataset case study showing real precision gaps for darker skin, wrapped in a PR formalism that doesn't hold together; referee it for the empirical result, not the new metric.","tokens_in":20669,"tokens_out":2395,"would_cite":true,"duration_ms":27507,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that AI-based skin cancer classifiers trained on HAM10000 are systematically less accurate for darker-skinned patients, and it proposes Predictive Representativity, an outcome-level audit metric, to expose such gaps.","keywords":["algorithmic fairness","racial bias in AI","Predictive Representativity","transportability","medical AI","outcome equity","HAM10000","Fitzpatrick phototype"],"falsifier":"The central claim would be overturned if a replication on a larger externally collected test set with Fitzpatrick annotations found no significant malignant-lesion precision gap between lighter (types II–III) and darker (types IV–VI) skin for any of the five architectures trained on HAM10000; the formal quantity would be undermined if the empirical cPR estimator, computed with a specified divergence and bootstrap intervals, failed to track the metric-level subgroup gaps on the same data.","tokens_in":19494,"feed_emoji":"🩺","tokens_out":12198,"duration_ms":111751,"temperature":0.7,"pith_summary":"This paper introduces Predictive Representativity, an audit metric that measures whether a model's predictions align with true outcomes equally across subpopulations, and uses it to test five standard skin-cancer classifiers trained on HAM10000 against an independent Colombian test set labeled with Fitzpatrick skin phototypes. The paper claims that despite proportional sampling in the training data, every architecture underperforms on darker skin, with ResNet50's precision for malignant lesions at 0.897 for lighter skin versus 0.484 for darker skin ($p\\leq0.001$). It argues that representativity is a dynamic property of model predictions, not a static feature of dataset composition, and proposes an External Transportability Criterion for deciding when fairness generalizes to a new population. If the paper is right, outcome-level audits are needed to expose subgroup harm that aggregate accuracy and proportional sampling both hide, which matters because darker-skinned melanoma patients already face delayed diagnosis and higher mortality.","feed_headline":"Skin-cancer AI favors lighter skin in every model tested","feed_subtitle":"An outcome-level audit metric, Predictive Representativity, exposes gaps that proportional sampling hides.","key_machinery":"The central object is Predictive Representativity, defined as $PR(P,S,A)=\\mathbb{E}_{X\\sim P_S}[D(P_S(Y|X)\\|\\hat{P}_S(Y|X))]-\\mathbb{E}_{X\\sim P}[D(P(Y|X)\\|\\hat{P}(Y|X))]$, the difference between the average statistical divergence between true and predicted label distributions within a subpopulation $S$ and the same divergence over the whole population $P$, for a divergence $D$ that the paper leaves open (Kullback-Leibler, Jensen-Shannon, and total variation are listed as candidates). In practice the paper uses metric-level PR, $PR_{M_i}(S)=M_i(S)-M_i(P)$, which compares a domain-relevant performance metric on the subpopulation against its population value; the case study applies this to precision, sensitivity, AUC-PR, specificity, accuracy, AUC-ROC, and F1-score. The External Transportability Criterion declares a model transportable to a new population $P'$ for subgroup $S'$ when $|PR(P',S',A)|\\leq\\epsilon$. The empirical estimator replaces the theoretical distributions with point masses at the true and predicted labels, and confidence intervals are obtained by nonparametric bootstrap.","core_discovery":"The central claim is that AI-based skin cancer classifiers trained on HAM10000 exhibit substantial, statistically significant predictive inequity across skin phototypes when evaluated on an independent clinical test set from Bogotá, Colombia. The discovery is that this inequity is invisible to proportional sampling: darker skin phototypes were included in the training data in proportions roughly consistent with their prevalence, yet all five benchmark CNN architectures — ResNet-50, DenseNet-121, MobileNet-V2, EfficientNet-V2-B0, and VGG-16 — show markedly worse precision, AUC-PR, and F1-score for malignant lesions on darker skin (Fitzpatrick IV–VI) than on lighter skin (II–III). The paper interprets this as evidence that representativity must be redefined as an outcome-level, context-sensitive property, and that Predictive Representativity operationalizes that shift.","pith_inferences":["Editorial inference: because metric-level PR is a difference of standard performance metrics, it can be applied to any classification task with labeled subgroups, so the auditing recipe extends beyond dermatology to other regulated domains.","Editorial inference: the formal divergence-based definition and the empirical point-mass estimator are not automatically the same quantity unless a specific divergence is chosen and consistency is proven; until then the case-study numbers are best read as ordinary subgroup performance gaps.","Editorial inference: a natural next experiment is to audit newer datasets that provide explicit skin-tone labels to see whether deliberate dark-skin curation shrinks the observed PR gaps or merely re-weights lesions the models already handle well.","Editorial inference: the paper's decision to exclude synthetic skin darkening can be tested directly by measuring PR after augmentation that preserves real dermatological features, which would show whether the gap closes or new artifacts appear."],"forward_implications":["Dataset-level representativeness in the sampling sense does not guarantee equitable conditional predictions, so claims of dataset diversity based on proportional composition alone are insufficient for fairness claims.","Any of the five architectures trained on HAM10000 will underperform on darker-skinned patients in a way that is clinically dangerous, because malignant lesions are more likely to be misclassified as benign in that group.","Aggregate accuracy of about 0.72 on the external BOSQUE test set coexists with a malignant-lesion precision gap of roughly 0.41 between skin-tone groups, so reporting only overall accuracy hides the subgroup failure.","The External Transportability Criterion provides a threshold-based check, $|PR(P',S',A)|\\leq\\epsilon$, for deciding whether a model's fairness record transfers to a new population.","Predictive Representativity is intended as a post-hoc auditing metric rather than a training objective, because directly optimizing for parity across subgroups could overfit to subgroup idiosyncrasies and hurt generalization."],"supporting_citations":[{"why":"Supplies HAM10000, the dataset on which every classifier in the case study is trained and whose diversity claims the paper disputes.","marker":"Tschandl et al., 2018"},{"why":"Earlier analysis reporting HAM10000's approximately 20:1 lighter-to-darker skin ratio and missing phenotype metadata, which the case study confirms and extends.","marker":"Morales-Forero et al., 2024"},{"why":"The BOSQUE Test set, the independent Colombian dermoscopic benchmark with Fitzpatrick phototype labels used for external validation.","marker":"Jaramillo Arboleda et al., 2025"},{"why":"Formal transportability theory that the External Transportability Criterion connects to for judging when fairness generalizes across populations.","marker":"Pearl & Bareinboim, 2011"},{"why":"Supplies the epidemiological context (lower melanoma incidence but severe outcomes in darker skin) that makes the precision gap clinically consequential.","marker":"Gohara, 2008"},{"why":"Supplies racial and ethnic variation data for melanoma incidence and survival that motivate the focus on darker-skinned patients.","marker":"Wu et al., 2011"}],"fun_headline_variants":["Skin cancer AI underperforms on darker skin despite proportional data","New audit metric reveals hidden racial bias in skin cancer AI","Proportional sampling can't fix AI skin cancer bias, study shows","Outcome-level fairness metric uncovers bias in skin cancer classifiers","AI skin cancer models consistently worse for darker patients, audit finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the empirical estimator in Eq. (3), which compares point-mass divergence in a subgroup against the full population, faithfully estimates the theoretical divergence-based Predictive Representativity; the paper does not state which divergence is used, and for Kullback-Leibler divergence that empirical quantity is undefined whenever the predicted and true labels differ, so if the identification fails the case study is measuring ordinary subgroup performance gaps rather than the new PR quantity.","fun_headline_variants_meta":{"raw":{"variants":["Skin cancer AI underperforms on darker skin despite proportional data","New audit metric reveals hidden racial bias in skin cancer AI","Proportional sampling can't fix AI skin cancer bias, study shows","Outcome-level fairness metric uncovers bias in skin cancer classifiers","AI skin cancer models consistently worse for darker patients, audit finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00021,"raw_usage":{"total_tokens":1410,"prompt_tokens":947,"completion_tokens":463,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":377}},"tokens_in":563,"tokens_out":463,"duration_ms":5719,"temperature":1.0,"reasoning_tokens":377,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:25:42.967194+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"The central claim would be overturned if a replication on a larger externally collected test set with Fitzpatrick annotations found no significant malignant-lesion precision gap between lighter (types II–III) and darker (types IV–VI) skin for any of the five architectures trained on HAM10000; the formal quantity would be undermined if the empirical cPR estimator, computed with a specified divergence and bootstrap intervals, failed to track the metric-level subgroup gaps on the same data.","supporting_citations":[{"cited_title":", Rosendahl, C","cited_arxiv_id":null,"evidence_quote":"Supplies HAM10000, the dataset on which every classifier in the case study is trained and whose diversity claims the paper disputes."},{"cited_title":", Rueda Jaime, L","cited_arxiv_id":null,"evidence_quote":"Earlier analysis reporting HAM10000's approximately 20:1 lighter-to-darker skin ratio and missing phenotype metadata, which the case study confirms and extends."},{"cited_title":", Sanchez Zapata, M J","cited_arxiv_id":null,"evidence_quote":"The BOSQUE Test set, the independent Colombian dermoscopic benchmark with Fitzpatrick phototype labels used for external validation."},{"cited_title":"\\ Bareinboim, E","cited_arxiv_id":null,"evidence_quote":"Formal transportability theory that the External Transportability Criterion connects to for judging when fairness generalizes across populations."},{"cited_title":"APACrefauthors \\ 2008","cited_arxiv_id":null,"evidence_quote":"Supplies the epidemiological context (lower melanoma incidence but severe outcomes in darker skin) that makes the precision gap clinically consequential."}],"review_version":1}