{"id":"727f2f30-8de5-467f-ae80-2d5125cb1916","arxiv_id":"2607.07717","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Rare-label fairness in multi-label CXR depends jointly on finding, subgroup, and operating threshold; tail-aware thresholds cut subgroup FNR more than class-level losses or GroupDRO alone.","lead":"After converting long-tailed chest X-ray model scores into yes/no decisions, rare findings are still missed far more often in some sex, age, race, and insurance groups. The paper shows that the operating threshold, not just ranking metrics or class reweighting, largely decides who is left below the line.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.5","headline":"M5–M6 FNR gaps that carry the joint finding-subgroup-threshold claim rest on sparse VinDr cells and a deliberately lenient worst-group-recall rule (τ=0.05), risking overstated isolation of a general score-to-action failure.","rationale":"The reader correctly flags the representativeness of the F1 vs. τ=0.05 worst-group-recall policies under VinDr sparsity as the weakest assumption. The paper is transparent about cell sizes, supplies bootstrap CIs, support checks, seed aggregates and MIMIC corroboration, and never claims a deployable remedy—only a measurement point. The existence of a pure threshold effect is true by construction once thresholds differ; the soft spot is only the magnitude and generality of that effect in the sparse regime that supplies the striking numbers. No internal inconsistency or stronger soundness flaw appears. Hence the CONDITIONAL verdict, medium correctness risk and call for code/artifacts remain appropriate; the multi-τ / multi-objective check above would directly raise or lower confidence in the load-bearing quantitative support.","tokens_in":10456,"tokens_out":670,"duration_ms":19299,"concrete_test":"Recompute M6 tail thresholds on the VinDr validation split for τ∈{0.05,0.10,0.20} and for an alternative objective that maximizes macro-tail recall (not worst-group) under the same floors; report paired-bootstrap M5–M6 ΔFNR for tail and sex/age worst-group endpoints plus alert rates. If worst-group ΔFNR falls below ~0.15 or loses significance for τ≥0.10 while alert burden stays elevated, the isolation of a robust score-to-action effect fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central audit claim—that rare-label fairness depends jointly on finding, subgroup and operating threshold, not ranking metrics alone—is quantitatively carried by the controlled M5–M6 comparison (identical scores/ECE/Brier, different thresholds only) and the large VinDr FNR drops (tail 0.665→0.269, sex wg 0.705→0.157, age wg 0.822→0.133; Tables 2/4). M6 chooses tail thresholds by maximizing worst-group recall subject to precision floor τ=0.05 (Sec. 2). With only 3–19 positives per tail class and many subgroup-tail cells ≤5 (Table 1, Sec. 3.1, Limitations), both the F1 baselines and the worst-group objective are low-resolution; the headline point estimates can be inflated by the lenient floor and by which sparse cells happen to be worst-off. MIMIC shows only smaller absolute reductions with high residual FNR (0.866→0.741), so VinDr magnitudes supply the load-bearing support for “first-order” threshold importance. If those magnitudes shrink under modestly stricter or non-worst-group objectives, the claim that thresholding isolates a general fairness failure (beyond ranking) weakens precisely in the sparse-tail regime the paper emphasizes.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies pre-deployment fairness in long-tailed multi-label CXR classification as an audit of who is missed after scores become decisions. Using a diagnostic ladder (BCE, ASL, class-balanced ASL, GroupDRO, group-tail ASL, and the same scores with tail-aware thresholds) on VinDr-CXR and MIMIC-CXR/CXR-LT, it separates class-level long-tail losses, subgroup-aware weighting, group robustness, and threshold selection. The central claim is that rare-label fairness depends jointly on finding, subgroup, and operating threshold, not ranking metrics or label frequency alone. On VinDr, group-tail weighting plus tail-aware thresholding reduces tail FNR from 0.665 to 0.269 and worst-group FNRs for sex and age substantially, with modest macro-mAP gain; on MIMIC the same score-to-threshold step lowers tail and worst-group FNR across sex, age, race, and insurance, though residual FNRs remain high. Paired bootstrap contrasts and GroupDRO references support that aggregate group robustness alone does not remove rare subgroup misses after thresholding.","tokens_in":10870,"tokens_out":1290,"duration_ms":9427,"significance":"If the result holds, the paper makes a useful methodological contribution to medical imaging fairness: it reframes underdiagnosis as a thresholded class-subgroup cell rather than a pure ranking or class-frequency problem, and shows that the score-to-action step can dominate observed tail harm. Strengths include the clean M5–M6 isolation (identical scores, ECE, and Brier; only the threshold rule changes), dual-cohort evaluation with richer MIMIC axes, paired bootstrap contrasts, explicit positive-support reporting, and workload/alert-burden trade-offs. The narrow audit framing and limitations discussion are appropriately cautious. The work is more measurement than remedy, but that is a legitimate and needed contribution for CXR triage fairness reporting.","major_comments":[{"comment":"Sec. 2 (Diagnostic ladder) and Tables 2/4: the load-bearing M5–M6 FNR reductions rest on a deliberately lenient tail-aware rule (maximize worst-group recall subject to precision floor τ=0.05). With VinDr tail support of only 3–19 positives per class and many subgroup-tail cells ≤5 (Table 1; Limitations), both F1 baselines and the worst-group objective are low-resolution. The paper already shows τ=0.10 still helps but with different alert rates (Sec. 3, Fig. 2); a fuller sensitivity analysis over τ and non-worst-group objectives (e.g., equalized opportunity or fixed-FPR) is needed to show that the isolation of a general score-to-action fairness failure is not an artifact of this particular sparse-cell objective.","section":"Sec. 2 Diagnostic ladder; Tables 2, 4; Fig. 2"},{"comment":"Sec. 3.1–3.2 and Limitations: VinDr supplies the large absolute FNR drops that make the joint finding–subgroup–threshold claim vivid, while MIMIC shows only smaller absolute reductions with high residual FNR (0.866→0.741). Incomplete VinDr metadata (informative sex on 49% of test images; PatientAge missing in 69%) further weakens worst-group estimates. The manuscript should either strengthen VinDr evidence (more seeds already help; disease-level support checks are good) or rebalance the claim so that MIMIC’s directional but modest effect, not VinDr point estimates, carries the primary quantitative weight.","section":"Sec. 3.1–3.2; Limitations; Table 1"},{"comment":"Sec. 2 and Tables 2–3: GroupDRO is correctly labeled a reference rather than a ladder rung, but the claim that “aggregate group robustness alone does not remove rare subgroup misses” would be stronger if GroupDRO groups were defined at the same class–subgroup granularity used for M5/M6 audit cells, or if this mismatch were quantified. As written, it is unclear whether the residual misses reflect a true distinction between group robustness and thresholded rare-positive recall, or a coarser group definition.","section":"Sec. 2; Tables 2–3"}],"minor_comments":[{"comment":"Fig. 1 and Fig. 2 are informative but dense; adding explicit n+ per tail class (or a small support table) next to the disease-level discussion in Sec. 3.1 would help readers judge resolution without hunting through the text.","section":"Fig. 1–2; Sec. 3.1"},{"comment":"Clarify the exact training-frequency cutoff used to define “tail” classes within each cohort (Table 1 states counts but not the rule).","section":"Table 1; Sec. 2 Data"},{"comment":"A short note on why ConvNeXt-Tiny alone is sufficient for the audit claim (vs. at least one alternative backbone) would address architecture-generality concerns already flagged in Limitations.","section":"Sec. 2; Limitations"},{"comment":"Minor prose issues: occasional missing spaces after periods in the abstract/intro (“thresholdselection”, “Existinglong-tailmethods”) and inconsistent hyphenation of “worst-group” / “worst group”.","section":"Abstract; Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid measurement contribution and fits a fairness/medical-imaging venue better than a pure methods venue. The central design (identical-score M5–M6) is sound; the main risk is over-weighting sparse VinDr magnitudes. I would not reject on that basis—major revision with threshold sensitivity and claim rebalancing should suffice. No concerns about misconduct or citation gaming."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is narrow and real: after you turn long-tailed CXR scores into decisions, who gets missed is not fixed by macro-mAP or by aggregate group robustness. The paper’s best move is the diagnostic ladder that holds scores fixed between M5 and M6, so the FNR drop is pure threshold policy. That isolation is cleaner than most fairness-in-imaging papers manage.\n\nWhat is new is the audit unit—rare class \times subgroup \times thresholded decision—plus the controlled comparison on VinDr and MIMIC/CXR-LT. Group-tail weighting plus tail-aware thresholds cut VinDr tail FNR 0.665\to0.269 and worst-group sex/age FNR sharply, with paired bootstrap support; MIMIC shows the same direction across sex, age, race, and insurance, with high residual miss rates left on the table. They also show GroupDRO alone does not clear rare subgroup misses, and they report missed positives per 100 and alert burden so the workload trade-off is visible. Citations are appropriate; the math is ordinary empirical comparison, not circular.\n\nSoft spots are real but the paper mostly owns them. VinDr tail cells are tiny (3–19 positives; many subgroup cells ≤5), so the headline magnitudes rest on low-resolution estimates and a deliberately lenient τ=0.05 worst-group-recall rule. MIMIC reductions are smaller and residual FNR stays high, which actually supports their measurement claim more than a “solved” claim. Single backbone, incomplete VinDr metadata, no released code. The stress-test concern about overstated isolation is fair as a caution on magnitude, not a refutation of the joint claim—the paper already frames itself as an audit tool, not a deployable fix.\n\nThis is for people who audit clinical multi-label models and care about operating points. Worth a serious referee. I would engage, cite the audit framing, and push for code plus multi-architecture checks.","headline":"Clean audit paper: rare CXR fairness is a finding\times subgroup\times threshold problem, and the M5–M6 isolation is the real contribution—not a new loss.","tokens_in":11502,"tokens_out":496,"would_cite":true,"duration_ms":4368,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Rare chest X-ray findings can rank well yet still miss subgroup patients once scores become yes/no decisions; the threshold, not just the model, decides who is left out.","keywords":["fairness","long-tailed learning","multi-label classification","chest X-ray","subgroup bias","underdiagnosis","thresholded FNR","group robustness"],"falsifier":"On the same VinDr and MIMIC test splits, re-run the M5–M6 comparison with alternative clinically plausible threshold policies (for example equalized opportunity per subgroup, fixed alert-rate budgets, or higher precision floors) and check whether the large reductions in tail and worst-group false-negative rates disappear or reverse while ranking metrics stay fixed.","tokens_in":11360,"feed_emoji":"🫁","tokens_out":746,"duration_ms":5764,"temperature":0.7,"pith_summary":"Chest X-ray models are often judged by ranking scores, but a real triage system only acts when a score crosses a threshold. This paper shows that rare findings can still leave true-positive patients below that line, and that the misses concentrate in particular disease–subgroup cells rather than following label frequency alone. Using a diagnostic ladder on VinDr-CXR and MIMIC-CXR/CXR-LT, the authors separate class-level long-tail losses, subgroup-aware weighting, group-robust training, and the final threshold rule. On VinDr, group-tail weighting plus a tail-aware operating point cuts tail false-negative rate from 0.665 to 0.269 and sharply lowers worst-group rates for sex and age while macro-mAP rises slightly; the same score-to-threshold step also lowers worst-group rates across sex, age, race, and insurance on MIMIC, though residual miss rates stay high. The claim is deliberately narrow: rare-label fairness in this setting is jointly determined by finding, subgroup, and operating threshold, so fairness reports must include support, subgroup-conditioned misses, workload trade-offs, and the policy that turns scores into action.","feed_headline":"Who gets missed when rare X-ray scores become decisions","feed_subtitle":"Thresholds, not ranking alone, decide which rare-positive subgroups stay below the line","key_machinery":"A diagnostic ladder of model probes (BCE, asymmetric loss, class-balanced weighting, GroupDRO, group-tail weighting, then tail-aware thresholds on the same scores) that isolates class imbalance, subgroup support, group robustness, and the score-to-decision step, with the audit unit defined as a positive rare-label case within a subgroup after a class-specific threshold.","core_discovery":"Rare-label fairness in long-tailed multi-label chest X-ray classification depends jointly on the finding, the subgroup, and the operating threshold, not on label frequency or ranking metrics alone. Class-level long-tail methods and aggregate group robustness do not fully determine who remains below threshold; changing the tail threshold on an unchanged score model materially changes subgroup false-negative rates.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Who stays below threshold when rare CXR scores become decisions","Rare CXR fairness hinges on finding, subgroup and threshold","Tail-aware thresholds cut rare-positive misses across CXR subgroups","Ranking alone cannot predict who is missed under CXR thresholds","Operating thresholds decide rare-label misses in CXR subgroups"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The paper treats its chosen validation F1 thresholds and the specific tail-aware rule (maximize worst-group recall under a 0.05 precision floor) as representative enough operating policies that the observed score-to-threshold gaps generalize beyond these choices and the very small VinDr tail cells.","fun_headline_variants_meta":{"raw":{"variants":["Who stays below threshold when rare CXR scores become decisions","Rare CXR fairness hinges on finding, subgroup and threshold","Tail-aware thresholds cut rare-positive misses across CXR subgroups","Ranking alone cannot predict who is missed under CXR thresholds","Operating thresholds decide rare-label misses in CXR subgroups"]},"model":"grok-4.5","effort":"low","cost_usd":0.004562,"raw_usage":{"total_tokens":1395,"prompt_tokens":856,"num_sources_used":0,"completion_tokens":85,"cost_in_usd_ticks":45620000,"prompt_tokens_details":{"text_tokens":856,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":454,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":856,"tokens_out":85,"duration_ms":3792,"temperature":1.0,"reasoning_tokens":454,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T23:48:20.428522+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"On the same VinDr and MIMIC test splits, re-run the M5–M6 comparison with alternative clinically plausible threshold policies (for example equalized opportunity per subgroup, fixed alert-rate budgets, or higher precision floors) and check whether the large reductions in tail and worst-group false-negative rates disappear or reverse while ranking metrics stay fixed.","supporting_citations":[],"review_version":1}