{"id":"6f82a11d-8e38-40b8-9a56-6711aa33d2ac","arxiv_id":"2508.04457","paper_version":2,"verdict":"CONDITIONAL","confidence":"LOW","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A broad benchmark of 13 uncertainty quantification methods on multi-label chest X-ray classification, with three methods newly extended to the multi-label setting.","lead":"This paper benchmarks 13 uncertainty quantification methods for multi-label chest X-ray classification on the MIMIC-CXR-JPG dataset, extending Evidential Deep Learning, HetClass networks, and Deep Deterministic Uncertainty to the multi-label setting. It maps which methods give reliable estimates of predictive, epistemic, and aleatoric uncertainty on real clinical images.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Disentanglement claims depend on an unvalidated joint-label information decomposition; per-label entropies would invalidate the epistemic/aleatoric split.","rationale":"The reader's weakest assumption is that epistemic/aleatoric disentanglement transfers to real multi-label clinical data without ground truth. My concern is a more specific, technical version of that assumption: even if the information-theoretic framework is valid in principle, it must be instantiated over the joint multi-label distribution. The corrupted full text leaves this unverifiable, and the correlations among chest X-ray findings make the per-label-versus-joint distinction potentially decisive. If the authors already compute joint entropies and include synthetic noise/shift controls, the concern would be resolved; if they use per-label marginals or lack such controls, the disentanglement rankings are not supported. I therefore agree with the reader's conditional verdict but sharpen the reason. I do not see grounds to reject the paper outright: the reliability/calibration benchmark could still be valuable, and the disentanglement issue is checkable once the intact text and experimental details are available.","tokens_in":19792,"tokens_out":5998,"duration_ms":73388,"concrete_test":"Check the intact Section 3 equations: does H[y|x,D] sum over the 2^L joint label vectors or over L per-label marginals? Then run a synthetic control on MIMIC-CXR-JPG (or a surrogate): add label noise at known rates (e.g., 0%, 10%, 20%) and add an OOD shift (e.g., image corruptions or a different source), and verify that the estimated aleatoric component increases monotonically with label-noise rate while the epistemic component increases with OOD shift. If either fails, the disentanglement benchmark should be re-reported as calibration/reliability only.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is not just a calibration ranking; it is that epistemic and aleatoric uncertainty can be disentangled on MIMIC-CXR-JPG. That claim rests on the information-theoretic decomposition total = expected-aleatoric + epistemic (mutual information), which is only the named decomposition when entropies are evaluated over the joint label vector y in {0,1}^L. The supplied text is corrupted, so the defining equations in Section 3 cannot be checked, but fragments do not confirm whether they use the joint distribution or per-label marginals. This is not pedantic: with correlated labels, sum_l I(y_l; theta | x,D) can differ from I(y; theta | x,D) in either direction, so a per-label computation would measure a different quantity and the disentanglement rankings would be unsupported. MIMIC-CXR labels are strongly co-occurring, making the risk concrete. Moreover, MIMIC-CXR-JPG provides no ground-truth split of aleatoric versus epistemic uncertainty, so the benchmark itself cannot catch such a mis-specification; the disentanglement component needs a synthetic control with known noise and known distribution shift to be load-bearing.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents an empirical benchmark of 13 uncertainty quantification (UQ) methods for multi-label chest X-ray classification on MIMIC-CXR-JPG, using both convolutional (ResNet) and transformer (ViT) architectures. The authors state that they extend Evidential Deep Learning, HetClass neural networks, and Deep Deterministic Uncertainty to the multi-label setting, and that they evaluate predictive, epistemic, and aleatoric uncertainties as well as the ability of methods to disentangle the latter two. The central claim is that the benchmark reveals method- and architecture-specific strengths and limitations. The supplied full text, however, is almost entirely corrupted: equations, tables, and experimental details are largely unreadable, so the protocol and the quantitative results can only be inferred from the abstract and occasional fragments.","tokens_in":20025,"tokens_out":3125,"duration_ms":38219,"significance":"If the benchmark is fully specified and technically correct, it would be a useful contribution: a public medical imaging dataset, 13 UQ methods, two architecture families, multi-label extensions of three previously single-label methods, and a head-to-head comparison of calibration and disentanglement properties. Such a comparison is genuinely lacking in the literature. The paper also has a clear falsifiable structure: given the full protocol, any of the reported rankings could be checked. However, the significance is conditional on details that cannot be verified in the current text: the exact information-theoretic decomposition, the metrics, the hyperparameters, the number of seeds, and the data splits. No code, configuration files, or reproducibility artifacts are visible in the submitted manuscript.","major_comments":[{"comment":"The central disentanglement claim rests on an information-theoretic decomposition of total uncertainty into expected aleatoric and epistemic components. In the corrupted text, the defining equations are unreadable, so I cannot determine whether entropies are computed over the joint label vector y in {0,1}^L or over per-label marginals. This is not a pedantic distinction: with correlated labels, the sum of per-label mutual informations differs from the mutual information of the joint label vector. MIMIC-CXR labels are strongly co-occurring, so the numerical results could support different rankings depending on the choice. Since MIMIC-CXR-JPG provides no ground-truth aleatoric/epistemic split, the benchmark cannot by itself catch such a mis-specification. Please provide the exact equations in Section 3, state explicitly whether the decomposition uses the joint or marginal entropy, and add","section":"Section 3 (entropy decomposition)"},{"comment":"The tables and the protocol sections are unreadable in the submitted full text. I cannot verify the reported numbers, the number of independent runs, the standard deviations, the train/validation/test splits, the metric definitions, or the hyperparameters for each of the 13 methods (e.g., evidential regularizer strength, HetClass variance terms, DDU feature-space thresholds). Without these, the benchmark ranking is not reproducible. A resubmission must include readable tables with numeric entries, clearly defined metrics, and a complete experimental protocol.","section":"Experimental tables and protocol"},{"comment":"MIMIC-CXR-JPG labels are derived from noisy automated extractors and are not perfectly reliable. If the benchmark evaluates calibration and uncertainty against these labels, a model that is correctly uncertain about a genuinely ambiguous or mislabeled image will be penalized. The manuscript does not appear to address this, either by analyzing label noise, by using a cleaned subset, or by discussing how label noise affects the ranking. At minimum, the authors should discuss this limitation and ideally provide a robustness check, because the claim of a validated UQ ranking on a real clinical task is stronger than a ranking on noisy surrogate labels.","section":"MIMIC-CXR label noise"}],"minor_comments":[{"comment":"The submitted PDF/text contains extensive mojibake and repeated garbled blocks, including the section headings and the reference list. A clean, readable manuscript is a precondition for review.","section":"Full text"},{"comment":"The abstract says the methods are evaluated 'across a wide range of tasks' but does not define what a task is (per-pathology classification, subpopulation, or something else). Please clarify this in the introduction.","section":"Abstract and terminology"},{"comment":"Even in the readable fragments, the notation for random variables is occasionally ambiguous (for instance, whether expectations are over the posterior over parameters or over the data distribution). Standardized notation would help.","section":"Notation"}],"recommendation":"major_revision","confidential_remarks":"The current submission is not reviewable in its present form because the full text is corrupted. I have recommended major revision rather than reject because the abstract-level claims are plausible and the issues are fixable: provide a clean manuscript, specify the joint versus marginal entropy decomposition, add a synthetic control for the disentanglement claim, and address label noise. Given the importance of the claimed benchmark, I would be willing to review a corrected version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about arXiv:2508.04457. First, it is a real benchmarking effort: 13 uncertainty quantification methods on MIMIC-CXR-JPG, with both ResNet and ViT backbones, and it adapts EDL, HetClass, and DDU to the multi-label setting. Those adaptations plus the sheer scale of the comparison are the genuine novelties. Second, the full text I was sent is corrupted beyond the abstract and fragments, so I cannot audit the protocol, tables, or equations. My verdict is conditional on the intact manuscript.\n\nWhat the paper does well is clear even from the abstract: it targets a practical question the field is actively deciding — which UQ estimate to deploy for medical imaging triage — and it tests methods across architectures rather than cherry-picking one. If the rankings hold, this becomes a reference benchmark. The multi-label extensions are not trivial, and the authors appear to ground them in the existing information-theoretic framework from prior work rather than inventing a parallel theory.\n\nThe soft spots, in order of weight. First, the disentanglement claim rests on an information-theoretic decomposition where the split into epistemic and aleatoric parts is only the named decomposition when entropies are computed on the joint label vector. I could not verify from the corrupted text whether the authors used the joint distribution or per-label marginals. With MIMIC-CXR's strongly co-occurring labels, per-label entropies would measure a different quantity and the disentanglement rankings would be unsupported. This is not pedantic; it is the load-bearing assumption of the 'disentanglement' half of the paper. They need a synthetic control with known noise and known shift, or at minimum an explicit statement and justification of the joint-label computation, or that half of the claims should be dropped. Second, MIMIC-CXR labels are noisy, so calibration and reliability metrics target partially arbitrary ground truth. That is a limitation of the dataset, not a fatal flaw, but it should be acknowledged. Third, the EDL and HetClass extensions have their own objectives that bake in a separation of aleatoric and epistemic parts, so evaluating 'disentanglement' on those methods has mild circularity. Not disqualifying, but worth naming.\n\nThe reader's skepticism about the disentanglement assumption is fair, and the stress-test concern is well aimed. But I would not desk reject this. It is a useful empirical contribution, and the core ranking claims could stand even if the disentanglement claims need repair. Who benefits: anyone working on UQ for medical imaging or multi-label problems. I would send it to a serious referee, with the explicit instruction to check the joint- versus per-label entropy computation and to ask for the synthetic control. Whether I cite it depends on seeing the intact tables and equations.\n\nMy recommendation: engage with it, but only after the authors confirm the decomposition and ideally add the control. If they cannot, the calibration ranking may still be worth something, but the headline disentanglement result should not be trusted.","headline":"A serious, broad UQ benchmark for multi-label chest X-ray that likely deserves referee time, but the disentanglement claims hinge on details the corrupted text hides.","tokens_in":20537,"tokens_out":956,"would_cite":false,"duration_ms":13296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A chest X-ray benchmark evaluates 13 uncertainty methods and tests whether they can separate lack-of-knowledge from inherent noise.","keywords":["uncertainty quantification","multi-label classification","chest X-ray","epistemic uncertainty","aleatoric uncertainty","evidential deep learning","vision transformer","MIMIC-CXR-JPG"],"falsifier":"Take the MIMIC-CXR-JPG benchmark and add a controlled amount of label noise plus a controlled distribution shift, such as a held-out hospital or imaging protocol. If methods' aleatoric estimates rise with label noise while their epistemic estimates rise with the shift, the disentanglement claim is supported; if the epistemic estimate rises with label noise or the aleatoric estimate rises with the shift, that method's disentanglement claim fails.","tokens_in":19648,"feed_emoji":"🩸","tokens_out":6026,"duration_ms":64947,"temperature":0.7,"pith_summary":"This paper is a benchmark study asking whether commonly used uncertainty quantification methods can be trusted in a real clinical setting: multi-label chest X-ray classification on the MIMIC-CXR-JPG dataset. It evaluates 13 methods on both a convolutional ResNet and a Vision Transformer, and extends three methods—Evidential Deep Learning, HetClass neural networks, and Deep Deterministic Uncertainty—to multi-label prediction. The goal is to see not only which methods give well-calibrated uncertainty estimates, but whether they can separate epistemic uncertainty (what the model does not know because of missing data) from aleatoric uncertainty (noise inherent to the image and label). The paper's central assertion is that its benchmark provides a reliable picture of method- and architecture-specific strengths and limitations for both uncertainty estimation and epistemic/aleatoric disentanglement on a real medical task.","feed_headline":"Benchmark ranks 13 uncertainty methods for chest X-ray AI","feed_subtitle":"Thirteen methods on ResNet and ViT show the best uncertainty estimate depends on the task","key_machinery":"The central machinery is the information-theoretic decomposition of a model's predictive uncertainty into an aleatoric component (irreducible noise) and an epistemic component (reducible lack of knowledge). The paper adapts this decomposition to multi-label chest X-ray prediction and uses it to turn each of the 13 methods into a source of both components. The comparison of these two components across methods and architectures is what allows the benchmark to assess disentanglement quality, not just calibration.","core_discovery":"The paper's central claim is that uncertainty quantification, including the separation of epistemic from aleatoric uncertainty, can be meaningfully benchmarked in a real multi-label medical imaging setting rather than only on synthetic or natural-image data. To make this concrete, the authors run 13 uncertainty quantification methods on the MIMIC-CXR-JPG chest X-ray dataset under two architectures, ResNet and Vision Transformer, and extend Evidential Deep Learning, HetClass neural networks, and Deep Deterministic Uncertainty to multi-label outputs. The benchmark is designed to answer which methods produce well-calibrated uncertainty and which methods disentangle “what the model does not know","pith_inferences":["Beyond the paper: the disentanglement claims could be tested more directly by injecting controlled label noise and distribution shift into the same dataset; the split would be confirmed only if aleatoric estimates track the noise and epistemic estimates track the shift.","Beyond the paper: because chest X-ray findings co-occur and annotations are noisy, per-label uncertainty components may not be independent; pooled disentanglement metrics could hide leakage between the two components.","Beyond the paper: the benchmark implies practical guidance that uncertainty-based referral to a human reader should be tuned per finding and per architecture, not adopted from a single winner's overall score."],"forward_implications":["If the benchmark is right, practitioners get a direct comparison of 13 uncertainty quantification methods on a realistic multi-label medical task, so method choice can be based on measured calibration and disentanglement rather than single-dataset folklore.","The multi-label extensions of Evidential Deep Learning, HetClass neural networks, and Deep Deterministic Uncertainty become usable baselines for future chest X-ray and other multi-label medical imaging research.","Architecture choice matters: conclusions about which uncertainty method to trust drawn from a ResNet should not be assumed to transfer to a Vision Transformer, and vice versa.","The benchmark supplies a template for evaluating epistemic/aleatoric disentanglement in settings where no ground-truth split of uncertainty is available.","No single method appears to dominate across all tasks, so uncertainty-aware deployment will likely require matching methods to specific findings and backbones."],"supporting_citations":[],"fun_headline_variants":["13 uncertainty methods on real chest X-rays","Chest X-ray AI: 13 uncertainty methods ranked","Multi-label chest X-ray: uncertainty benchmark","Which uncertainty method for chest X-ray AI? Depends","Epistemic vs aleatoric: 13 ways for chest X-rays"],"cache_read_input_tokens":2816,"weakest_assumption_plain":"The load-bearing premise is that lack-of-knowledge uncertainty and inherent-noise uncertainty can be cleanly separated in real multi-label chest X-ray data, even though the dataset never labels which part of an error is which.","fun_headline_variants_meta":{"raw":{"variants":["13 uncertainty methods on real chest X-rays","Chest X-ray AI: 13 uncertainty methods ranked","Multi-label chest X-ray: uncertainty benchmark","Which uncertainty method for chest X-ray AI? Depends","Epistemic vs aleatoric: 13 ways for chest X-rays"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1434,"prompt_tokens":675,"completion_tokens":759,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":680}},"tokens_in":419,"tokens_out":759,"duration_ms":7913,"temperature":1.0,"reasoning_tokens":680,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T23:57:03.232185+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the MIMIC-CXR-JPG benchmark and add a controlled amount of label noise plus a controlled distribution shift, such as a held-out hospital or imaging protocol. If methods' aleatoric estimates rise with label noise while their epistemic estimates rise with the shift, the disentanglement claim is supported; if the epistemic estimate rises with label noise or the aleatoric estimate rises with the shift, that method's disentanglement claim fails.","supporting_citations":[],"review_version":1}