{"id":"f97ad0a6-be25-4b7d-971d-39325c39720e","arxiv_id":"2505.15997","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"An ensemble of three vision transformers with conformal prediction achieves 90.38% coverage on skin-lesion classification, but the reported improvement over single models is not evaluated with a fixed error rate.","lead":"This paper combines three vision transformer models trained on different skin-lesion datasets, averages their outputs, and applies conformal prediction to produce uncertainty sets. It reports a coverage rate of 90.38% and argues the ensemble handles domain-shifted images more reliably than single models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage improvement claim is uninterpretable because the target miscoverage rate alpha is never reported; with fixed alpha all valid conformal models should yield similar coverage.","rationale":"The reader's verdict is REJECT, and my stress-test identifies a closely related but more fundamental problem: the headline coverage comparison is not interpretable because the target miscoverage rate alpha is never reported. In split conformal prediction, the user specifies alpha, and the calibration quantile is computed to control coverage at 1-alpha. If the same alpha were used for all models, valid CP should give approximately equal empirical coverage; large gaps imply either different alpha values or invalid calibration for the baseline models. The paper's discussion in Sections II-C and III-B does not mention alpha, and Eq. (1) is internally inconsistent about whether the prediction set uses scores below q_hat or below 1-q_hat. This makes the central claim of a 9.95% coverage improvement vacuous as stated. The reader's weakest assumption about exchangeability and label alignment is also relevant, but the missing alpha is more load-bearing because it undermines the comparison even under perfect exchangeability. I agree with the rejection verdict, so no change is needed. The concrete test of fixing alpha and recomputing would settle whether any real improvement exists behind the reported numbers.","tokens_in":6270,"tokens_out":2889,"duration_ms":25792,"concrete_test":"Re-run all experiments with a fixed, explicitly declared alpha (e.g., alpha = 0.1), using the same merged calibration set to compute the quantile for every compared model and the same merged test set for evaluation. If CE-ViTs coverage still exceeds HAM10000 by about 10 points while average prediction set sizes are comparable, the improvement claim has content; if all coverages cluster near 90%, the claimed improvement collapses. Also fix Eq. (1) to C(X_test) = {y : s_y(X_test) >= 1 - q_hat} and recompute all reported prediction-set sizes and coverage values.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that CE-ViTs achieves 90.38% coverage, a 9.95% improvement over the HAM10000 model—cannot be evaluated because the paper never states the target miscoverage level alpha used in split conformal prediction (Section II-C, III-B). In split conformal prediction, the calibration quantile is chosen so that coverage is approximately 1-alpha (Eq. 2). If alpha were held fixed and all calibration sets were valid for the merged test set, empirical coverage should be near 1-alpha for every model; a 10-point gap then signals different effective alpha or invalid calibration, not improved uncertainty quantification. The paper compares raw coverage rates without reporting alpha or demonstrating that the single-dataset models were calibrated under exchangeable conditions on the merged test distribution. Additionally, Eq. (1) defines the prediction set as {y : S(X_test) < q_hat}, while the text says classes with SoftMax output less than 1-q_hat are included; these definitions are contradictory and suggest the implemented quantile/threshold step may be incorrect. Thus the reported 9.95% coverage improvement is not a valid comparison of conformal methods and does not support the stated claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CE-ViTs, an ensemble of vision transformers trained on three skin-lesion datasets (HAM10000, Dermofit, ISIC) whose softmax outputs are averaged and then post-processed with split conformal prediction using a merged calibration set. The authors claim that CE-ViTs achieves 90.38% coverage on a pooled test set, a 9.95% improvement over the HAM10000 expert model, and that it produces larger prediction sets for misclassified samples (average size 3.075 vs. 1.86 for DMF). The paper also reports accuracy, precision, recall, and F1 metrics for the individual models and the ensemble.","tokens_in":6401,"tokens_out":2758,"duration_ms":24361,"significance":"If the claims were properly supported, combining ensemble learning with conformal prediction for domain-adaptive medical image classification would be a useful contribution. However, the evaluation as presented does not support the headline coverage claim: the target miscoverage rate alpha is never reported, the coverage values are compared without controlling for alpha, and the prediction-set definition in Eq. (1) contradicts the accompanying text. The paper also lacks error bars, confidence intervals, and a clear statement of the exchangeability assumptions for the merged calibration/test split. These are not presentation issues but invalidate the main empirical comparison. The paper does make use of a standard conformal-prediction framework, but the implementation and evaluation as described are not sufficient to establish the claimed improvements.","major_comments":[{"comment":"The target miscoverage rate alpha is never stated, although the conformal guarantee in Eq. (2) is explicitly conditioned on alpha. The headline result, 90.38% coverage versus 80.43% for the HAM10000 model, is therefore uninterpretable: if all models were calibrated at the same alpha on the same test distribution, their coverage values should all be close to 1-alpha up to finite-sample error. A 9.95 percentage-point gap indicates either different effective alpha values or invalid calibration, not improved uncertainty quantification. The authors must report alpha for every experiment, demonstrate that the same alpha was used, and show that calibration is valid on the pooled test distribution before any coverage comparison can be meaningful.","section":"Section II-C and Section IV"},{"comment":"Equation (1) defines the prediction set as C(X_test) = {y : S(X_test) < q_hat}, but the text immediately states that the set includes classes whose softmax output is less than 1 - q_hat. These definitions are contradictory. If the conformity score is the softmax output, the standard conformal set for classification is typically {y : S(X_test) >= 1 - q_hat} (or equivalently {y : S(X_test) > q_hat} with a different score definition). The implemented threshold therefore appears to be inconsistent with the stated formula, and this could directly affect all reported coverage values and set sizes.","section":"Section II-C, Eq. (1) and following text"},{"comment":"The validity of the split conformal procedure requires that the calibration samples and test samples are exchangeable. The paper states that calibration data is a merged set of HAM10000, DMF, and ISIC and that the test set is 'combined,' but it does not specify how the pooled split was created—whether it was stratified by dataset, whether dataset proportions were preserved, or whether the split was random at the sample level. If the pooled test set is not exchangeable with the pooled calibration set, the reported coverage is not a valid guarantee. Moreover, per-dataset coverage is not reported, so the claimed domain-adaptation benefit over individual models is not substantiated.","section":"Section III-A and III-B"},{"comment":"All coverage and set-size results are reported as point estimates with no error bars, confidence intervals, or sample sizes. The finite-sample conformal bound in Eq. (2) has width 1/(n+1), so without n the reported 9.95% difference cannot be distinguished from finite-sample variation. The paper should report n, standard errors, or confidence intervals for coverage and set size, and should show whether the difference between CE-ViTs and the HAM10000 model is statistically significant.","section":"Section IV, Tables 1–2 and Figures 3–4"}],"minor_comments":[{"comment":"The phrase 'To aim this challenge' should read 'To address this challenge.'","section":"Abstract"},{"comment":"The notation S = sigma f(X) is ambiguous; please clarify whether f denotes the logits and sigma the softmax, and define the conformity score explicitly.","section":"Section II-C"},{"comment":"The footnotes for Accuracy 1 and Accuracy 2 are confusing: Accuracy 1 is computed on each model's own training dataset, which is not a common test set. For a fair comparison, report accuracy on a single pooled test set for all models.","section":"Section IV, Table 1"},{"comment":"The symbols C_correct and C_incorrect are not defined in the text or table caption; please define them as average prediction-set sizes for correctly and incorrectly classified test samples.","section":"Section IV, Table 2"},{"comment":"The 'uncertainty value' is never formally defined. Please specify how uncertainty is computed and what the histograms show.","section":"Section IV, Figure 4"},{"comment":"The dataset name is given as 'Dermofit' in the introduction and abstract but as 'DMF' in the methodology and tables; please use one consistent name.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The manuscript's central empirical claim is not supported because the conformal error rate alpha is never reported and the coverage comparison is therefore uninterpretable. The contradiction between Eq. (1) and the text suggests a possible implementation error. These are load-bearing issues that would require a complete redesign of the experiments and likely new results, rather than a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe headline result—CE-ViTs reaches 90.38% coverage, 9.95% above the HAM10000 model—is not interpretable because the paper never states the target miscoverage rate alpha. In split conformal prediction, coverage is controlled by the calibration quantile: if alpha is fixed and calibration is valid, every model should land near 1-alpha. A ten-point gap between models means either alpha differed or the calibration/test sets weren't exchangeable. Without alpha, the claim is circular: 90.38% looks exactly like what you'd get from alpha=0.1.\n\nWhat the paper does well: it applies a sensible pipeline—fine-tuned ViTs, simple softmax averaging, split conformal with a merged calibration set—to a clinically relevant task, and it reports accuracy, set size, coverage, and uncertainty. The idea that ensembling helps under domain shift is reasonable, and the multi-dataset setup is a step beyond single-source evaluation.\n\nThe soft spots are serious. Eq. (1) defines the prediction set as {y: S(x) < q_hat}, while the text says classes are included when the softmax output is less than 1 - q_hat. Those two definitions cannot both be right; the implemented prediction set is unclear. No error bars, no multiple seeds, no code or data release. The comparison of single models vs. the ensemble may have used different calibration procedures; the paper does not specify whether each model's CP was calibrated on its own dataset or the merged calibration set. If the single models were calibrated on their native splits and tested on the pooled test set, their low coverage is just invalid calibration, not evidence that the ensemble is better.\n\nOn novelty: this is an application of known components. The authors' earlier work already applied conformal prediction to skin lesions; ensembling softmax outputs before conformal calibration is a standard trick. There is no new theory or algorithmic contribution.\n\nFor a reader wanting a quick case study of ensemble CP in dermatology, there is a kernel of usefulness, but as written the central claim does not hold up. I would desk-reject, though I'd invite a resubmission that reports alpha, fixes Eq. (1), adds uncertainty bars, and clarifies the calibration protocol.","headline":"Coverage comparison is uninterpretable because alpha is never reported; the paper is a routine application with a contradictory prediction-set definition.","tokens_in":7000,"tokens_out":3483,"would_cite":false,"duration_ms":26801,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CE-ViTs claims that averaging three ViT classifiers before conformal calibration lifts label coverage on mixed skin-lesion data to 90.38 percent, up from 80.43 for the best single model.","keywords":["skin lesion classification","conformal prediction","vision transformers","ensemble learning","domain adaptation","uncertainty quantification","medical imaging"],"falsifier":"Calibrate on the pooled calibration split as the paper does, then evaluate coverage separately on each dataset's test split; if the per-source coverage on the most domain-shifted source falls far below the nominal level, the pooled coverage figure does not hold under the domain shift the paper claims to address.","tokens_in":5984,"feed_emoji":"🩺","tokens_out":6052,"duration_ms":50442,"temperature":0.7,"pith_summary":"This paper tries to establish that an ensemble of vision transformers gives more trustworthy skin-lesion classifications than any single model when the data come from different sources. The proposed CE-ViTs framework averages the softmax outputs of three ViT models, each trained on one of HAM10000, Dermofit, or Skin Cancer ISIC, and then applies split conformal prediction with calibration data merged from all three sources. The authors report that this raises coverage of the true label to 90.38 percent on a mixed test set, an improvement of 9.95 percentage points over the best single model, and that misclassified samples receive larger prediction sets with higher uncertainty. If the result holds, ensembles plus conformal calibration are a practical route to safer decision support in medical imaging.","feed_headline":"Ensembled ViTs lift skin-lesion coverage to 90.38%","feed_subtitle":"Averaging three transformer models before conformal calibration beats any single model on mixed skin data.","key_machinery":"The machinery is an ensemble of three Vision Transformers whose softmax outputs are combined by simple averaging, followed by split conformal prediction. The conformal layer treats the averaged softmax score as a conformity score, estimates one quantile from a calibration set that pools HAM10000, Dermofit, and ISIC data, and builds for each test image a set of classes consistent with that quantile. The averaging step is what the paper credits with adapting to multiple domains; the conformal step is what converts the averaged scores into prediction sets with a stated coverage bound.","core_discovery":"The central claim is that CE-ViTs improves conformal prediction through domain-adaptive ensembling rather than through better single-model accuracy. Each ViT is fine-tuned on a different skin dataset, and at inference the three softmax vectors are averaged; this averaged score is then fed to a conformal module that computes an empirical quantile from merged calibration data and outputs a prediction set per image. The paper reports coverage of 90.38 percent, exceeding the HAM10000 expert model's 80.43 percent, while increasing the average prediction set size for wrongly classified samples from 1.86 to 3.075 and producing low-to-intermediate uncertainty for correct predictions and high uncertainty for incorrect ones. The authors interpret these results as evidence that the ensemble's pooled prediction sets are more likely to contain the true lesion class under domain shift, and that overconfident misclassifications are reduced.","pith_inferences":["The paper reports only pooled coverage; a natural next step is to report coverage separately for each source, since a mixed test set can hide poor performance on one domain.","The authors list weighted ensembles as future work; an implicit extension is tuning ensemble weights on calibration data to optimize set size at the same coverage level.","The validity guarantee depends on exchangeability of pooled calibration and pooled test data, so deployment with different source proportions would need group-conditional or per-source calibration rather than one global quantile.","A fair comparison of efficiency should hold coverage fixed: two methods with the same 90 percent target should be compared by average prediction set size, otherwise a larger set trivially buys more coverage."],"forward_implications":["If the pooled coverage claim holds, a skin-lesion screening tool can output a short list of plausible classes and tell the clinician that the true label falls in that list roughly 90 percent of the time.","Merging calibration data from several sources before computing the quantile is a transferable recipe for conformal prediction under mixed domain shifts.","The prediction-set pattern reported for CE-ViTs, small sets for correct cases and larger, higher-uncertainty sets for incorrect cases, is the behavior a decision-support system needs to avoid silent failures.","Because the conformal layer only sees averaged scores, the same ensembling-with-calibration recipe could be applied to other classifier families, not only vision transformers."],"supporting_citations":[{"why":"Supplies the HAM10000 skin-lesion images used for training one expert model and for merged calibration.","marker":"[19]"},{"why":"Supplies the Skin Cancer ISIC images used for training one expert model and for merged calibration.","marker":"[20]"},{"why":"Supplies the pre-trained ViT weights and fine-tuning recipe for all three expert models.","marker":"[21]"},{"why":"Defines the conformal prediction-set construction the framework applies to averaged softmax scores.","marker":"[17]"},{"why":"Provides the distribution-free coverage guarantee cited in the paper, the theoretical basis for validity.","marker":"[18]"},{"why":"Supplies the domain-adaptive ensemble learning mechanism that motivates averaging predictions across datasets.","marker":"[10]"},{"why":"Provides prior empirical validation of conformal prediction on skin lesion classification, motivating the safety-critical framing.","marker":"[11]"}],"fun_headline_variants":["Ensembled ViTs boost skin-lesion coverage to 90.4%","Conformal ensemble of ViTs lifts skin coverage by 10%","Domain-adaptive conformal ViTs raise prediction coverage","CE-ViTs: Ensemble transformers beat single-model lesion coverage","Pooled vision transformers improve lesion set reliability"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 90.38 percent coverage number is only a validity guarantee if the pooled test images and pooled calibration images come from the same mixture of sources and the seven lesion classes are labeled consistently across HAM10000, Dermofit, and ISIC; changing the source mixture or adding a new source would invalidate the guarantee.","fun_headline_variants_meta":{"raw":{"variants":["Ensembled ViTs boost skin-lesion coverage to 90.4%","Conformal ensemble of ViTs lifts skin coverage by 10%","Domain-adaptive conformal ViTs raise prediction coverage","CE-ViTs: Ensemble transformers beat single-model lesion coverage","Pooled vision transformers improve lesion set reliability"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1280,"prompt_tokens":960,"completion_tokens":320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":235}},"tokens_in":576,"tokens_out":320,"duration_ms":3630,"temperature":1.0,"reasoning_tokens":235,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:08:28.870449+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Calibrate on the pooled calibration split as the paper does, then evaluate coverage separately on each dataset's test split; if the per-source coverage on the most domain-shifted source falls far below the nominal level, the pooled coverage figure does not hold under the domain shift the paper claims to address.","supporting_citations":[{"cited_title":"Skin cancer mnist: Ham10000,","cited_arxiv_id":null,"evidence_quote":"Supplies the HAM10000 skin-lesion images used for training one expert model and for merged calibration."},{"cited_title":"Skin cancer isic,","cited_arxiv_id":null,"evidence_quote":"Supplies the Skin Cancer ISIC images used for training one expert model and for merged calibration."},{"cited_title":"Pre-trained model,","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained ViT weights and fine-tuning recipe for all three expert models."},{"cited_title":"Domain adaptive ensemble learning,","cited_arxiv_id":null,"evidence_quote":"Supplies the domain-adaptive ensemble learning mechanism that motivates averaging predictions across datasets."},{"cited_title":"Empirical Validation of Conformal Prediction for Trustworthy Skin Lesions Classification","cited_arxiv_id":"2312.07460","evidence_quote":"Provides prior empirical validation of conformal prediction on skin lesion classification, motivating the safety-critical framing."}],"review_version":1}