{"id":"cfbe246d-51a1-4284-8271-76a58245b788","arxiv_id":"2602.08470","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An ensemble trained with distributionally robust optimization at several reweighting intensities yields credal predictions whose uncertainty better separates in-distribution from out-of-distribution samples.","lead":"CreDRO trains a set of neural networks, each using a different amount of worst-case data reweighting, then combines their outputs into probability intervals to measure uncertainty. In tests, this distribution-shift-aware ensemble improves out-of-distribution detection and selective prediction over several existing uncertainty methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No control experiment isolates the δ-interpolation mechanism: gains may come from fixed-level DRO training rather than from spanning a range of i.i.d. relaxations.","rationale":"The reader's weakest assumption concerned the faithfulness of the top-δ truncation heuristic and the coverage of the δ-grid. I agree that the δ-grid's role is underdetermined, but I locate the more concrete problem in the experimental design rather than in the absence of a divergence-link: the paper never includes a fixed-δ ensemble as a control. Because EN-DRO uses the same interpolation as CreDRO, the comparison against DE shows that 'some DRO-style training plus spread' beats random initialization, but it does not identify the spread as the cause. The δG ablation in §4.3 also varies only the lower endpoint while always keeping the spread to 1.0; it cannot detect whether the spread matters. Given that individual members improve in accuracy under DRO, a plausible alternative explanation is that top-δ truncation acts as a regularizer that improves each member, and the ensemble's better OOD detection is a byproduct of better ID calibration rather than of disagreement about distribution shifts. This is a testable, non-adversarial concern, and it does not change the reader's conditional verdict: the paper is promising but needs the missing control to support its strongest claim. I therefore recommend UNCHANGED, with the concrete test above as the required condition for accepting the mechanism.","tokens_in":27178,"tokens_out":10394,"duration_ms":116699,"concrete_test":"Re-run the CIFAR10 OOD detection experiment (M=20, ResNet18, δG=0.5, same schedule as §4.1) with the only change being δ_i = 0.5 for every ensemble member instead of Eq. (6). Compare AUROC on SVHN, Places365, CIFAR100, FMNIST, and ImageNet against Table 1. If the fixed-δ ensemble's scores fall within one standard deviation of CreDRO's (97.4/92.7/92.5/96.4/91.1), the uniform δ-interpolation is not the driver of the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CreDRO's epistemic uncertainty stems from training ensemble members under *different* degrees of i.i.d.-assumption relaxation via the δ_i grid in Eq. (6). The experiments never compare CreDRO against an ensemble in which every member uses the same δ_i while keeping architecture, training schedule, and credal construction identical. EN-DRO, described as 'an ensemble trained with our DRO framework in Algorithm 1', also uses the δ_i interpolation, so the DE-vs-EN-DRO comparison conflates 'DRO training' with 'spread of DRO levels'. CreDE uses a fixed DRO level but changes the architecture (double output head) and loss, so it is not a control either. Table 4 varies the lower endpoint δG but always retains the spread to 1.0, so it cannot show that the spread itself matters. Meanwhile Table 7 shows DRO members are individually more accurate (≈95.4% vs ≈94.3% for CreWra), so the AUROC gains could be explained by better per-member calibration/accuracy rather than by shift-meaningful disagreement. If a fixed-δ ensemble performs as well as CreDRO, the central claim reduces to 'hard-example reweighting improves members', and the 'varying i.i.d. relaxation' mechanism in Eq. (6) is not the source of the reported EU quality. A theoretical CVaR/density-ratio interpretation of a single δ is available, but it does not establish that the grid is what matters. The absence of a fixed-δ control is the load-bearing gap.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CreDRO, a deep-ensemble method in which each member is trained under a different degree of i.i.d.-assumption relaxation, implemented via batch-wise truncation to the top-δ high-loss samples. The resulting softmax predictions are converted to a box credal set, and epistemic uncertainty is quantified as the upper-minus-lower entropy of that set. The authors claim that CreDRO captures distribution-shift-related epistemic uncertainty, and they report consistent gains over deep-ensemble and state-of-the-art credal baselines across OOD detection benchmarks and a medical selective-classification task.","tokens_in":27491,"tokens_out":6569,"duration_ms":71046,"significance":"The central idea—defining epistemic uncertainty as disagreement among models trained under different plausible train–test shift scenarios rather than only under random initialization—is timely and conceptually appealing. If the mechanism is validated, the paper would offer a practical way to inject distribution-shift awareness into credal ensembles with no architectural change. The experimental effort is extensive: multiple OOD benchmarks, corrupted-data tests, selective classification on Camelyon17, and ablations for ensemble size, δG, credal-set construction, and member accuracy. The plan to release code is a positive for reproducibility. However, the paper's central claim about the source of the EU quality is not yet isolated experimentally, and the interpretation of δ as a level of distribution shift is heuristic rather than derived.","major_comments":[{"comment":"The paper attributes CreDRO's EU quality to 'disagreement among models trained with varying relaxations of the i.i.d. assumption' (Abstract; §3.1). No experiment isolates the variation mechanism. EN-DRO also uses the δ_i grid of Eq. (6), so the DE-vs-EN-DRO comparison does not control for the spread; CreDE uses a fixed DRO level but changes the architecture and loss. The δG ablation (Table 4) always keeps the spread to 1.0, so it cannot reveal whether the spread matters. Table 7 shows per-member accuracy improves from ≈94.3% (CreWra) to ≈95.4% (CreDRO), so the AUROC gains could plausibly be explained by better per-member predictions. I request an ablation in which all M members train with the same δ (e.g., δ=0.5, 0.75, 0.9), while keeping architecture, schedule, and credal construction identical. If a fixed-δ ensemble matches CreDRO, the central claim reduces to 'hard-example reweighting","section":"§3.1, Eq. (6); Table 1; Table 4; Table 7"},{"comment":"The top-δ truncation is presented as a 'flexible approximation' of adversarially reweighted learning, but no derivation connects the truncation to a distribution-shift neighborhood in Eq. (2) or (4). The interpretation of δ_i as a 'degree of relaxation of the i.i.d. assumption' is therefore a heuristic. This matters because the conceptual contribution is precisely that the ensemble disagreement reflects distribution-shift uncertainty. Please either (a) provide a formal characterization of the induced reweighting (e.g., as a CVaR/divergence-ball maximization with a known δ), or (b) soften the causal claim to describe the method as an empirically useful heuristic. The fixed-δ control from the previous comment would also help distinguish the effect of varying δ from the effect of DRO training per se.","section":"§3.1, Eq. (4)–(6)"}],"minor_comments":[{"comment":"Clarify the hyperparameter selection for CreRL and CreEns. 'Report their best results' is too vague; state whether α was selected on a validation split and provide the sensitivity of the baselines to α in an appendix or table.","section":"§4.1"},{"comment":"CreDE is missing for CIFAR100 vs CIFAR100-C; the explanation is in Appendix C but should be summarized in the caption or main text so the reader is not left puzzled by the absent curve.","section":"Figure 4 / §4.5"},{"comment":"The last step says 'Minimize' but does not explicitly state that this defines the per-batch loss for an optimizer update; please clarify notation and connect Step 5 to the standard SGD/backprop update.","section":"Algorithm 1"},{"comment":"Some advantages are within 1–2 standard deviations (e.g., Places: CreDRO 92.7±0.1 vs CreRL 91.8±0.2); consider reporting paired significance tests or effect sizes, and specify whether runs share seeds across methods for fairer comparisons.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test note's main concern is well founded: the paper's central mechanism—the δ_i spread—is not isolated by any control, and the current experiments could be explained by per-member accuracy gains alone. The δG ablation is insufficient because it always retains a spread to 1.0. I recommend requiring the fixed-δ ablation before publication. The paper is otherwise solid and the empirical scope is unusually broad. Also, the baseline hyperparameter selection ('best results') should be made explicit; this is fixable and less central."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"CreDRO is a simple, well-engineered recipe: train each ensemble member with a different top-δ truncation of high-loss samples, then take min/max softmax bounds into a box credal set. The OOD detection and selective classification results are consistently better than strong baselines across several benchmarks, and the ablations (ensemble size, δG, box vs convex hull) are reasonably thorough. The idea that EU should reflect disagreement under varying relaxations of the i.i.d. assumption is a genuinely useful reframing, and the per-member δ grid is new relative to CreDE's single fixed DRO level.\n\nThe soft spot is the one the stress-test identifies, and I think it lands. There is no control experiment where every member is trained with the same δ but otherwise identical setup (same architecture, loss, credal construction). EN-DRO also uses the δ grid, CreDE changes the architecture and loss, and Table 4 varies δG while keeping the spread to 1.0. So we don't actually know whether the spread of δ_i is what drives the gains, or whether it's just that each member is trained with hard-example reweighting, which individually improves accuracy and calibration (Table 7 shows DRO members at ~95.4% vs ~94.3% for CreWra). If a fixed-δ ensemble performs the same, the paper's central claim—that EU captures shift-meaningful disagreement—is not supported; it would reduce to 'hard-example reweighting improves members.' The lack of any theoretical link between top-δ truncation and a divergence measure makes this gap load-bearing, though not fatal.\n\nMinor issues: no code in the preprint (promised on publication), no significance tests, CreRL/CreEns reported at their best α, and CreDE results missing on CIFAR100-C with a somewhat hand-wavy explanation. The self-citations are heavy but not abusive; they're mostly to the authors' own credal framework, and CreDRO doesn't reduce to those prior results.\n\nWho this is for: people working on epistemic uncertainty, selective classification, OOD detection. It's an empirical paper with a clean recipe and consistently favorable results; the central mechanism just isn't isolated yet. A serious referee should engage; the fixed-δ control is the one experiment that would clarify the claim. I'd send it to review, and I'd expect revision rather than rejection.","headline":"A useful empirical recipe for credal ensembles, but the central mechanism — the spread of δ_i — is never isolated by a fixed-δ control experiment.","tokens_in":28013,"tokens_out":1437,"would_cite":true,"duration_ms":15901,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CreDRO trains an ensemble with varied degrees of i.i.d. relaxation so epistemic uncertainty reflects potential train–test shifts, not just random initialization.","keywords":["epistemic uncertainty","credal sets","distributionally robust optimization","deep ensembles","out-of-distribution detection","selective classification","probability intervals","ensemble disagreement"],"falsifier":"Run CreDRO with the δ_i assignment shuffled across members (or reversed) while keeping the same multiset of δ values; if OOD detection AUROC is unchanged, the explanatory story—ordered degrees of i.i.d. relaxation—is not what drives the gains. Additionally, train a variant that replaces top-δ truncation with exact sample reweighting from an explicit divergence ball; large performance differences would show the heuristic, not DRO, is responsible.","tokens_in":27050,"feed_emoji":"🎯","tokens_out":4657,"duration_ms":40300,"temperature":0.7,"pith_summary":"The paper argues that epistemic uncertainty in deep learning should come from disagreement about how much the training and test distributions may differ, not mainly from which random initialization a model happened to get. To do this it proposes CreDRO, an ensemble in which each member is trained with a different degree of distributionally robust optimization: member i keeps only the top δ_i highest-loss samples in each batch, interpolating from a worst-case δ_G to full empirical risk minimization. At inference, the members' softmax outputs define a box credal set—a convex set of probability vectors bounded by per-class min/max probabilities—and epistemic uncertainty is measured by the entropy gap across that set. On out-of-distribution detection and selective classification in histopathology, the paper reports that CreDRO outperforms existing credal classifiers and deep ensembles.","feed_headline":"Distribution shift becomes the source of epistemic uncertainty","feed_subtitle":"CreDRO spreads an ensemble across shift severities, improving OOD detection and selective classification.","key_machinery":"The load-bearing object is the truncation schedule δ_i = (1−δ_G)/(M−1)·(i−1)+δ_G applied inside the adversarially reweighted learning objective. For each batch, samples are sorted by loss and only the top δ_i fraction is backpropagated; this is the paper's pragmatic stand-in for group DRO, with δ parameterizing the degree of assumed train–test divergence. The second ingredient is the box credal set K_B = {p : p_k ∈ [min_i p_{i,k}, max_i p_{i,k}], Σ p_k = 1}, built from the ensemble's softmax outputs. Epistemic uncertainty is then the difference H(K_B) − H(K_B), computable by two constrained entropy optimizations whose cost depends on the number of classes, not the ensemble size.","core_discovery":"The central claim is that disagreement between models trained under different relaxations of the i.i.d. assumption is a more meaningful source of epistemic uncertainty than disagreement from random initialization. The paper formalizes this by assigning each ensemble member a different truncation fraction δ_i in an adversarially reweighted DRO objective: during each batch, only the top δ_i fraction of samples by loss are used for backpropagation, which implicitly upweights hard-to-learn examples and simulates a hypothesized distribution shift. The δ_i are spread uniformly from a user-chosen worst-case δ_G (default 0.5) to 1 (vanilla ERM). The resulting softmax probabilities are converted to a","pith_inferences":["If the mechanism is really the ordering of members by shift sensitivity, then a direct test is to shuffle or reverse the δ_i assignment across members: the OOD detection gain should degrade, since the ensemble would no longer span meaningful shift levels.","The less competitive performance of a closely related method that uses a single fixed DRO parameter suggests that the diversity of δ values, not DRO alone, is what matters; a reader could isolate this by training a single DRO model and perturbing it with different top-δ schedules.","A natural extension the paper only sketches is regression, where Gaussian predictions would form a credal set of mixtures; one could test whether the same shift-derived disagreement transfers to continuous targets.","Because the paper offers no theoretical link between δ and a conventional divergence measure, the interpretation of δ as a 'degree of distribution shift' remains a heuristic; connecting δ to, say, a chi-square or KL ball would make the method's semantics testable."],"forward_implications":["Epistemic uncertainty now reflects potential distribution shift, so OOD and shifted-domain samples receive higher uncertainty, improving out-of-distribution detection when uncertainty is used as the score.","Selective classification in a realistic medical domain shift (histopathology from unseen scanners) improves, enabling safer abstention based on shift-induced disagreement.","The method requires no change to network architecture—only per-member top-δ loss truncation during training—so it can wrap existing deep ensembles and loss functions.","Performance is stable across the choice of δ_G because the ensemble spans a range of δ values; smaller δ_G produces wider probability intervals, consistent with more pessimistic shift assumptions.","DRO-trained ensembles alone, without credal post-processing, also outperform standard deep ensembles, locating part of the benefit in the training procedure rather than in the credal representation."],"fun_headline_variants":["Uncertainty from distribution shift, not just random seeds","CreDRO: Ensembles that disagree on distribution shifts","Rethinking epistemic uncertainty: shift-based disagreement","DRO ensemble turns shift into uncertainty signal","Epistemic uncertainty from shift, not seed luck"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The whole construction rests on the assumption that keeping the top-δ highest-loss samples in a batch behaves like a true DRO reweighting and that the uniform grid of δ values covers the plausible train–test shifts; if either fails, the ensemble diversity is arbitrary and the epistemic uncertainty is not really about distribution shift.","fun_headline_variants_meta":{"raw":{"variants":["Uncertainty from distribution shift, not just random seeds","CreDRO: Ensembles that disagree on distribution shifts","Rethinking epistemic uncertainty: shift-based disagreement","DRO ensemble turns shift into uncertainty signal","Epistemic uncertainty from shift, not seed luck"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1078,"prompt_tokens":684,"completion_tokens":394,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":320}},"tokens_in":428,"tokens_out":394,"duration_ms":4720,"temperature":1.0,"reasoning_tokens":320,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T03:15:16.725658+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CreDRO with the δ_i assignment shuffled across members (or reversed) while keeping the same multiset of δ values; if OOD detection AUROC is unchanged, the explanatory story—ordered degrees of i.i.d. relaxation—is not what drives the gains. Additionally, train a variant that replaces top-δ truncation with exact sample reweighting from an explicit divergence ball; large performance differences would show the heuristic, not DRO, is responsible.","supporting_citations":[],"review_version":1}