{"id":"531c3893-af39-4a0f-aeee-785cfc079ef6","arxiv_id":"2509.04816","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Unmodified mixtures of experts can provide predictive uncertainty estimates via entropy, mutual information, and expert-variance, and these estimates outperform a two-expert average ensemble on conditional correctness metrics under out-of-distribution data.","lead":"This paper shows that predictive uncertainty for semantic segmentation can be read off from a mixture-of-experts (MoE) model without retraining or changing its architecture. It reports that such MoE uncertainty estimates beat a simple two-model ensemble on out-of-distribution traffic images, though standard calibration metrics and a Cityscapes scaling experiment tell a more mixed story.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed MoE superiority over ensembles is not established: the ensemble baseline is a uniform average of two domain-specialized experts, and the conditional-correctness metrics are not controlled for the MoE's much higher segmentation accuracy.","rationale":"The most load-bearing concern is the weak ensemble baseline and the resulting accuracy confound in the conditional-correctness metrics. The comparison ensemble is a uniform average of two domain-specialized experts, not a deep ensemble with independently trained diverse members, and it has substantially lower segmentation accuracy than the MoE. Since the headline metrics score the alignment between uncertainty and the model's own correctness, the MoE's advantage on those metrics may simply reflect its better predictions rather than better uncertainty estimation. The paper's explicit note in Section 4.2 and its own Cityscapes results support treating this as an open question rather than a settled advantage. The reader's weakest assumption identifies the same baseline problem, so I agree with the conditional verdict. A strong deep-ensemble comparison and reported error bars could rescue the claim, but without them the abstract's general statement that MoEs yield more reliable uncertainty estimates than ensembles is overstated.","tokens_in":12705,"tokens_out":8633,"duration_ms":77087,"concrete_test":"Train a proper deep ensemble of three DeepLabv3+ models with identical architecture, each randomly initialized and trained on the full combined highway-urban A2D2 training set (matching the MoE's total training budget). Compute PE, MI, and EV from this ensemble and recalculate p(accurate|certain), p(uncertain|inaccurate), PAvPU, and AU-PAvPU on the ambiguous A2D2 subset and on corrupted A2D2 test images. If the deep ensemble matches or beats the MoE on these conditional-correctness metrics, the paper's claim that MoEs provide more reliable uncertainty estimates than ensembles is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that MoEs provide more reliable uncertainty estimates than ensembles under OOD depends on Section 4.2's ensemble baseline, which is a uniform average of the highway and urban experts. The section itself concedes that this is not a deep ensemble in the sense of Lakshminarayanan et al. [25]. Because the two experts were trained on semantically disjoint splits, this averaged predictor reaches only 0.342 mIoU on the A2D2 test set while the MoE reaches 0.429-0.444 (Fig. 6). The headline conditional metrics (p(accurate|certain), p(uncertain|inaccurate), PAvPU, and AU-PAvPU in Fig. 7 and Table 2) score how well an uncertainty map tracks the model's own errors, so a model with substantially higher segmentation accuracy can achieve better values on these metrics even if its uncertainty estimates are not intrinsically better. The stacked PE/MI construction (Eq. 2) further injects the MoE output pMoE as an additional 'expert' in the averaged predictive distribution, which the authors note 'may, however, artificially reduce perceived variance'; this makes the comparison with the ensemble's PE/MI, which averages only the two expert outputs, non-equivalent. Finally, the paper's own Cityscapes experiments (Table 3) show ensembles matching or beating MoEs on calibration, indicating the claimed advantage is not a general property of MoEs but is specific to this A2D2 configuration. No error bars or seed variance are reported, so it is unclear whether any of the differences exceed run-to-run noise. The load-bearing assumption is therefore that the uniform two-expert average is a legitimate ensemble baseline and that the conditional-metric advantage is not an artifact of accuracy differences or the stacked formulation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies extraction of predictive uncertainty from model-level mixtures of experts (MoEs) for semantic segmentation. Three estimators are considered: predictive entropy (PE), mutual information (MI), and a proposed expert variance (EV), computed from the gate-weighted combination of pretrained expert outputs. Routing uncertainty is measured by gate entropy. Experiments on A2D2 with two semantically disjoint experts compare MoE variants against a two-model averaging ensemble and MC dropout baselines; additional Cityscapes experiments vary the number of experts. The main claimed finding is that MoEs provide more reliable uncertainty estimates than ensembles in conditional-correctness metrics under out-of-distribution data, without architectural modification. The paper also finds that simple gates give better-calibrated routing uncertainty than classwise gates, and that scaling experts on Cityscapes gives modest NLL improvement but no consistent calibration gain.","tokens_in":13010,"tokens_out":5128,"duration_ms":42594,"significance":"If the central claim were established, the paper would offer a practical recipe: an unmodified, already-trained MoE can yield uncertainty maps that track its own errors better than a uniform two-expert ensemble, at no extra inference cost beyond the MoE's own forward pass. The paper is honest about several limitations: the ensemble baseline is not a deep ensemble (Section 4.2), the stacked PE/MI construction includes the MoE output as an extra component (Section 3.2), and the Cityscapes results show no consistent MoE advantage (Section 5.2). Nevertheless, the main comparative claim is currently supported only in a narrow, accuracy-confounded setting and against a weak baseline; the contribution is therefore incremental rather than definitive, and the paper would need substantial additional experiments to justify the abstract's general wording. The release of code and the evaluation on two datasets are strengths, as is the explicit acknowledgment of the degeneracy of 100% peak values in Table 2.","major_comments":[{"comment":"The ensemble baseline is the uniform average of the highway and urban experts, which the paper itself notes is not a deep ensemble in the sense of Lakshminarayanan et al. [25]. Because these experts were trained on semantically disjoint subsets, the ensemble reaches only 0.342 mIoU on A2D2 while the MoE reaches 0.429–0.444 (Fig. 6). The conditional-correctness metrics in Fig. 7 and Table 2 score how well an uncertainty map tracks the model's own errors, so a model with substantially higher segmentation accuracy can achieve better values even if its uncertainty estimates are not intrinsically better. The claim that MoEs yield more reliable uncertainty estimates than ensembles is therefore not supported by this comparison. The paper should add a proper deep ensemble baseline trained on the same data with random initializations, or otherwise control for the accuracy difference (e.g., by reporting normalized or accuracy-matched conditional metrics).","section":"§4.2, Fig. 6, Fig. 7, Table 2"},{"comment":"The stacked approach to computing PE and MI adds the MoE output pMoE as an extra component in the averaged predictive distribution, giving the MoE N+1 terms while the ensemble averages only N expert outputs. This makes the PE/MI values non-comparable across the two models, and the paper acknowledges that including the MoE output 'may, however, artificially reduce perceived variance.' Since the weighted formulation (which has N components) is only applicable to simple gates, the comparison in Fig. 7 and Table 2 is not symmetric across model types. The paper should either restrict the headline comparison to the weighted formulation where possible or provide a principled justification for why the stacked formulation is a fair equivalent of the ensemble estimate.","section":"§3.2, Eq. (2)"},{"comment":"The 100% peak values in Table 2 are flagged as degenerate ('Note that a 100 % peak occurs when there are very few inaccurate or very few certain pixels at the respective threshold'), yet these peaks are presented as evidence that MoEs achieve the 'highest peak values' for conditional metrics and the AU-PAvPU score. No error bars or seed variance are reported anywhere in the paper, and the threshold-dependence of these metrics is not analyzed. Consequently, the claimed superiority in conditional correctness under data shift is not statistically supported. The paper should report variance over multiple training seeds and preferably provide confidence intervals or significance tests for the AU-PAvPU differences.","section":"§4.4, Table 2"},{"comment":"The Cityscapes experiments directly contradict the abstract's sweeping claim that MoEs yield more reliable uncertainty estimates than ensembles. Table 3 shows that ensembles match or outperform MoEs on ECE, MCE, and Brier score, and the paper states that 'MoEs have slightly worse mIoU and worse calibration metrics overall.' The conclusion nonetheless repeats that MoEs produce 'competitive and often superior uncertainty estimates compared to standard ensembles.' This overgeneralization needs to be corrected: the advantage observed in conditional-correctness metrics is specific to the A2D2 semantically disjoint expert configuration, and the paper should explicitly state the boundary conditions under which the claimed benefit transfers.","section":"§5.2, Table 3, §6"}],"minor_comments":[{"comment":"In the final paragraph, 'interoperability' appears to be a typo for 'interpretability' in the phrase 'improve uncertainty robustness and interoperability.'","section":"§6"},{"comment":"'Semantical split' is used in both the abstract and Section 4.1; the standard term is 'semantic split.'","section":"§4.1 and Abstract"},{"comment":"The asterisk markers for the 100% values are not explained in the table caption; the explanation appears only in the body text. Add a note to the caption to avoid ambiguity.","section":"Table 2"},{"comment":"EV is defined per class c, but Figures 7 and 8 appear to treat EV as a per-pixel scalar. Please clarify how the class-wise variances are aggregated to produce the reported uncertainty maps and metric values.","section":"§3.2, Eq. (3)"},{"comment":"The selection of the classwise gate without the additional convolutional layer for the data-shift experiments is justified by 'consistently showed strong calibration on OOD data (see Figure 4),' but Figure 4 does not show this variant dominating all calibration metrics (e.g., MCE). Please clarify the selection criterion.","section":"§4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable applied contribution, but the central comparative claim is currently undermined by the weak ensemble baseline and the lack of error bars. The authors should be encouraged to run a genuine deep ensemble baseline and report seed variance; if these experiments confirm the advantage, the paper could be publishable. The reliance on the authors' own previous works for the A2D2 split and trained models is transparent and acceptable, though the novelty relative to those works should be highlighted more clearly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful part of this paper is the engineering: it takes predictive entropy, mutual information, and a simple expert-variance measure, and shows how to compute them from an unmodified model-level Mixture of Experts for semantic segmentation. The stacked and weighted formulations for PE and MI are sensible adaptations, and the authors are unusually candid about their limitations. They explicitly note that their ensemble is not a deep ensemble, flag the degenerate 100% peaks in Table 2, and report Cityscapes results where ensembles match or beat MoEs on calibration. Code is public. That is a fair empirical study, and it deserves a serious referee rather than a desk reject.\n\nThe soft spots are real but not fatal if the claims are reined in. The headline advantage over ensembles on OOD rests on a weak baseline: an average of two domain-specialized experts, not a deep ensemble with shared-data random initialization. On A2D2, the MoE hits 0.43–0.44 mIoU versus 0.34 for that averaged ensemble, so the conditional correctness metrics (p(accurate|certain), p(uncertain|inaccurate), PAvPU) are not controlled for segmentation accuracy. A better model can score higher on those metrics even if its uncertainty is not intrinsically sharper. The stacked formulation also injects the MoE output into the averaged predictive distribution, which the authors themselves say may artificially reduce variance, making the comparison with the ensemble's PE/MI non-equivalent. There are no error bars or seed variance reported, so I cannot tell whether the remaining differences exceed run-to-run noise. Their own Cityscapes data shows the advantage is not a general property of MoEs.\n\nWho should read it: people working on efficient uncertainty estimation for segmentation, or on MoE-based perception, will get a careful empirical reference and a starting point for a stronger baseline comparison. With a proper deep ensemble baseline, error bars, and a qualified conclusion, it would be a solid paper. As is, the central claim is overstated.\n\nRecommendation: send to peer review, but as major revision: add a real deep ensemble baseline, report variance across seeds, and either remove or sharply qualify the claim that MoEs produce more reliable uncertainty than ensembles.","headline":"The paper's new contribution is the adaptation of standard ensemble uncertainty metrics to unmodified model-level MoEs, with code and transparent caveats, but the headline claim that MoEs beat ensembles on OOD is not supported by the weak baseline and accuracy-metric confound.","tokens_in":13579,"tokens_out":1921,"would_cite":true,"duration_ms":18881,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A mixture of experts can produce well-calibrated predictive uncertainty in semantic segmentation without architectural modifications.","keywords":["mixture of experts","predictive uncertainty","semantic segmentation","uncertainty calibration","out-of-distribution detection","gate entropy","expert variance","autonomous driving"],"falsifier":"Train a deep ensemble with independent initializations on the combined highway-urban training data and compare p(uncertain|inaccurate) and AU-PAvPU on the ambiguous A2D2 test split and on corrupted Cityscapes; if the MoE no longer outperforms that ensemble on the conditional metrics, the central claim fails. Independently, run the same MoE uncertainty extraction on a second OOD set, such as night or rain images, to see whether the advantage transfers or is specific to the ambiguous A2D2 split.","tokens_in":12525,"feed_emoji":"🚗","tokens_out":8226,"duration_ms":64313,"temperature":0.7,"pith_summary":"This paper tries to establish that a mixture-of-experts (MoE) model trained for semantic segmentation of traffic scenes already contains what is needed for reliable predictive uncertainty, and that extracting it requires no architectural changes. Taking the MoE's gate-weighted probability output as an aggregated predictive distribution, the authors compute predictive entropy, mutual information, and a new expert-variance score, producing per-pixel uncertainty maps. On a highway/urban split of a driving dataset, these MoE-based uncertainty estimates outperform a two-expert averaged ensemble and a Monte-Carlo dropout baseline on conditional correctness metrics, especially on out-of-distribution and corrupted inputs. The paper also finds that simple gates give better-calibrated routing uncertainty than classwise gates, and that increasing the number of experts on Cityscapes slightly improves calibration. If right, this means safety-critical perception can obtain calibrated uncertainty per pixel from a single unmodified model instead of paying the training and inference cost of an ensemble.","feed_headline":"MoE uncertainty beats ensemble uncertainty on shifted driving data","feed_subtitle":"No architectural changes needed: an unmodified two-expert model flags wrong pixels better than an averaged ensemble.","key_machinery":"The load-bearing object is the gate-weighted predictive distribution of a model-level mixture of experts, $\\bar{p}(c|x)=\\sum_{e=1}^{N} w_e(x)\\,p_e(c|x)$, where $w_e(x)$ is the input-dependent gate weight for expert $e$ and $p_e(c|x)$ is that expert's per-class probability. From this one distribution the paper derives predictive entropy, mutual information, and a proposed expert-variance score $\\mathrm{EV}(x,c)=\\frac{1}{N}\\sum_e (p_e(c|x)-\\bar{p}(c|x))^2$, without modifying the trained architecture. Gate entropy, computed on the routing distribution over experts, forms a separate uncertainty channel. The argument's load is carried by the fact that the gate's input-dependent weights already encode where each expert is competent, so weighted expert disagreement becomes a usable uncertainty signal.","core_discovery":"The central discovery is that the gating mechanism of a model-level MoE, normally used only to improve segmentation output, doubles as an uncertainty estimator. For each input, the authors form the aggregate predictive distribution $\\bar{p}(c|x)=\\sum_e w_e(x)p_e(c|x)$ from gate weights and expert probabilities, then derive three uncertainty scores: the Shannon entropy of $\\bar{p}$, the mutual information between $\\bar{p}$ and the individual experts' predictions, and the variance of the experts around the MoE output. On out-of-distribution A2D2 data and under increasing corruption severity, these MoE uncertainty maps align with misclassification better than the averaged ensemble and the Monte-Carlo dropout baseline on conditional correctness metrics such as $p(\\mathrm{accurate}|\\mathrm{certain})$, $p(\\mathrm{uncertain}|\\mathrm{inaccurate})$, and Patch Accuracy vs Patch Uncertainty (PAvPU). The paper also reports that simple gate architectures give better-calibrated routing uncertainty (gate entropy) than classwise gates, and that scaling to 10 experts on Cityscapes improves negative log-likelihood without consistent gains on other calibration metrics.","pith_inferences":["If the MoE advantage is real, then uncertainty estimation becomes a by-product of expert diversity: any collection of pretrained specialists can be fused by a trained gate and immediately yield uncertainty maps, an option the paper does not compare against a full deep ensemble.","The metric split the paper observes, with conditional correctness favoring MoEs while ECE and Brier often favor ensembles, suggests that calibration error and decision-useful uncertainty are different quantities; safety evaluations should report both.","A direct test of the paper's thesis would be to train a deep ensemble with independently initialized members on the same combined data and compare AU-PAvPU; the paper's ensemble baseline is only an average of the two domain experts, so the main claim is not yet tested against the strongest ensemble baseline.","Gate entropy's near-identical distribution on in-distribution and out-of-distribution data hints that routing confidence is not inherently distribution-aware; an extension would be to feed gate entropy into a supervised OOD detector rather than using it as a standalone signal."],"forward_implications":["An unmodified MoE can emit a calibrated per-pixel uncertainty map alongside its segmentation prediction, so downstream systems can threshold on uncertainty without extra forward passes or retraining.","MoE uncertainty can serve as an out-of-distribution or anomaly cue: the same ambiguous A2D2 scenes that decrease segmentation accuracy produce higher uncertainty under the MoE estimates.","Routing uncertainty and prediction uncertainty can be reported as separate numbers, giving a two-channel view of whether the model is unsure about which expert should act (gate entropy) or about what the output should be (predictive entropy).","Adding more experts to an MoE is a plausible but modest lever for uncertainty calibration, improving NLL in the paper's Cityscapes experiments even when segmentation accuracy plateaus."],"supporting_citations":[{"why":"Supplies the MoE architecture, gate variants, and the highway/urban A2D2 split that the experiments are built on.","marker":"[32]"},{"why":"Earlier result that these MoEs beat single models and ensembles on segmentation accuracy, motivating the uncertainty study.","marker":"[33]"},{"why":"The source of the predictive-entropy and mutual-information extraction methods, adapted from deep ensembles to MoEs.","marker":"[25]"},{"why":"Defines the conditional correctness metrics p(accurate|certain), p(uncertain|inaccurate), and PAvPU used to score the uncertainty maps.","marker":"[27]"},{"why":"The A2D2 dataset, including the ambiguous subset treated as out-of-distribution data.","marker":"[12]"},{"why":"The Monte-Carlo dropout baseline whose stochastic forward passes are compared against MoE uncertainty.","marker":"[10]"},{"why":"The data-shift evaluation protocol and calibration metrics (ECE, MCE, NLL, Brier) used under corruption.","marker":"[39]"},{"why":"The Cityscapes dataset used for experiments with 2, 5, and 10 experts.","marker":"[6]"},{"why":"The DeepLabv3+ architecture used as each expert network.","marker":"[5]"}],"fun_headline_variants":["MoE uncertainty outperforms ensembles on shifted driving scenes","No architectural tweaks: MoEs yield well-calibrated uncertainty for segmentation","Mixture-of-experts uncertainty: better than ensembles on OOD data","MoE-based uncertainty beats ensembles on out-of-distribution driving data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central comparison is an ensemble that merely averages the same two domain-specialized experts, not a deep ensemble with independently trained members; on top of that, the out-of-distribution results rely on one ambiguous A2D2 split standing in for out-of-distribution data.","fun_headline_variants_meta":{"raw":{"variants":["MoE uncertainty outperforms ensembles on shifted driving scenes","No architectural tweaks: MoEs yield well-calibrated uncertainty for segmentation","Mixture-of-experts uncertainty: better than ensembles on OOD data","MoE-based uncertainty beats ensembles on out-of-distribution driving data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000705,"raw_usage":{"total_tokens":3207,"prompt_tokens":1003,"completion_tokens":2204,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":2129}},"tokens_in":619,"tokens_out":2204,"duration_ms":14364,"temperature":1.0,"reasoning_tokens":2129,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:26:43.496668+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a deep ensemble with independent initializations on the combined highway-urban training data and compare p(uncertain|inaccurate) and AU-PAvPU on the ambiguous A2D2 test split and on corrupted Cityscapes; if the MoE no longer outperforms that ensemble on the conditional metrics, the central claim fails. Independently, run the same MoE uncertainty extraction on a second OOD set, such as night or rain images, to see whether the advantage transfers or is specific to the ambiguous A2D2 split.","supporting_citations":[{"cited_title":"Mar- ius Z¨ollner","cited_arxiv_id":null,"evidence_quote":"Supplies the MoE architecture, gate variants, and the highway/urban A2D2 split that the experiments are built on."},{"cited_title":"Evaluating mixture-of-experts architectures for net- work aggregation","cited_arxiv_id":null,"evidence_quote":"Earlier result that these MoEs beat single models and ensembles on segmentation accuracy, motivating the uncertainty study."},{"cited_title":"Simple and scalable predictive uncertainty esti- mation using deep ensembles","cited_arxiv_id":null,"evidence_quote":"The source of the predictive-entropy and mutual-information extraction methods, adapted from deep ensembles to MoEs."},{"cited_title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning","cited_arxiv_id":null,"evidence_quote":"The Monte-Carlo dropout baseline whose stochastic forward passes are compared against MoE uncertainty."},{"cited_title":"Sculley, Joshua V","cited_arxiv_id":null,"evidence_quote":"The data-shift evaluation protocol and calibration metrics (ECE, MCE, NLL, Brier) used under corruption."},{"cited_title":"The cityscapes dataset for semantic urban scene understanding","cited_arxiv_id":null,"evidence_quote":"The Cityscapes dataset used for experiments with 2, 5, and 10 experts."},{"cited_title":"Encoder-decoder with atrous separable convolution for semantic image segmentation","cited_arxiv_id":null,"evidence_quote":"The DeepLabv3+ architecture used as each expert network."}],"review_version":2}