{"id":"d51b388b-324c-450b-ad6a-700035e9cec4","arxiv_id":"2507.11574","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CMCO attaches conformalized Monte Carlo dropout intervals to DeepONet field predictions, achieving high average coverage but severe undercoverage in some test samples.","lead":"The authors combine Monte Carlo dropout with split conformal prediction inside a DeepONet to attach uncertainty intervals to neural operator predictions of physical fields. They test the method on turbulent flow, plastic deformation, and cosmic radiation dose reconstruction, reporting high average coverage but poor coverage in some test samples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conformal guarantee is broken by the unexplained z=1.96 multiplier in Eq. (6); reported 98–99% coverage reflects overcoverage, not calibration.","rationale":"The reader correctly identifies exchangeability of calibration and test data as a load-bearing assumption, especially for the cosmic radiation case where atypical days break it. However, the single most decisive flaw is internal: Eq. (6) applies z=1.96 to a quantity q_j that is already the conformal quantile. This is not a harmless scaling factor; it invalidates the theoretical guarantee and inflates empirical coverage. The claim 'consistently attains near-nominal empirical coverage' is therefore not supported by the reported numbers, because those numbers are generated by intervals roughly 1.96 times wider than the conformal intervals. The paper's method is easily repairable—remove z and report the resulting coverage—so rejection is not warranted, but acceptance requires the corrected evaluation. This is a partial agreement with the reader because the exchangeability issue is real but secondary; even under perfect exchangeability the method as written would not provide the advertised distribution-free guarantee. My recommendation is CONDITIONAL: require the authors to either justify z from a conformal argument (unlikely, since it contradicts Eq. (5)) or remove it and report Table 3 for the conformal intervals. The other issues flagged by the reader (no code, no CRP-Operator comparison, unexplained z) remain relevant, but the z-multiplier defect is the most load-bearing because it undermines the central claim regardless of dataset-shift considerations.","tokens_in":14938,"tokens_out":3821,"duration_ms":52400,"concrete_test":"Rerun the calibration and test evaluation with Eq. (6) replaced by the standard conformal interval I_j(u_t) = [mu_j(u_t) - q_j sigma_j(u_t), mu_j(u_t) + q_j sigma_j(u_t)], i.e., set z = 1 while keeping Eqs. (4) and (5) unchanged. Recompute Table 3 (average, min, max coverage; counts >=95% and <95%) for all three test cases. If the average coverage drops to roughly 95% and the extreme undercoverage cases persist, the z=1.96 multiplier was merely hiding overcoverage; if coverage drops materially below nominal, the near-nominal coverage claim is an artifact of the multiplier and the method is not calibrated as claimed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is that CMCO produces distribution-free prediction intervals with near-nominal coverage. The construction in Eqs. (4)–(6) does not actually implement split conformal prediction. Eq. (4) defines normalized residuals e_{i,j} = |y_{i,j} - mu_j(u_i)| / sigma_j(u_i), and Eq. (5) computes q_j as the empirical (1-alpha) quantile of those residuals. Standard split conformal would then form the interval mu_j(u_t) +/- q_j sigma_j(u_t), which has marginal coverage at least 1-alpha under exchangeability. Instead, Eq. (6) multiplies by z = 1.96, giving mu_j(u_t) +/- 1.96 q_j sigma_j(u_t). Since q_j is already a calibrated quantile of the normalized residual distribution, this scaling changes the actual coverage to P(e <= 1.96 q_j), which is not controlled at 95% and systematically inflates the intervals. This is an internal inconsistency, not merely a deviation from convention: the theoretical guarantee claimed in Section 5 and the abstract is not instantiated by the implemented interval. It also explains why average coverages in Table 3 are 98.63–99.68%, well above nominal, and it obscures the severity of the undercoverage cases (e.g., 1.14% in Case III). The reader's exchangeability concern is real, but it is secondary: even when exchangeability holds, Eq. (6) does not give the claimed conformal coverage. The fix is simple—remove z or fold it into the conformal quantile—but without it the headline empirical results cannot be interpreted as evidence for the method's central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CMCO, a framework that combines Monte Carlo dropout with split conformal prediction on a sequential DeepONet to produce distribution-free prediction intervals for virtual sensing. It is evaluated on three tasks: turbulent kinetic energy reconstruction in a lid-driven cavity, von Mises stress field prediction in elastoplastic deformation, and global cosmic radiation dose estimation from sparse neutron monitor inputs. The central claim is that CMCO achieves near-nominal 95% marginal coverage with minimal computational overhead.","tokens_in":15309,"tokens_out":3061,"duration_ms":36590,"significance":"If the coverage guarantee were correctly implemented, CMCO would be a practically attractive UQ wrapper for neural operators: it requires only a single trained model, has low inference overhead, and is evaluated on three physically meaningful problems. The paper also provides extensive empirical coverage statistics and visualizations. However, the central theoretical claim is undermined by a nonstandard interval construction that inflates the intervals, and the reported coverage is systematically above nominal, so the results as presented do not substantiate the headline claim.","major_comments":[{"comment":"The prediction interval is defined as mu(ut) +/- z * q * sigma(ut) with z = 1.96, where q is the conformal quantile of normalized residuals from Eq. (5). Standard split conformal prediction with normalized nonconformity scores would construct the interval as mu(ut) +/- q * sigma(ut), which has marginal coverage at least 1 - alpha under exchangeability. The extra multiplier z changes the effective coverage to P(e <= 1.96 q), which is not controlled at the nominal level and systematically inflates the intervals. This invalidates the theoretical guarantee claimed in Section 5 and explains the 98-99% average coverage in Table 3.","section":"Section 2.3, Eq. (6)"},{"comment":"The reported average empirical coverages are 99.68% (Case I), 98.96% (Case II), and 98.63% (Case III), all far above the nominal 95% target. The abstract and Section 5 describe these as 'near-nominal', which is inaccurate. Moreover, the minimum coverages of 92.59%, 53.33%, and 1.14% show that averaging hides severe undercoverage for individual samples. The paper should report the full coverage distribution or bootstrap confidence intervals, and the overcoverage should be acknowledged as a direct consequence of the z multiplier in Eq. (6).","section":"Section 4, Table 3"},{"comment":"The split conformal guarantee relies on exchangeability of calibration and test data. In the cosmic radiation case, the test inputs are 7-day sequences of neutron monitor readings ordered in time; atypical days such as May 10, 2023 (Figure 9b) are not exchangeable with the calibration set, which likely explains the catastrophic 1.14% coverage for that sample. The manuscript does not address this violation or propose a remedy (e.g., sliding-window or weighted conformal prediction), so the distribution-free claim is not established for this application.","section":"Section 3.3 and Figure 9"}],"minor_comments":[{"comment":"The caption states that 'Prediction intervals were obtained using conformalized RP-DeepONet', but the paper proposes CMCO (conformalized MC-dropout DeepONet). This is inconsistent with the rest of the manuscript and should be corrected.","section":"Figure 9 caption"},{"comment":"The normalized residual divides by sigma_j(u_i), which can be zero or near zero in regions where the MC-dropout ensemble is degenerate. The authors should specify a numerical stability correction (a small epsilon) for this division.","section":"Section 2.3, Eq. (4)"},{"comment":"The main text states that the lid-driven cavity branch network is a four-layer GRU with Tanh activations, while the appendix specifies 'Type: LSTM' and ReLU activations for the same case. These inconsistencies should be reconciled.","section":"Section 3.1 vs. Appendix"},{"comment":"The notation 'z · q · sigma' is ambiguous because q and sigma are vectors and z is a scalar. The paper should explicitly state that the multiplication is elementwise, or use a clearer componentwise notation.","section":"Section 2.3, Eq. (6)"}],"recommendation":"major_revision","confidential_remarks":"The conformal calibration procedure (Eqs. 4-5) is described as adopted from the authors' own prior work [31], and the cosmic radiation dose model is taken from reference [2]. The incremental novelty of CMCO over CRP-Operator appears limited to replacing randomized priors with MC dropout. This is a scope/novelty concern the editor may wish to weigh. The technical issue with the z multiplier is the primary reason for major revision; once that is corrected, the empirical coverage claims should be re-evaluated and the paper will be substantially stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is a practical combination of two known ideas, but the main claim doesn't hold in the form written because Eq. (6) multiplies the conformal quantile by z=1.96. That is not split conformal prediction, and the reported 98–99% coverage is overcoverage, not calibration.\n\nWhat's genuinely new: applying MC dropout at inference time in a single DeepONet, computing per-location normalized residuals on a calibration set, and forming per-location quantiles is a clean, low-cost wrapper. The three test cases are realistic, and the cosmic radiation dose example is a nice stress test with global spatial structure. The paper also reports point-forecast errors and failure-rate diagnostics, which is more than many operator-learning UQ papers do.\n\nThe soft spot is load-bearing. Eqs. (4)-(5) define q_j as the empirical (1-α) quantile of |y-μ|/σ. Under exchangeability, μ ± q_j σ has marginal coverage at least 1-α. Eq. (6) instead uses μ ± 1.96 q_j σ. Since q_j is already a calibrated quantile, multiplying by 1.96 changes the actual coverage to P(e ≤ 1.96q_j), which is not controlled and is systematically larger than nominal. That explains the 98.63-99.68% averages in Table 3. The authors call this 'near-nominal,' but three to four points above target is not near-nominal when the target is 95%. The fix is straightforward—drop z or absorb it into the conformal quantile—but without it, the theoretical guarantee in Section 5 does not apply to the intervals actually evaluated.\n\nThe exchangeability concern from the reader is real but secondary. The May 10, 2023 cosmic case with 1.14% coverage is a regime shift, and the paper doesn't discuss what that does to the marginal-coverage claim. It's worth one paragraph of discussion, not a reason to reject.\n\nOther notes: no baseline comparison against CRP-Operator, the authors' own earlier conformal work, and code is promised only after acceptance. Those are addressable.\n\nWho this is for: people building UQ wrappers for neural operators will get a useful recipe and a realistic evaluation template, but they should not quote the coverage numbers without fixing Eq. (6). It deserves a serious referee. I'd recommend engaging and asking for the correction, re-run coverage, and a discussion of when exchangeability fails.","headline":"A practical MC-dropout + conformal wrapper for DeepONets, but the z=1.96 multiplier in Eq. (6) breaks the conformal guarantee and makes the reported 98–99% coverage overcoverage, not near-nominal calibration.","tokens_in":15845,"tokens_out":2825,"would_cite":false,"duration_ms":34387,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single trained neural operator, paired with Monte Carlo dropout and split conformal calibration, produces distribution-free 95% spatial prediction intervals in virtual sensing.","keywords":["conformal prediction","Monte Carlo dropout","neural operators","DeepONet","uncertainty quantification","virtual sensing","distribution-free prediction intervals","cosmic radiation dose"],"falsifier":"Take the calibrated dose model, hold out a solar-storm day not represented in the calibration period, and compute the fraction of spatial grid points covered. A coverage far below 95% — the paper already reports 1.14% on May 10, 2023 — would falsify the claim that near-nominal coverage holds on any new input, and would show the guarantee is conditional on exchangeability rather than universal.","tokens_in":14758,"feed_emoji":"🎯","tokens_out":9098,"duration_ms":102289,"temperature":0.7,"pith_summary":"The paper proposes the Conformalized Monte Carlo Operator (CMCO), a way to attach calibrated uncertainty bands to neural-operator predictions used for virtual sensing. Its central claim is that a single trained Sequential DeepONet can be turned into a probabilistic surrogate by running Monte Carlo dropout forward passes at inference time, and that split conformal calibration of the per-location residuals then produces distribution-free prediction intervals targeting 95% marginal coverage, with no retraining, no ensembling, and no custom loss. On turbulent-flow kinetic energy, elastoplastic stress, and global cosmic-radiation dose reconstructed from sparse neutron monitors, the calibrated intervals reach average coverage of 99.68%, 98.96%, and 98.63%, respectively. The method matters because real-time safety monitoring needs uncertainty estimates that remain valid without distributional assumptions.","feed_headline":"DeepONet plus conformal calibration yields 95% prediction intervals","feed_subtitle":"Spatial uncertainty for virtual sensors, no retraining or ensembles — across flow, plasticity, and cosmic dose.","key_machinery":"The load-bearing object is the calibrated interval $$I(u_t) = [\\mu(u_t) - z \\cdot q \\cdot \\$\\sigma$(u_t), \\; \\mu(u_t) + z \\cdot q \\cdot \\$\\sigma$(u_t)],$$ with $z = 1.96$. The mean $\\mu$ and spread $\\sigma$ come from $n_c$ stochastic forward passes of the trained DeepONet with dropout active at inference, and the vector $q$ is a per-location conformal quantile: for each spatial index $j$, $q_j$ is the $\\lceil (1-\\alpha)(n+1)\\rceil / n$ quantile of the normalized calibration residuals $|y_{i,j} - \\mu_j(u_i)|/\\sigma_j(u_i)$. That normalization is what makes the bandwidth distribution-free: it rescales the MC-dropout variance by the observed error at each location, so the interval width adapts spatially through data alone, with no new training.","core_discovery":"The central claim is that conformalized MC dropout gives operator learning a lightweight, distribution-free UQ layer: for a test input, the interval is $I(u_t) = [\\mu(u_t) - z \\cdot q \\cdot \\sigma(u_t), \\; \\mu(u_t) + z \\cdot q \\cdot \\sigma(u_t)]$ with $z = 1.96$, where $\\mu$ and $\\sigma$ are the mean and standard deviation of $n_c$ stochastic forward passes of the same trained network, and $q$ is a vector of per-location conformal quantiles computed from calibration residuals. The paper reports that this construction achieves average empirical coverage of 99.68% on lid-driven cavity turbulence, 98.96% on elastoplastic deformation, and 98.63% on cosmic-radiation dose estimation, all near or above the nominal 95% level. It also reports that under-coverage concentrates in hard samples: the per-sample minimum coverage across cases is 92.59%, 53.33%, and 1.14%, with the lowest value occurring on an atypical radiation day, so the guarantee is marginal and breaks under distribution shift.","pith_inferences":["Because the conformal quantile $q_j$ is estimated independently at each location, the method leaves spatial structure in $q$ unexploited; smoothing $q$ or conditioning it on local density and gradient strength could recover the under-covered tail samples without inflating widths elsewhere.","The 1.14% minimum coverage on an atypical radiation day is a concrete measure of how fast a fixed calibration set goes stale, so a natural sequel is an adaptive or online conformal update that monitors exchangeability and refreshes $q$ when it breaks.","The same normalized-residual calibration can wrap any base uncertainty proxy, so MC dropout is replaceable by heteroscedastic head uncertainties or randomized priors without changing the conformal layer.","In digital-twin deployments, calibration data and operational data will rarely be exchangeable, and the paper's own Table 3 numbers quantify the size of the safety margin operators should assume rather than treating 95% coverage as a hard guarantee."],"forward_implications":["Existing trained Sequential DeepONets can be deployed with uncertainty by enabling dropout and storing $n_c$ forward passes, with no architectural change or retraining.","One calibration pass produces intervals whose marginal coverage is guaranteed by split conformal theory without assuming Gaussian errors or any parametric family.","The intervals are spatially resolved, so a monitoring system can flag dangerous regions such as cavity corners, necking zones, or the South Atlantic Anomaly rather than reporting a single global uncertainty.","Failure-rate and relative-error scatter plots can serve as a diagnostic to identify samples where the model is simultaneously inaccurate and overconfident, which is the dangerous regime for safety-critical sensing."],"supporting_citations":[{"why":"introduces the DeepONet branch-trunk architecture that the framework wraps.","marker":"[12]"},{"why":"introduces the Sequential DeepONet variant used as the backbone for time-dependent inputs.","marker":"[15]"},{"why":"supplies the split conformal calibration procedure of normalized residuals and per-location quantiles.","marker":"[31]"},{"why":"the Gaussian-process viewpoint cited for interpreting MC dropout as approximate Bayesian inference.","marker":"[32]"},{"why":"the randomized-prior viewpoint cited for the approximate posterior interpretation behind the dropout ensemble.","marker":"[33]"},{"why":"provides the analytic radiation-transport data used as ground truth for the cosmic dose field.","marker":"[34]"},{"why":"prior work that established the proxy-to-field mapping from neutron monitor sequences to dose fields, which this paper extends with uncertainty quantification.","marker":"[2]"}],"fun_headline_variants":["Conformalized neural operators: distribution-free intervals for virtual sensing","No retraining, no ensembles: CMCO delivers calibrated UQ for physical fields","Spatial uncertainty without the cost: conformalized MC dropout for operators","Operator learning meets conformal prediction: near-95% coverage out of the box","Plug-and-play UQ for neural operators: conformal intervals, no custom losses"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee rests on calibration and test input-output pairs being exchangeable, meaning the residual pattern seen during calibration must still be representative when the model is deployed.","fun_headline_variants_meta":{"raw":{"variants":["Conformalized neural operators: distribution-free intervals for virtual sensing","No retraining, no ensembles: CMCO delivers calibrated UQ for physical fields","Spatial uncertainty without the cost: conformalized MC dropout for operators","Operator learning meets conformal prediction: near-95% coverage out of the box","Plug-and-play UQ for neural operators: conformal intervals, no custom losses"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000835,"raw_usage":{"total_tokens":3671,"prompt_tokens":1003,"completion_tokens":2668,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":619,"completion_tokens_details":{"reasoning_tokens":2567}},"tokens_in":619,"tokens_out":2668,"duration_ms":21036,"temperature":1.0,"reasoning_tokens":2567,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:20:24.434025+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the calibrated dose model, hold out a solar-storm day not represented in the calibration period, and compute the fraction of spatial grid points covered. A coverage far below 95% — the paper already reports 1.14% on May 10, 2023 — would falsify the claim that near-nominal coverage holds on any new input, and would show the guarantee is conditional on exchangeability rather than universal.","supporting_citations":[{"cited_title":"Sequential deep operator networks (s-deeponet) for predicting full-field solutions under time-dependent loads","cited_arxiv_id":null,"evidence_quote":"introduces the Sequential DeepONet variant used as the backbone for time-dependent inputs."},{"cited_title":"Distribution free uncertainty quantification for neuroscience-inspired deep neural operators","cited_arxiv_id":null,"evidence_quote":"supplies the split conformal calibration procedure of normalized residuals and per-location quantiles."},{"cited_title":"Gaussian processes in machine learning","cited_arxiv_id":null,"evidence_quote":"the Gaussian-process viewpoint cited for interpreting MC dropout as approximate Bayesian inference."},{"cited_title":"Randomized prior functions for deep reinforcement learning","cited_arxiv_id":null,"evidence_quote":"the randomized-prior viewpoint cited for the approximate posterior interpretation behind the dropout ensemble."},{"cited_title":"Analytical model for estimating terrestrial cosmic ray fluxes nearly anytime and anywhere in the world: Extension of parma/expacs","cited_arxiv_id":null,"evidence_quote":"provides the analytic radiation-transport data used as ground truth for the cosmic dose field."}],"review_version":1}