{"id":"0e55df0e-0c9f-44f2-ae2d-47abd411a6e3","arxiv_id":"2605.22891","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Pointwise metrics compress marginal spectra in multimodal inverse problems, and a three-part protocol using CRPS, spectrum fidelity, and calibration reverses model rankings on synthetic and particle-physics benchmarks.","lead":"The paper shows that pointwise metrics like RMSE produce artificially narrow output distributions in multimodal inverse problems by the law of total variance. A smart generalist should read it because standard evaluation can distort the spectral features scientists actually measure in reconstruction tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Claim that bias is independent of dataset size/training overlooks finite-data approximation to conditional mean/median","rationale":"The reader's weakest_assumption targets the downstream-utility premise, which is outside the mathematical claim. The load-bearing issue internal to the strongest_claim is the overstatement of independence from finite-data factors; the core narrowing result survives in the population limit but requires qualification for the stated generality.","tokens_in":1692,"tokens_out":317,"duration_ms":23457,"concrete_test":"On the paper's synthetic benchmark with analytic posterior, compute the exact marginal variance of the true conditional mean; then train the same architecture on subsampled training sets of size 10^2, 10^3, 10^4 and measure the empirical variance of its point predictions; check whether the gap to the theoretical value shrinks monotonically with N.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The law of total variance gives Var(E[θ|data]) = Var(θ) − E[Var(θ|data)] exactly for the population MSE minimizer, so its marginal distribution is narrower. The paper applies the same statement to MAE minimizers (conditional medians) and asserts the bias amount is independent of architecture/training/dataset size. Neither holds exactly: (1) no variance decomposition exists for the conditional median, and (2) any finite-sample estimator only approximates the population conditional mean/median, so the realized narrowing depends on how closely that approximation is achieved, which varies with N, model capacity, and optimization.","agreement_with_reader":"partial"},"referee_report":{"model":"grok-4.3","summary":"The manuscript argues that pointwise metrics (RMSE, MAE) structurally mislead in multimodal inverse problems: by the law of total variance, point estimators produce marginal spectra narrower than the true posterior whenever posterior width is nonzero, with the bias independent of architecture/training/dataset size and compressing tails/modes/shapes needed for downstream science. It proposes a three-part protocol (CRPS for per-event distributional accuracy, spectrum-fidelity diagnostic for population marginal accuracy, coverage calibration for uncertainty) and demonstrates ranking reversals on a synthetic analytic-posterior benchmark and a particle-physics many-to-one inverse problem.","tokens_in":1827,"tokens_out":520,"duration_ms":16566,"significance":"If the core claims hold, the work is significant for ML evaluation in scientific inverse problems (e.g., particle physics), where it shows that metric choice can reverse model rankings and alter scientific conclusions. The analytic posterior in the synthetic benchmark is a strength for exact verification. The independence claim, if rigorously established, would be a notable result.","major_comments":[{"comment":"Abstract: the claim that 'the resulting bias is independent of architecture, training, and dataset size' does not hold exactly. The law of total variance decomposition applies to the population conditional mean (MSE minimizer) but has no direct analogue for conditional medians (MAE); any finite-sample estimator only approximates the population quantity, so realized narrowing depends on N, capacity, and optimization.","section":"Abstract"},{"comment":"Abstract: no explicit formula, derivation, or definition is supplied for the 'spectrum-fidelity diagnostic' that forms the second leg of the proposed protocol; this quantity is load-bearing for the claim that the protocol targets failure modes missed by pointwise metrics.","section":"Abstract"},{"comment":"Abstract: the premise that downstream scientific measurements 'rely on' the full posterior spectrum (tails, modes, shapes) rather than low-order moments or point estimates is asserted without derivation or empirical support; this is central to the significance argument but remains an assumption.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract is information-dense; consider separating the protocol description from the bias argument for readability.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The skeptic concern about finite-sample approximation and the lack of a variance identity for medians is directly relevant and load-bearing; addressing it would strengthen the independence claim."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for these precise comments on the abstract. They highlight areas where greater rigor and explicitness will strengthen the manuscript. We address each point below and have made targeted revisions.","responses":[{"response":"We agree that the law of total variance supplies an exact population-level statement for the conditional mean (MSE minimizer) and that finite-sample estimators only approach this limit. The original wording was intended to convey that the bias is architectural- and data-size-independent once the estimator converges to the population quantity, but the phrasing was imprecise. For MAE the decomposition is not identical, though the qualitative compression of marginal spectra still occurs under multimodality. We have revised the abstract to read 'in the population limit, independent of architecture...' and added a clarifying sentence in Section 2.1 distinguishing the MSE case from the MAE case while preserving the core structural claim.","revision_made":"yes","referee_comment":"[Abstract] Abstract: the claim that 'the resulting bias is independent of architecture, training, and dataset size' does not hold exactly. The law of total variance decomposition applies to the population conditional mean (MSE minimizer) but has no direct analogue for conditional medians (MAE); any finite-sample estimator only approximates the population quantity, so realized narrowing depends on N, capacity, and optimization."},{"response":"The spectrum-fidelity diagnostic is the integrated absolute difference between the empirical CDF of the reconstructed marginal and the true marginal CDF, evaluated over a fine grid of the observable. We have inserted a concise parenthetical definition and the explicit formula into the abstract and expanded the formal definition, including the discretization used in the experiments, in the revised Section 3.2.","revision_made":"yes","referee_comment":"[Abstract] Abstract: no explicit formula, derivation, or definition is supplied for the 'spectrum-fidelity diagnostic' that forms the second leg of the proposed protocol; this quantity is load-bearing for the claim that the protocol targets failure modes missed by pointwise metrics."},{"response":"The premise reflects standard practice in particle-physics unfolding and resonance extraction, where tail probabilities and spectral shapes directly enter cross-section and parameter fits. We have added a short paragraph in the introduction citing representative HEP references on the necessity of full-spectrum fidelity and included a quantitative illustration from the particle-physics benchmark showing how the compressed marginal produces a statistically significant bias in a downstream observable. While a domain-general derivation is outside the paper's scope, the revision supplies both literature grounding and empirical support.","revision_made":"partial","referee_comment":"[Abstract] Abstract: the premise that downstream scientific measurements 'rely on' the full posterior spectrum (tails, modes, shapes) rather than low-order moments or point estimates is asserted without derivation or empirical support; this is central to the significance argument but remains an assumption."}],"tokens_in":1402,"tokens_out":614,"duration_ms":16182,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The one or two things to know: pointwise metrics like RMSE or MAE produce narrower marginal distributions than the true posterior whenever the posterior has width, and the authors offer a three-part protocol to check per-event accuracy, marginal fidelity, and calibration. The protocol is the main new element. It combines CRPS for individual events, a spectrum-fidelity diagnostic for the population level, and coverage checks. This targets different failure modes that single point metrics miss. The paper does well to apply the law of total variance to explain the narrowing effect on tails and modes, and to show on synthetic and particle-physics cases that model rankings can reverse under the new metrics versus standard ones. The soft spots are around the strong claim that the bias is independent of architecture, training, and dataset size. The stress-test concern is on target: the variance identity is exact only for the population conditional mean under MSE, not for finite-sample estimators and not at all for medians under MAE. In practice the realized narrowing will vary with how well the model approximates the conditional quantity, which depends on data size and optimization. The abstract also gives no explicit formula for the spectrum-fidelity diagnostic and no quantitative results, so the ranking-reversal evidence cannot be assessed. The premise that downstream science needs the full spectrum rather than moments is stated but not evidenced. This paper is for researchers doing probabilistic reconstruction in scientific domains such as high-energy physics. A reader interested in evaluation methods would find the protocol structure worth considering as a starting point. It deserves peer review to supply the missing formulas, verify the experiments, and adjust the independence claim to match finite-data reality.","headline":"Pointwise metrics can narrow marginal spectra in multimodal inverse problems, but the independence-from-training claim does not hold exactly and the protocol lacks supporting details.","tokens_in":2324,"tokens_out":399,"would_cite":false,"duration_ms":25038,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":{"model":"grok-4.3","evidence":[],"headline":"Evaluation protocol for multimodal inverse problems via total variance; no RS machinery","alignment":"orthogonal","rationale":"Paper centers on law-of-total-variance compression of marginal spectra (Var[E[z|x]] ≤ Var[z]) and a three-metric protocol (CRPS + χ²_spec + conformal coverage) for inverse-problem evaluation. RS framework derives J-cost, φ-ladders, 8-tick periodicity, D=3, and constants from a single distinction (reality_from_one_distinction, AbsoluteFloorClosure, Cost/FunctionalEquation, AlexanderDuality). No shared structures (cosh-cost, ratio symmetry, J-forcing, parameter-free constants) appear; domain (ML evaluation of reconstruction) lies outside RS theorems.","tokens_in":58755,"confidence":"high","tokens_out":170,"duration_ms":5191,"cache_read_input_tokens":38528,"cache_creation_input_tokens":0},"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Point estimators minimizing MSE or MAE always produce narrower marginal spectra than the true posterior in multimodal inverse problems.","keywords":["multimodal inverse problems","pointwise metrics","evaluation protocol","posterior spectrum","CRPS","uncertainty calibration","particle physics reconstruction"],"falsifier":"A direct comparison on a problem with known analytic multimodal posterior where the marginal variance of point predictions equals the true posterior variance instead of being smaller.","tokens_in":2617,"feed_emoji":"📉","tokens_out":631,"duration_ms":19760,"temperature":0.7,"pith_summary":"Evaluation of scientific reconstructions relies heavily on pointwise metrics such as RMSE and MAE. The paper shows this leads to systematic bias because, by the law of total variance, any such point estimator compresses the spectrum whenever the posterior has width. This compression hides the very features like tails and modes that matter for later measurements. To address it, the authors introduce a protocol with three checks: distributional accuracy per event using CRPS, overall marginal spectrum match, and proper uncertainty calibration. Experiments on synthetic data and a particle physics inverse problem demonstrate that conclusions about which model is best can flip depending on the evaluation method used.","feed_headline":"Point estimators narrow spectra in multimodal inverse problems","feed_subtitle":"Variance law proves point metrics compress tails and modes; new protocol reverses model rankings on physics benchmarks.","key_machinery":"The law of total variance decomposition showing that point predictions from a multimodal posterior must have strictly smaller marginal variance than the true distribution.","core_discovery":"The central discovery is that pointwise metrics are structurally misleading for inverse problems with multimodal posteriors. By the law of total variance, point estimators trained to minimize MSE or MAE produce a marginal spectrum strictly narrower than the truth. The bias is independent of architecture, training, and dataset size. A three-part protocol is proposed: CRPS for per-event accuracy, a spectrum-fidelity diagnostic for population marginals, and coverage calibration for uncertainty. On benchmarks, model rankings reverse and calibration distinguishes further.","pith_inferences":["Similar compression effects likely appear in other reconstruction tasks with multimodal posteriors such as medical imaging or astronomical parameter estimation.","The protocol offers a concrete way to test whether full posterior sampling avoids the variance loss shown for point estimators.","Existing pipelines that rely only on point metrics may need re-examination to check how much spectral information was lost."],"forward_implications":["Model rankings obtained from pointwise metrics reverse when distributional metrics are used instead.","Calibration checks can separate models that appear equivalent under CRPS alone.","The choice of evaluation protocol determines the final scientific conclusion about model performance.","Downstream analyses depending on spectral features will be biased by the use of point estimators."],"fun_headline_variants":["Pointwise metrics compress spectra in multimodal inverses","Point estimators narrow marginal spectra structurally","Variance law reveals bias in pointwise inverse metrics","CRPS protocol reverses model rankings on physics data"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Downstream scientific measurements actually depend on the full shape of the posterior including tails and modes rather than just point estimates or low-order moments.","fun_headline_variants_meta":{"raw":{"variants":["Pointwise metrics compress spectra in multimodal inverses","Point estimators narrow marginal spectra structurally","Variance law reveals bias in pointwise inverse metrics","CRPS protocol reverses model rankings on physics data"]},"model":"grok-4.3","cost_usd":0.008948,"raw_usage":{"total_tokens":3932,"prompt_tokens":652,"num_sources_used":0,"completion_tokens":48,"cost_in_usd_ticks":89478000,"prompt_tokens_details":{"text_tokens":652,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3232,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":652,"tokens_out":48,"duration_ms":17481,"temperature":1.0,"reasoning_tokens":3232,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-25T05:37:52.267611+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A direct comparison on a problem with known analytic multimodal posterior where the marginal variance of point predictions equals the true posterior variance instead of being smaller.","supporting_citations":[],"review_version":1}