{"id":"1868db1f-47fc-4ac2-a4b1-ee4d64767a41","arxiv_id":"1908.07253","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper introduces n-MeRCI, a normalized metric that scores predictive uncertainty by its average rescaled width after calibrating to a target coverage rate.","lead":"This paper proposes a new metric, n-MeRCI, for evaluating how well a neural network's predicted uncertainty tracks its actual errors in regression tasks. The metric is designed to be scale-independent, robust to outliers, and easy to interpret, and it is demonstrated on a toy dataset and on monocular depth estimation.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"n-MeRCI's normalization in Eq. (2) is order-reversing when the α-quantile of errors is below MAE; this is exactly the heavy-tailed regime the metric is designed for, so the claimed '>1 means worse than constant' and '0 is the lower bound' properties are not guaranteed.","rationale":"The reader's primary weakest assumption was global multiplicative calibration; that is a real limitation but it concerns whether the metric measures correlation under heteroscedastic miscalibration. The more load-bearing and checkable defect is the normalization in Eq. (2): the denominator max_alpha(|\\hat y - y*|) - MAE can be negative exactly in the heavy-tailed, outlier-containing setting that α<100 is introduced to handle. When the denominator is negative, the intended scale is inverted: values below 1 can correspond to predictors worse than constant, and values below 0 can occur even though 0 is claimed as a lower bound. The synthetic counterexample in the test is not contrived; it mirrors the paper's own design of using a high percentile to discard up to 5% outliers, while keeping the mean in the denominator. The paper does not state or prove any condition under which max_alpha exceeds MAE, and the depth-estimation experiments do not report or check the sign of the denominator per depth interval. This is not an external consensus disagreement; it is an internal inconsistency between Eq. (2) and the properties claimed in Section III-D. The experimental ranking of Multi Epochs could survive if all reported denominators are positive, but the metric as defined is not a reliable score in the outlier-robust regime. The paper should either restrict the definition to settings where the denominator is positive, replace the normalization with a form that is provably order-preserving, or explicitly characterize when the claimed lower bound holds. Until then, acceptance should be conditional on this fix and on re-checking Fig. 5's denominator signs.","tokens_in":8289,"tokens_out":16258,"duration_ms":158960,"concrete_test":"Run a synthetic check: let N=100 absolute errors be 95 copies of 1 and 5 copies of 1000, and set α=95. Define σ_i=2 for the 95 small-error points and σ_i=1000 for the 5 large-error points. Compute the smallest λ covering 95% of points: λ_95=0.5, giving MeRCI_95=25.95. Compute MAE=50.95 and max_95=1. Then Eq. (2) yields n-MeRCI=(25.95-50.95)/(1-50.95)=0.50, not >1, even though MeRCI exceeds the constant predictor's score of max_95=1. If the claimed ordering property held, this value would be above 1. The same computation with σ_i=100 on the small-error points and σ_i=1000 on the large-error points gives n-MeRCI≈0.99, again below 1. This single reproducible computation settles whether the normalization preserves the paper's interpretation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central interpretability claims of Section III-D fail because the denominator of Eq. (2), max_alpha(|\\hat y - y*|) - MAE, can be negative. For non-negative errors, the α-quantile is below the mean whenever a small fraction of very large errors dominates the average. The paper deliberately uses α=95 to tolerate up to 5% outliers, but those outliers still enter MAE, so this is not a corner case: it is the target regime. If the denominator is negative, the inequality is reversed: a predictor whose MeRCI exceeds max_alpha (worse than the constant predictor by the paper's own definition) receives n-MeRCI < 1. Concretely, take 95 errors of size 1 and 5 errors of size 1000, α=95. Then MAE=50.95 and max_95=1, so the denominator is -49.95. Let σ_i=2 on the 95 small-error points and σ_i=1000 on the 5 large-error points. The smallest λ covering 95% of points is λ_95=0.5 (only the small-error points are covered), giving MeRCI_95=0.5·(95·2+5·1000)/100=25.95. Thus n-MeRCI=(25.95-50.95)/(1-50.95)=0.50, although MeRCI is 25 times the constant predictor's score. Inflating σ on the discarded 5% outliers makes the score even lower, so the metric rewards wasting uncertainty on the very points it claims to be robust against. Relatedly, for α<100 the lower bound 0 is false: setting σ near 0 on the (1-α) smallest-error points keeps α coverage while making MeRCI < MAE, yielding negative n-MeRCI. The paper's stated '0 for the oracle' and '>1 for worse than constant' are therefore not properties of Eq. (2) without an additional condition that max_alpha exceeds MAE.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes n-MeRCI, a normalized evaluation metric for predictive uncertainty in regression. The metric first rescales all predicted uncertainties by a common factor chosen so that an alpha-percentile of absolute errors is covered, yielding MeRCI_alpha as the average rescaled uncertainty (Eq. 1). It then normalizes by the scores of an oracle and a constant uncertainty predictor, giving n-MeRCI_alpha in Eq. (2), with claimed values 0 for the oracle, 1 for a constant predictor, and >1 for predictors worse than constant. The authors validate the metric on a toy regression problem and on NYU-Depth v2 monocular depth estimation, comparing Monte Carlo Dropout, Deep Ensembles variants, Multi Epochs, Multi Networks, Bagging, and a Learned Error approach.","tokens_in":8734,"tokens_out":5501,"duration_ms":59603,"significance":"If the metric fulfilled its stated properties, it would fill a real gap: a scale-invariant, outlier-robust, scalar measure of how well predicted uncertainties track true errors, applicable to regression without distributional assumptions. The paper is clearly written and the experimental comparison is relevant to the robotics and deep-learning communities. However, the central normalization claimed in Section III-D is not a mathematical property of Eq. (2), and the experimental protocol contains gaps (no error bars, double outlier removal, single-image qualitative support). The contribution is therefore not yet established, though the underlying idea is worth further development.","major_comments":[{"comment":"The denominator max_alpha(|y_i - y*_i|) - MAE is not always positive, and the paper's interpretability claims fail when it is negative. For example, take 95 errors of size 1 and 5 errors of size 1000, with alpha=95. Then MAE=50.95 and max_95=1, so the denominator is -49.95. With sigma_i=2 on the 95 small-error points and sigma_i=1000 on the 5 large-error points, the smallest lambda covering 95% of points is lambda_95=0.5, giving MeRCI_95=25.95, which is 25 times the constant predictor's MeRCI of 1. Yet Eq. (2) gives n-MeRCI_95=(25.95-50.95)/(1-50.95)=0.50, i.e. below 1, contradicting the stated '>1 means worse than constant' property. Increasing the uncertainties on the discarded 5% outliers makes n-MeRCI even lower (more negative), so the metric rewards wasting uncertainty on the very points it claims to ignore. The claimed lower bound of 0 is also not guaranteed: by putting tiny uncertainties on the (1-alpha) smallest-error points, one can keep alpha coverage while making the numerator of Eq. (2) negative, yielding n-MeRCI < 0. These are not corner cases; they are the heavy-tailed and outlier-contaminated regimes the alpha parameter is explicitly designed to handle. The paper must either redefine the normalization, restrict the claims to distributions where Q_alpha(epsilon) > MAE, or provide a corrected analysis.","section":"Section III-D, Eq. (2)"},{"comment":"The metric rests on the assumption that a single global scaling factor lambda can make the uncertainties compatible with the errors. This is only appropriate when the predictive uncertainty is correctly specified up to a global constant; if the model is overconfident in some regions and underconfident in others, no single lambda can fix the mismatch. In that common heteroscedastic miscalibration regime, n-MeRCI may assign a good score to a model with poor local correlation between uncertainty and error, or a poor score to a model with good local correlation. The paper does not state or test this assumption, and the toy and depth experiments do not include a diagnostic for whether the assumption holds.","section":"Section III-C, Eq. (1)"},{"comment":"The quantitative depth evaluation applies two separate outlier-removal mechanisms: the paper states 'We withdraw 5% of the worst points' and then computes n-MeRCI with alpha=95. Since n-MeRCI_95 is already designed to ignore up to 5% of outliers at the metric level, the additional removal changes the evaluation and means the reported scores are not the n-MeRCI defined in Eq. (2) on the full test set. This makes the ranking of methods, including the central claim that Multi Epochs is best, depend on an undocumented preprocessing choice. Please report results on the full set, state the number of points removed, or justify why double removal is appropriate.","section":"Section VI-C"},{"comment":"The experimental validation lacks error bars or confidence intervals, although the text itself acknowledges that the toy results are 'very noisy' and that the NYU evaluation is based on a single trained model per method. The comparison in Fig. 5 reports only point estimates per depth interval, and Fig. 3 uses one image to explain the ranking. Without repeated runs or statistical testing, the observed differences between Multi Epochs, Monte Carlo Dropout, and Multi Inits (e.g. 0.45 vs 0.47 in Fig. 3, or 0.11 vs other averages in Fig. 5) cannot be distinguished from noise. At minimum, report standard deviations or confidence intervals and the number of images/depth pixels used for each average.","section":"Section V-C and VI-C"}],"minor_comments":[{"comment":"The notation 'max_alpha' is ambiguous: the text first describes an alpha-th percentile for the scaling factor, then calls a percentile an upper bound 'max_alpha'. Please define a single symbol such as Q_alpha(...) and use it consistently in Eq. (1), Eq. (2), and the discussion.","section":"Section II and Section III"},{"comment":"There are several typos and wording issues: 'evaluated a on a toy dataset' in the abstract, 'inliners' should be 'inliers' in Section III-C, and the percentage inconsistency in Section V-B where 'although there is 85% of outliers' should read '15% of outliers'. These should be corrected.","section":"Abstract and Section II"},{"comment":"The description of Fig. 2 says the score 'decreases as the number of inliners increases, to some extent', but the observed deterioration around alpha=60% shows non-monotonic behavior; a more careful explanation of why the percentile curve is not monotonic would help the reader.","section":"Section V-B"},{"comment":"Figure captions for Fig. 4 and Fig. 5 report average MAE and average n-MeRCI in brackets, but the figures themselves are only referenced as plots w.r.t. depth intervals; please state in the text how the per-interval scores are aggregated and whether the averages are over pixels or over images.","section":"Section VI-C"}],"recommendation":"major_revision","confidential_remarks":"The main issue is mathematical rather than experimental: the normalization in Eq. (2) does not have the claimed range properties in the heavy-tailed regime the paper targets. This is fixable in principle by redefining the denominator or by using an inlier-restricted MAE, but it requires re-running the experiments. The self-citation to [21] for the depth architecture is not problematic. I recommend major revision rather than rejection because the core idea is useful and the identified flaw is specific and localizable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nn-MeRCI is a simple, scale-invariant scalar for comparing uncertainty estimates, and that idea is genuinely useful. It rescales every sigma by one global factor so alpha percent of errors are covered, then normalizes the mean rescaled width against a constant predictor. That formulation is new relative to what they cite, even though it is close to interval-score ideas they do not cite. The toy and depth experiments cover a reasonable set of methods and give the field a concrete number to rank them by.\n\nThe soft spot is structural. Eq. (2)'s denominator, max_alpha - MAE, can be negative. In heavy-tailed error distributions—exactly why alpha=95 is chosen—the 95th percentile of errors is often below the mean absolute error. When that happens the metric's ordering reverses: a predictor that blows up uncertainty on the excluded 5% can score below 1, and even below the oracle's 0 region, while being far worse than a constant predictor. The paper's central claims—that 0 is the perfect lower bound and >1 means worse than constant—are not properties of Eq. (2) without the extra condition max_alpha > MAE. That condition is neither stated nor tested. The stress-test example is enough to make the point. This is a load-bearing flaw, not a minor corner case.\n\nThere are also smaller issues: max_alpha is not precisely defined (it's presumably the alpha-th percentile), no error bars are given on the comparisons, and the experimental protocol removes 5% of worst points while alpha=95, so the robust-outlier claim is double-counted. The paper would benefit from comparing against interval score, quantile coverage, and calibration/sharpness diagnostics from the forecasting literature. The qualitative evaluation is a single image, so I would not rest much on it.\n\nThe Multi Epochs conclusion is plausible, but it currently rests on a metric whose interpretability is not reliable in the regime it targets. That ordering could survive a corrected normalization, but it is not established here.\n\nWho gets value? Anyone working on predictive uncertainty evaluation, as a cautionary example and a starting point for a fixed scalar metric. Worth sending to review, but with clear expectation of major revision before it can be adopted.","headline":"A scale-invariant uncertainty metric with a genuinely useful core, but the headline interpretability claims break down in exactly the heavy-tailed regime it was designed to handle.","tokens_in":9228,"tokens_out":3455,"would_cite":false,"duration_ms":35970,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"n-MeRCI is a new normalized metric for whether predicted uncertainties track true errors, with 0 for perfect, 1 for no better than constant, and above 1 for worse.","keywords":["predictive uncertainty","uncertainty evaluation","regression","deep learning","monocular depth estimation","calibration","sharpness","n-MeRCI"],"falsifier":"Construct a synthetic regression with known errors and assign each point an uncertainty equal to the reciprocal of its true error, rescaled so that 95% of the intervals still cover the errors. Since the uncertainties are anti-correlated with the errors, a metric that truly measures correlation should return a score at or above 1; if n-MeRCI scores near 0 on this construction, the global rescaling step is hiding the miscalibration.","tokens_in":8143,"feed_emoji":"📏","tokens_out":7294,"duration_ms":63546,"temperature":0.7,"pith_summary":"The paper proposes n-MeRCI, a regression metric that scores how well a model's predicted uncertainties track its actual errors. It rescales every uncertainty by one common factor chosen so that a chosen percentage of errors fall inside the rescaled intervals, then compares the average rescaled uncertainty with two reference points: the mean absolute error of a perfect oracle and the score of a constant uncertainty predictor. The result is 0 for a perfect oracle, 1 for a predictor no better than a constant, and above 1 for one that is misleading. The authors validate the metric on a toy regression and on monocular depth estimation, where they report that an ensemble built from several epochs of one network gives the best uncertainty estimates among the methods compared. The practical payoff is a way to judge, before acting on a prediction, whether the uncertainty attached to it can be trusted.","feed_headline":"n-MeRCI puts a single number on how well uncertainty tracks error","feed_subtitle":"In depth-estimation tests, checkpoint ensembles beat Monte Carlo Dropout and bagging at flagging true errors.","key_machinery":"The load-bearing object is the rescaled confidence interval: each predicted uncertainty $\\sigma_i$ is multiplied by a single global factor $\\lambda_\\alpha$ taken as the $\\alpha$-th percentile of the ratios $|\\hat{y}_i-y_i^*|/\\sigma_i$, so that $\\alpha\\%$ of the true errors lie inside the interval centered on the prediction. n-MeRCI then normalizes the mean width of these intervals by subtracting the oracle's mean absolute error and dividing by the same quantity for a constant uncertainty predictor. This ties calibration and sharpness together: $\\lambda_\\alpha$ enforces calibration at one percentile, while the average width penalizes uncertainty estimates that are unnecessarily wide.","core_discovery":"The central claim is that the quality of predictive uncertainty can be reduced to a single scale-invariant, outlier-robust number. For a chosen percentile $\\alpha$, the metric finds the minimal global factor $\\lambda_\\alpha$ such that $\\alpha\\%$ of the absolute errors are covered by the rescaled intervals $\\lambda_\\alpha\\sigma_i$, averages these rescaled uncertainties, subtracts the mean absolute error, and divides by the corresponding value for a constant uncertainty predictor. This yields 0 for an oracle, 1 for a constant predictor, and values above 1 for uncertainty estimates worse than none. Applied to monocular depth estimation on NYU-Depth v2, the paper finds Multi Epochs—the standard deviation of predictions from one network saved at different epochs after convergence—to be the best uncertainty estimator (average n-MeRCI$_{95}=0.11$, MAE $=0.29$), while Learned Error is the worst ($0.65$).","pith_inferences":["If the metric becomes standard, it may expose a blind spot: a model that is well calibrated globally but overconfident in some regions and underconfident in others can still receive a good score, so practitioners should pair n-MeRCI with per-region or per-depth-interval inspection.","The same rescaling construction could be turned into a training objective: minimize the normalized mean rescaled interval directly rather than treating it only as an evaluation metric.","The Multi Epochs result points to a cheap general recipe for uncertainty in deep regression: use checkpointed predictions from the final epochs of a single training run, relying on stochastic oscillations to highlight error-prone regions.","Because coverage is checked at a single percentile, two predictors with identical n-MeRCI scores can differ in tail behaviour; computing the score at a second percentile would probe tail miscalibration."],"forward_implications":["Uncertainty evaluation no longer requires a full predictive distribution; any deep regression method that outputs a scalar $\\sigma$ can be scored directly.","Because the metric is invariant to a global rescaling of uncertainties, methods that simply inflate their variances are not rewarded.","The percentile parameter makes the score robust to a controlled fraction of outliers, so a few bad points do not dominate the verdict.","On monocular depth estimation, Multi Epochs produces the most trustworthy uncertainty estimates among the compared methods, at no additional training cost beyond saving checkpoints.","A score above 1 identifies an uncertainty predictor that is worse than ignoring uncertainty entirely."],"supporting_citations":[{"why":"Supplies the calibration-and-sharpness framing of forecast quality that n-MeRCI turns into a single score.","marker":"[12]"},{"why":"Defines Monte Carlo Dropout, one of the uncertainty estimation methods the metric is applied to and ranks.","marker":"[7]"},{"why":"Underlies the Deep Ensembles / Multi Inits baseline that n-MeRCI evaluates.","marker":"[8]"},{"why":"Defines Bagging, the ensemble uncertainty method n-MeRCI ranks on the toy dataset.","marker":"[18]"},{"why":"Provides the NYU-Depth v2 indoor dataset on which the depth-estimation validation is run.","marker":"[20]"},{"why":"Supplies the hourglass network architecture and the 16K-image protocol used to test all uncertainty methods.","marker":"[21]"}],"fun_headline_variants":["n-MeRCI quantifies uncertainty's correlation to true error","n-MeRCI scores uncertainty's link to actual error","n-MeRCI: one score for uncertainty-error correlation","n-MeRCI puts a number on uncertainty's true-error fit","n-MeRCI links uncertainty estimates to real errors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The metric assumes that each uncertainty estimate is correct up to one global multiplier; if a model is overconfident in some regions and underconfident in others, no single rescaling factor can fix that mismatch, and the score stops measuring the true correlation between uncertainty and error.","fun_headline_variants_meta":{"raw":{"variants":["n-MeRCI quantifies uncertainty's correlation to true error","n-MeRCI scores uncertainty's link to actual error","n-MeRCI: one score for uncertainty-error correlation","n-MeRCI puts a number on uncertainty's true-error fit","n-MeRCI links uncertainty estimates to real errors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001576,"raw_usage":{"total_tokens":6271,"prompt_tokens":911,"completion_tokens":5360,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":527,"completion_tokens_details":{"reasoning_tokens":5277}},"tokens_in":527,"tokens_out":5360,"duration_ms":40595,"temperature":1.0,"reasoning_tokens":5277,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:22:03.447425+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a synthetic regression with known errors and assign each point an uncertainty equal to the reciprocal of its true error, rescaled so that 95% of the intervals still cover the errors. Since the uncertainties are anti-correlated with the errors, a metric that truly measures correlation should return a score at or above 1; if n-MeRCI scores near 0 on this construction, the global rescaling step is hiding the miscalibration.","supporting_citations":[{"cited_title":"Dropout as a bayesian approximation: Representing model uncertainty in deep learning,","cited_arxiv_id":null,"evidence_quote":"Defines Monte Carlo Dropout, one of the uncertainty estimation methods the metric is applied to and ranks."},{"cited_title":"Bagging predictors,","cited_arxiv_id":null,"evidence_quote":"Defines Bagging, the ensemble uncertainty method n-MeRCI ranks on the toy dataset."},{"cited_title":"Deep multi-scale architectures for monocular depth estimation,","cited_arxiv_id":null,"evidence_quote":"Supplies the hourglass network architecture and the 16K-image protocol used to test all uncertainty methods."}],"review_version":1}