{"id":"1ec544be-1040-4617-9184-cf867ebd92bb","arxiv_id":"2412.15301","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ρ-Norm Scaling, a temperature-scaling variant with a learnable norm and KL regularization, reduces expected calibration error on several image classifiers without changing accuracy.","lead":"Classifier confidence scores are often too sure, so this paper adds a post-hoc calibration step that divides logits by a learnable norm before applying softmax, and adds a KL regularizer to keep the calibrated distribution close to the original. On image benchmarks it reports much lower calibration error than temperature scaling or vector scaling while leaving accuracy unchanged.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (2)'s ρ-norm is undefined for negative logits and non-integer ρ, so the method as written is not computable and the reported results depend on an unspecified variant.","rationale":"The reader's weakest assumption identifies the same issue: Eq. (2) writes the ρ-norm without absolute values, so for negative logits and non-integer ρ the calibrator is undefined. I agree this is the single most load-bearing concern because the entire empirical evaluation rests on this mapping: if the formula is taken literally, the method cannot run; if absolute values are silently introduced, the theoretical bounds and proofs no longer match the evaluated method. This is a correctness and reproducibility risk, not merely a stylistic omission. The concern does not force outright rejection because the fix is straightforward — replacing z_j^ρ with |z_j|^ρ and re-proving the propositions — and the reported numbers may still be reproducible under that correction. Since the reader already conditioned the verdict on exactly this ambiguity, my analysis leaves the CONDITIONAL verdict unchanged. I would add that the absence of code makes the ambiguity unresolvable from the manuscript, so the condition should explicitly require either code release or a corrected formal definition.","tokens_in":14275,"tokens_out":7013,"duration_ms":60303,"concrete_test":"Obtain the authors' implementation or re-implement both variants: (a) the literal Eq. (2) with z_j^ρ and no absolute value, and (b) the corrected Lp norm (Σ|z_j|^ρ)^{1/ρ}. Run Algorithm 1 on CIFAR-100 ResNet50 logits. If variant (a) produces NaN/complex values or negative denominators for typical logits, the paper's stated definition cannot be what produced Table 1; if variant (b) reproduces the reported ECE of 0.007 ± 0.018, the paper needs a corrected definition and revised proofs for Propositions 1 and 2.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central construction, the calibrator g in Eq. (2), is not well defined. The norm is written ||z||_ρ = (z_1^ρ + ... + z_m^ρ)^{1/ρ} with no absolute values, and Algorithm 1 grid-searches ρ ∈ {1, 1.25, ..., 3}. Classifier logits are routinely negative. For non-integer ρ, z_j^ρ is not real-valued, so the expression cannot be computed on standard logits; even for ρ=1, the denominator γΣz_j + β can be negative, violating the positivity assumption used in Proposition 2. If the implementation actually uses (Σ|z_j|^ρ)^{1/ρ}, that is a different mapping, and the proof of Proposition 1 — which relies on Σ r_j^ρ = 1/γ^ρ and on real-valued powers of possibly negative r_j — must be restated for that norm. Because no source code is provided, the ECE values in Table 1 cannot be traced to any precisely specified function, so the empirical support for the central claim is untestable as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a post-hoc calibration method, ρ-Norm Scaling, which replaces the temperature in temperature scaling with a denominator γ||z||_ρ + β, where the norm is parameterized by a learnable ρ selected by grid search. The calibrator is optimized with a two-level loss that combines a bin-level square calibration error with an instance-level KL regularization between the pre- and post-calibration probability distributions. Experiments on CIFAR-10/100, SVHN, 102 Flower, and Tiny-ImageNet report substantially lower ECE and AdaECE than temperature scaling and other baselines, together with ablation studies that isolate the effect of the structure and the objective. The paper also states three propositions, the first bounding the calibrated confidence, the second and third giving order-preserving sufficient conditions for accuracy preservation.","tokens_in":14534,"tokens_out":10647,"duration_ms":83163,"significance":"If the empirical results hold, the proposed method would be a valuable addition to the post-hoc calibration toolbox: it is simple, accuracy-preserving, and appears to give large ECE reductions. The paper has several strengths: the problem framing is clear, the ablations in Tables 2 and 3 isolate the contributions of the structure and the objective, and the authors include multiple datasets and architectures, with five runs and standard deviations. The theoretical claims are meant to support the design, but as written the central construction is not well defined and one proposition is numerically false, which prevents the current version from being assessed reliably. The lack of source code and of any reporting of the fitted γ, β, and ρ values further undermines the testability of the empirical claims.","major_comments":[{"comment":"The ρ-norm in Eq. (2) is defined as ||z||_ρ = (z_1^ρ + ... + z_m^ρ)^{1/ρ} with no absolute values. Algorithm 1 grid-searches non-integer ρ ∈ {1, 1.25, ..., 3}, and classifiers routinely produce negative logits. For non-integer ρ, z_j^ρ is not real, so the expression is undefined on standard logits and the method as stated is not computable. If the implementation instead uses (Σ |z_j|^ρ)^{1/ρ}, that is a different mapping, and the proofs of Proposition 1 and the order-preserving argument must be restated for that norm. Since no source code is provided, the numerical results in Table 1 cannot be traced to any precisely specified function. This is the load-bearing issue for the paper's central claim.","section":"Methodology, Eq. (2) and Algorithm 1"},{"comment":"The proof of Proposition 1 is not valid as written. The constraint in Eq. (7) is Σ r_j^ρ = 1/γ^ρ without absolute values, but the extremal solutions in Eqs. (15)-(16) set r_j negative for j ≠ c, which is undefined for non-integer ρ. Moreover, the claimed bound is numerically false: take m = 2, ρ = 2, γ = 1. The true maximum of g_c under the constraint is 1/(1 + e^{-√2}) ≈ 0.805, while the upper bound from Eq. (3) evaluates to 1/(1 + e^{-(1/2)^{3/2}}) ≈ 0.587, which is below the true maximum. Proposition 1 therefore cannot be used to justify the confidence-restricting behavior of the method.","section":"Appendix, proof of Proposition 1 and Eq. (3)"},{"comment":"The proof of Proposition 2 is garbled. It contains the line \"γ√(Σ z_j^2)+β\" (Euclidean norm) instead of the ρ-norm used in Eq. (2), and the fragment \"p(zj)=e^{zj/a}/b\" is not a coherent derivation. The claim itself is true in the simple case where γ||z||_ρ + β > 0 (which holds for γ, β > 0 and a well-defined norm), because the denominator is positive and z_j → z_j/(positive constant) is strictly order-preserving. The proof needs to be rewritten cleanly for the actual ρ-norm in Eq. (2) (with absolute values if that is the intended definition).","section":"Appendix, proof of Proposition 2"},{"comment":"The paper never reports the fitted values of γ, β, or the selected ρ for each dataset and model. Table 4 shows only a subset of candidate ρ values for ResNet35 and does not state which ρ was selected by Algorithm 1. The reproducibility checklist states \"This paper lists all final (hyper-)parameters used for each model/algorithm in the paper's experiments (yes)\", but this is not the case. In addition, no statistical significance tests are reported (the checklist answers \"no\" on that item), so the claimed improvements over the second-best baseline are not assessed for significance. These omissions make the strong empirical claims difficult to verify.","section":"Experiments and Reproducibility Checklist"},{"comment":"The paper claims state-of-the-art calibration performance but omits the closest parametric baselines. Parameterized Temperature Scaling (Tomani, Cremers, and Buettner 2022) is cited in the related work but never evaluated, and Dirichlet calibration (Kull et al. 2019) is also absent from Table 1. The comparison in Table 1 is limited to Histogram Binning, temperature scaling, vector scaling, and TS-AvUC. Adding these closely related methods is necessary to support the \"state-of-the-art\" claim, especially because the proposed structure is a generalization of parameterized scaling.","section":"Experiments, Table 1 and Related Work"}],"minor_comments":[{"comment":"The KL term in Eq. (5) is written as Σ_j g_j (log g_j - log s_j), which is KL(g || s). The text says it is the \"KL divergence of the original output distribution s(z) concerning the calibrated probability distribution g(z)\", which would suggest KL(s || g). The direction should be clarified.","section":"Methodology, Eq. (5)"},{"comment":"The loop condition reads \"while ρ ∈ {1, . . . ,3}\" but the paper's grid search is {1, 1.25, ..., 3}. This is a typo that affects the reproducibility of Algorithm 1.","section":"Algorithm 1, line 3"},{"comment":"The notation \"(↓ 0.016)\" after some entries is not defined in the caption. The caption mentions bold for best results and relative improvements, but the parenthetical arrow and number are unexplained.","section":"Table 1"},{"comment":"The caption says \"Using sample-level SCE alone in post-calibration results in a significant deviation from the original distribution.\" SCE is a bin-level (set-level) objective, so \"sample-level\" is misleading; the sentence should say \"set-level\" or \"bin-level\".","section":"Figure 4 caption"},{"comment":"Equations (10) and (11) contain unresolved \"±\" and \"∓\" signs. The signs must be fixed and the derivation must be completed before the proof can be checked.","section":"Appendix, proof of Proposition 1, Eqs. (10)-(11)"}],"recommendation":"major_revision","confidential_remarks":"The paper appears to be a serious empirical study with a clear structure, but the central mapping is not well defined as written. The most serious issue is not merely a missing absolute value in the norm: Proposition 1 is numerically false, so the theoretical support for the method needs substantive rework. The empirical results could be salvageable, but without source code and without reporting the fitted parameters and selected ρ, the reported improvements are not verifiable. I would encourage the authors to resubmit after (i) properly defining the norm, (ii) correcting the proofs, (iii) reporting γ, β, and ρ for every experiment, and (iv) adding the missing parametric baselines. If these fixes are made, the paper may become acceptable, but in its current form the load-bearing claims are not supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the core idea—divide logits by a learnable ρ-norm plus bias, then softmax—is a reasonable extension of temperature scaling, and the KL regularizer for preserving the pre-calibration distribution is a sensible addition to squared calibration error. The ablations in Tables 2 and 3 do suggest both pieces help. If the numbers are reproducible, ECE reductions of 3–4× over TS on CIFAR-100 and Tiny-ImageNet would be a useful result for the calibration subfield.\n\nSecond, the paper is not in a publishable state. Equation (2) defines the ρ-norm as (Σ z_j^ρ)^{1/ρ} with no absolute values, and Algorithm 1 grid-searches ρ ∈ {1, 1.25, ..., 3}. Classifier logits are routinely negative, so for non-integer ρ this is not real-valued. The actual mapping evaluated is therefore not specified. The proofs inherit the problem: Proposition 1's Lagrangian uses r_j^ρ and derivatives that assume real powers; Proposition 2's proof in the appendix is a single garbled sentence and does not actually prove the claim. Proposition 3 is stated as a general criterion but it is essentially the intra order-preserving property from Rahimi et al. 2020, so it is not new.\n\nThe empirical section also has gaps. The closest baselines—parameterized temperature scaling (Tomani et al. 2022) and logit normalization (Wei et al. 2022)—are cited but never benchmarked, which weakens the state-of-the-art claim. No code is shipped, and the fitted γ, β, ρ values are never reported, so the reader cannot verify that the reported numbers come from the intended mapping. The validation-ECE selection of ρ is a minor concern, not a deal-breaker.\n\nThe fix is straightforward: rewrite the norm with absolute values, redo the proofs for that mapping, add the missing baselines, and release code. If the authors do that, this becomes a solid little paper. As it stands, the central construction is not computable as written.\n\nWho should engage? Anyone working on post-hoc calibration. It deserves a serious referee, but the referee should treat it as a major-revision candidate, not near-acceptance. My advice: send it to review, but insist on the above changes.","headline":"A plausible new parametric calibrator, but the norm in Eq. (2) is undefined for negative logits with non-integer ρ, the proofs are shaky, and the closest baselines are missing; fixable but needs real work.","tokens_in":15001,"tokens_out":3503,"would_cite":false,"duration_ms":28161,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A post-hoc calibrator that divides logits by a learnable ρ-norm reduces expected calibration error to 0.007–0.019 on standard image benchmarks while preserving classifier accuracy.","keywords":["confidence calibration","expected calibration error","temperature scaling","post-hoc calibration","ρ-norm scaling","KL divergence regularization","order-preserving mapping","output amplitude"],"falsifier":"Take a classifier output vector with a negative entry, set $\\rho=1.5$, and evaluate $r_j = z_j/(\\gamma\\|z\\|_\\rho + \\beta)$ as written: the norm term $(\\sum_j z_j^\\rho)^{1/\\rho}$ is not a real number. If the implementation still runs, it must be using $|z_j|^\\rho$, in which case Proposition 1's bounds need to be re-derived; inspecting the released code or running this calculation settles which mapping is actually being evaluated.","tokens_in":14066,"feed_emoji":"🎯","tokens_out":10110,"duration_ms":68613,"temperature":0.7,"pith_summary":"This paper claims that the overconfidence of modern classifiers can be corrected after training by a simple parametric remapping: divide each logit by a learnable ρ-norm of the logit vector (plus constants) before applying softmax. The authors argue that this 'ρ-Norm Scaling' expands the expressiveness of temperature scaling and directly counteracts the growing output amplitude that pushes softmax probabilities toward one-hot saturation. They further claim that adding an instance-level KL-divergence penalty between pre- and post-calibration probability distributions to the bin-level squared calibration error prevents the calibrator from losing per-sample information. If correct, the method yields substantially lower ECE and AdaECE than temperature scaling on CIFAR-10/100, Tiny-ImageNet, SVHN, and 102 Flower, with accuracy unchanged because the mapping is order-preserving.","feed_headline":"Calibration error drops 3-4x with ρ-norm scaling","feed_subtitle":"Dividing logits by a learnable ρ-norm plus a KL regularizer cuts ECE to 0.007-0.019 on image benchmarks.","key_machinery":"The central object is the ρ-norm-scaling softmax: $g_c(z) = e^{r_c}/\\sum_j e^{r_j}$ with $r_j(z) = z_j/(\\gamma\\|z\\|_\\rho + \\beta)$, where $\\|z\\|_\\rho = (\\sum_j z_j^\\rho)^{1/\\rho}$. The ρ-norm term normalizes the logit amplitude before the exponential, so high-magnitude logits no longer saturate the softmax. The second mechanism is the multi-level objective $l = (\\mathrm{acc}(D)-\\mathrm{conf}(D))^2 + \\alpha \\sum_i \\mathrm{KL}(g(z_i) \\| s(z_i))$, where the KL term anchors the calibrated distribution to the original softmax distribution at the instance level.","core_discovery":"The central claim is that the calibration mapping $r_j(z) = z_j/(\\gamma\\|z\\|_\\rho + \\beta)$, with $\\rho$ selected by grid search and $\\gamma, \\beta$ learned, produces well-calibrated confidence while preserving classification decisions. Proposition 1 gives upper and lower bounds showing that for finite $\\gamma$ the confidence is pulled away from 1, mitigating overconfidence; Proposition 2 shows the mapping is strictly order-preserving for any $\\gamma, \\beta > 0$, so accuracy is unchanged; Proposition 3 generalizes this to any positive sample-dependent temperature $\\sigma(z)$. The authors also claim that optimizing this calibrator with $l = \\mathrm{SCE} + \\alpha \\mathrm{KL}$, where SCE is a batch-level squared calibration error and KL is the KL divergence from the calibrated distribution to the original softmax distribution, retains instance-level information and yields the reported state-of-the-art calibration.","pith_inferences":["The paper never says what happens when logits are negative and $\\rho$ is non-integer; if the implementation uses $(|z_j|^\\rho)^{1/\\rho}$, the bounds in Proposition 1 need to be re-derived for that absolute-value norm, and the reported results describe that mapping, not the one written.","Because the order-preserving property holds for any positive $\\sigma(z)$, the same idea could be extended to per-class or input-dependent temperature functions without losing accuracy—a natural next step the paper leaves implicit.","The KL regularizer could be replaced by other distributional divergences (e.g., Jensen-Shannon) or applied in logit space; the paper only tests KL, so the choice's specificity is untested.","As a practical extension, one could test whether the same gains appear on language models or long-tail datasets, where output amplitude and logit distributions differ from image classifiers."],"forward_implications":["Corollary of the experiments: post-hoc calibration can be improved without retraining the classifier, so accuracy is preserved while ECE drops sharply.","Proposition 1 implies ρ-Norm Scaling is guaranteed not to produce one-hot-like confidence for finite $\\gamma$, which can be used as a design check for calibrators.","Proposition 3 gives a general criterion—any positive logit scaling $\\sigma(z)$ preserves accuracy—so other amplitude-controlling mappings beyond ρ-norms could be explored.","The KL regularization offers a way to inject instance-level information into any bin-level calibration objective, not just SCE.","The method is cheap: only three parameters ($\\gamma$, $\\beta$, $\\rho$) with grid search over $\\rho \\in \\{1, 1.25, \\ldots, 3\\}$."],"supporting_citations":[{"why":"Documents that modern neural networks are overconfident and establishes ECE as the standard calibration metric that the paper targets.","marker":"Guo et al. 2017"},{"why":"Provides the intra order-preserving property and accuracy-preserving calibration framework that ρ-Norm Scaling's decision invariance builds on.","marker":"Rahimi et al. 2020"},{"why":"Introduces parameterized temperature scaling, the direct baseline and expressive-power target that ρ-Norm Scaling extends.","marker":"Tomani, Cremers, and Buettner 2022"},{"why":"Shows that bin-level calibration objectives like AvUC lose instance-level information, motivating the KL regularizer in the optimization objective.","marker":"Krishnan and Tickoo 2020"},{"why":"Discusses hard binning problems in calibration objectives, supporting the paper's choice of a squared calibration error and soft objective.","marker":"Karandikar et al. 2021"},{"why":"Defines ECE, the metric used for evaluation and for grid-search selection of $\\rho$.","marker":"Naeini, Cooper, and Hauskrecht 2015"},{"why":"Provides Beta calibration, a parametric baseline, and the broader idea of parametric post-hoc calibration structures.","marker":"Kull, Silva Filho, and Flach 2017"},{"why":"Introduces Platt scaling, the foundational parametric post-hoc calibration method that the paper's line of work extends.","marker":"Platt et al. 1999"}],"fun_headline_variants":["ρ-norm scaling cuts calibration error 3-4x","Learnable ρ-norm calibrator preserves accuracy","New calibration method reduces ECE to 0.007–0.019","Parametric ρ-norm scaling improves confidence calibration","Calibrate without losing accuracy: ρ-norm scaling"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The formula $(z_1^\\rho + \\cdots + z_m^\\rho)^{1/\\rho}$ is used even when some classifier output values $z_j$ are negative and $\\rho$ is not a whole number, which is only well-defined if absolute values are taken; the paper never states that, so the exact mapping being evaluated is ambiguous.","fun_headline_variants_meta":{"raw":{"variants":["ρ-norm scaling cuts calibration error 3-4x","Learnable ρ-norm calibrator preserves accuracy","New calibration method reduces ECE to 0.007–0.019","Parametric ρ-norm scaling improves confidence calibration","Calibrate without losing accuracy: ρ-norm scaling"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000243,"raw_usage":{"total_tokens":1513,"prompt_tokens":914,"completion_tokens":599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":517}},"tokens_in":530,"tokens_out":599,"duration_ms":3885,"temperature":1.0,"reasoning_tokens":517,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:58:03.181483+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a classifier output vector with a negative entry, set $\\rho=1.5$, and evaluate $r_j = z_j/(\\gamma\\|z\\|_\\rho + \\beta)$ as written: the norm term $(\\sum_j z_j^\\rho)^{1/\\rho}$ is not a real number. If the implementation still runs, it must be using $|z_j|^\\rho$, in which case Proposition 1's bounds need to be re-derived; inspecting the released code or running this calculation settles which mapping is actually being evaluated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents that modern neural networks are overconfident and establishes ECE as the standard calibration metric that the paper targets."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the intra order-preserving property and accuracy-preserving calibration framework that ρ-Norm Scaling's decision invariance builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces parameterized temperature scaling, the direct baseline and expressive-power target that ρ-Norm Scaling extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows that bin-level calibration objectives like AvUC lose instance-level information, motivating the KL regularizer in the optimization objective."},{"cited_title":"C.; and Roelofs, B","cited_arxiv_id":null,"evidence_quote":"Discusses hard binning problems in calibration objectives, supporting the paper's choice of a squared calibration error and soft objective."},{"cited_title":"P.; Cooper, G.; and Hauskrecht, M","cited_arxiv_id":null,"evidence_quote":"Defines ECE, the metric used for evaluation and for grid-search selection of $\\rho$."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Beta calibration, a parametric baseline, and the broader idea of parametric post-hoc calibration structures."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces Platt scaling, the foundational parametric post-hoc calibration method that the paper's line of work extends."}],"review_version":1}