{"id":"280d3843-efd4-4c0b-91f1-82da87686891","arxiv_id":"2502.03023","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Using the same data for tuning and calibration in conformal prediction introduces only small coverage bias for simple tuners, and this 'tuning bias' scales up with parameter count and down with calibration set size.","lead":"This paper studies what happens when conformal prediction uses the same data to tune parameters and to compute coverage thresholds, and reports that the resulting coverage bias is usually small for simple tuning. It proposes a scaling law: the bias grows with the number of tuned parameters and shrinks with calibration set size, and offers bounds to support it.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma D.5 is false: continuous bounded parametric score families can have infinite VC dimension, so the infinite-parameter scaling-law upper bound is unproven.","rationale":"The central claim of the paper is a rigorous parametric scaling law for tuning bias. The finite-parameter union bound (Proposition 4.2) is a standard and plausible argument, and the empirical scaling trends are useful. However, the infinite-parameter statement is the part needed for the headline scaling law and for the confidence-calibration corollaries such as Corollary 4.7. Lemma D.5's proof argues from continuity that d+2 points cannot be shattered; this is a known false intuition, and the sine-family counterexample shows the lemma's assumptions do not control VC dimension. Because Proposition 4.6 is a direct application of Lemma D.5, the rigorous proof of the scaling law over infinite parameter spaces is missing. This is an internal invalidity in the proof, not a disagreement with consensus, and it justifies the reader's rejection of the paper as submitted while leaving open a salvageable revision: the empirical observations and the finite-parameter bound may survive, but the infinite-parameter scaling law needs either a corrected complexity measure or a restriction to score families whose threshold classes have provably finite VC dimension.","tokens_in":25692,"tokens_out":9831,"duration_ms":90959,"concrete_test":"Analytical refutation: set d=1, X = {sqrt(p_1),...,sqrt(p_m)} with p_i the first m primes, Y={0}, S_lambda(sqrt(p_i),0)=sin(lambda sqrt(p_i)), and consider H={1{S_lambda <= t}: lambda in R, t in R}. For m=3, enumerate all 8 sign patterns; for each, Kronecker's theorem (rational independence of sqrt(p_i)) guarantees a lambda such that sin(lambda sqrt(p_i)) has the required sign, with t=0. If this succeeds, VC(H) > 2 = d+1, Lemma D.5 is refuted, and Proposition 4.6 must either be restricted to score families with proven finite VC dimension or given a new proof.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 4.6 and Corollary 4.7 rest entirely on Lemma D.5, which asserts VC(H_lambda) <= d+1 for every score function S_lambda that is continuous and bounded in lambda. This is false even for d=1. Take S_lambda(x) = sin(lambda x) on rationally independent points x_1,...,x_m. By Kronecker's theorem, (lambda x_i mod 2pi)_i is dense in the m-torus, so for every sign pattern there is a lambda with sin(lambda x_i) <= 0 exactly on the chosen subset; threshold t=0 realizes the pattern. Hence H_lambda has infinite VC dimension, not <=2. Lemma D.6 then gives no bound, so Proposition 4.6's C sqrt((d+1)/n) is unsupported and Corollary 4.7's TuningBias(CTS) <= TuningBias(CVS) does not follow. There is also a smaller gap in all cases: Theorem 4.1 bounds CovGap(C) by ER_lambda + eps_{alpha,n}, while TuningBias(C) = CovGap(C) - CovGap(C_hold-out); since the split-conformal gap need not equal eps_{alpha,n}, the claimed TuningBias <= ER_lambda acquires an additive O(1/n) term. That term does not change the scaling rate but means the displayed Proposition 4.2 bound is not exactly derived either.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the coverage gap ('tuning bias') incurred when the same dataset is used both to tune parameters of a conformal score function and to calibrate the conformal threshold. The authors report experiments on CIFAR-100 and ImageNet with RAPS, SAPS, score aggregation, temperature scaling, vector scaling, C-Adapter, and ConfTr fine-tuning, observing that tuning bias is small for low-complexity tuning and that it increases with the number of parameters and decreases with calibration-set size. The theoretical part models tuning as constrained ERM and bounds the coverage gap by an empirical process supremum R_Lambda plus a split-conformal term epsilon_alpha,n (Theorem 4.1). Finite parameter spaces are handled by a DKW union bound (Proposition 4.2), and infinite Lambda subset R^d by a VC-dimension bound (Proposition 4.6), leading to Corollary 4.7 comparing temperature scaling with vector scaling and Proposition 5.2 on order-preserving regularization. I find the empirical study informative but the theoretical core contains unsupported and false steps, detailed below.","tokens_in":26011,"tokens_out":13907,"duration_ms":127621,"significance":"If the theoretical claims were correct, the paper would provide a useful justification for reusing calibration data for low-complexity tuning and a first rigorous explanation of the observed scaling behavior. The empirical study is extensive and reproducible (code link is provided), and the finite-Lambda DKW argument is a sensible use of standard inequalities; the exact zero tuning bias for binary temperature scaling (Proposition 5.1) is a nice structural observation. However, the headline theoretical contributions are not established: the infinite-dimensional bound rests on a false VC-dimension lemma, and the comparison corollaries infer an ordering of biases from an ordering of upper bounds. Since these flaws are load-bearing for the claimed parametric scaling law, the paper in its current form does not support its abstract.","major_comments":[{"comment":"The statement 'TuningBias(C) <= ER_Lambda' does not follow from Theorem 4.1. The theorem gives CovGap(C) <= ER_Lambda + epsilon_alpha,n, while TuningBias(C) = CovGap(C) - CovGap(C_holdout). The argument would require CovGap(C_holdout) >= epsilon_alpha,n, but the split-conformal gap is only bounded above by about 1/(n+1) (Theorem 2.2) and can be much smaller; epsilon_alpha,n is an upper-bound constant, not the exact hold-out gap. Consequently Proposition 4.2 and Proposition 4.6 are not derived as stated; at best one obtains TuningBias(C) <= ER_Lambda + epsilon_alpha,n, an additive O(1/n) term. This does not change the asymptotic rate for fixed |Lambda|, but the displayed bounds should be corrected.","section":"Section 4.1 (after Theorem 4.1)"},{"comment":"Lemma D.5 is false as stated. Take Lambda = R and S_lambda(x) = sin(lambda x). For any finite set of rationally independent real numbers x_1,...,x_m (with 1, x_1,...,x_m linearly independent over Q), Kronecker's theorem implies that (lambda x_1 mod 2pi, ..., lambda x_m mod 2pi) is dense in the m-torus. Hence for every sign pattern there exists a lambda with sin(lambda x_i) > 0 on the chosen positive set and sin(lambda x_i) < 0 on the chosen negative set; the threshold t = 0 then realizes the pattern with 1{sin(lambda x_i) <= 0}. Thus H_Lambda shatters m points for every m and VC(H_Lambda) = infinity, although S_lambda is continuous and bounded in lambda. The same counterexample refutes the one-dimensional Lemma D.3, which claims VC(H_Lambda) <= 2. Since Proposition 4.6 is proved as a direct application of Lemmas D.5 and D.6, the bound C sqrt((d+1)/n) is unsupported, and Corollary 4.7 inherits this failure.","section":"Appendix D, Lemma D.5 (and Lemma D.3)"},{"comment":"Even if Proposition 4.6 were valid, Corollary 4.7 does not follow. The proposition supplies upper bounds TuningBias(C_TS) <= C sqrt(2/n) and TuningBias(C_VS) <= C sqrt((2K+1)/n); a smaller upper bound for TS does not imply TuningBias(C_TS) <= TuningBias(C_VS). The same invalid transfer appears in Appendix K in the proof of Proposition 5.2: from H_Lambda,op subset H_Lambda the authors conclude that the empirical process supremum over the smaller class is no larger, 'which means' TuningBias(C_op) <= TuningBias(C). This requires a direct argument about the tuning biases themselves, not merely about their upper bounds. Proposition 5.1's claim 0 <= TuningBias(C_VS) also lacks a lower-bound proof; a finite-sample difference of coverage gaps can be negative.","section":"Corollary 4.7 and Proposition 5.2"}],"minor_comments":[{"comment":"The Figure 5 caption says the tuning bias of ConfTr (ft.) is 'generally small', while the plotted gaps and the Appendix B.5 text describe it as 'significantly different' from the hold-out setting; these statements should be reconciled.","section":"Appendix B.5 and Figure 5 caption"},{"comment":"Lemma K.3 (the characterization of order-preserving matrix scaling) is stated without a proof; the surrounding text only remarks on it, so either a proof or an explicit conjecture label is needed.","section":"Appendix K"},{"comment":"The main text does not clearly state which dataset is used for the scaling-law experiments; Figure 2 caption says ImageNet, whereas Section 3.1 is introduced with CIFAR-100. Please clarify the dataset for each experiment.","section":"Section 3.2 and Figure 2"},{"comment":"There are several typos and corrupted labels, including 'tunig' in Appendix B.2, 'Paramters' in Figure 3d caption, and 'Image/glyph1197et' in Appendix B.1 captions.","section":"General presentation"},{"comment":"The theorem uses T subset R in the definition of R_Lambda without defining T; specify the admissible threshold set explicitly.","section":"Theorem 4.1"}],"recommendation":"reject","confidential_remarks":"The theoretical core is the selling point of the paper. Given the false Lemma D.5 and the invalid transfer from upper bounds to tuning-bias orderings, I do not see how a routine revision can preserve the stated claims. A future version that restricts the VC analysis to the actual score families, corrects the additive epsilon_alpha,n term, and either proves lower bounds or carefully rephrases Corollaries 4.7 and 5.2 as comparisons of upper bounds rather than of tuning biases could be viable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: the paper has a genuinely useful empirical observation and a correct finite-parameter union-bound argument, but the headline theoretical claim — the parametric scaling law for infinite parameter spaces — is not proven. Lemma D.5 is false as stated, and there is a logical gap in deriving TuningBias ≤ ER_Λ from Theorem 4.1.\n\nWhat is new: framing tuning bias as an empirical process over (λ, t) is a natural idea, but the paper works it out systematically, and the finite-parameter bound (Proposition 4.2) is a clean DKW + union bound. The experiments are reasonably extensive, and the observation that simple tuners like RAPS, SAPS, and temperature scaling introduce small tuning bias, while high-dimensional tuners like vector scaling and ConfTr(ft.) do not, is useful and reproducible (code is provided). Proposition 5.1 — for binary classification, temperature scaling has zero tuning bias — is a nice, clean result, and the proof via order preservation holds up.\n\nWhere it goes wrong. First, Lemma D.5 claims VC(H_Λ) ≤ d+1 for any score family continuous and bounded in λ. That is false even for d=1. S_λ(x) = sin(λx) on rationally independent points gives every sign pattern by Kronecker density, so the threshold class has infinite VC dimension; the proof's assertion that continuity prevents d+2 points from being shattered is simply wrong. Proposition 4.6 and Corollary 4.7 rest entirely on this lemma, so the infinite-parameter scaling law is unproven. Lemma D.3 has the same flaw in one dimension.\n\nSecond, Theorem 4.1 gives CovGap(C) ≤ ER_Λ + ε_{α,n}. TuningBias is CovGap(C) − CovGap(C_holdout), and the holdout gap is not ε_{α,n}; it can be much smaller. So the displayed bound in Proposition 4.2 is not exactly derived; you get an additive O(1/n) term. That does not change the rate, but it means the theorem as stated is not what the proof establishes.\n\nThird, Corollary 4.7 and Proposition 5.2 infer orderings of actual biases from orderings of upper bounds or from subset relations on the hypothesis class. That is invalid: a smaller ER bound does not imply a smaller CovGap(C) − CovGap(C_holdout). Proposition 5.2's conclusion may be empirically true, but the proof does not show it.\n\nOverall, the paper is not a throwaway. The empirical finding and the finite-space bound deserve to be cited, and the binary-temperature-scaling result is solid. But as a theory paper, the central infinite-parameter law is unsupported. A serious referee should engage, and the authors should be pushed to add real assumptions on the score family under which the VC bound holds, restrict the theory to finite parameter spaces, or reposition the scaling law as an empirical claim.","headline":"Empirically plausible and the finite-parameter bound is fine, but the infinite-parameter scaling law rests on a false VC-dimension lemma and on a gap between coverage-gap bounds and actual tuning bias.","tokens_in":26515,"tokens_out":4538,"would_cite":true,"duration_ms":43640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G15","62G20","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"Using the same data for tuning and calibration in conformal prediction introduces a tuning bias bounded by a parametric scaling law: it grows with parameter-space complexity and shrinks with calibration-set size.","keywords":["conformal prediction","tuning bias","exchangeability","coverage gap","empirical process","VC dimension","parametric scaling law","confidence calibration"],"falsifier":"Find a score function $S_\\lambda(x,y)$ that is continuous and bounded in $\\lambda\\in\\mathbb R$ and three fixed samples whose eight label patterns are all realized by varying $\\lambda$ and the threshold $t$ (for instance $S_\\lambda(x_i)=\\cos(a_i\\lambda)$ with well-separated frequencies $a_i$). Exhibiting such a triple refutes the lemma behind the $\\sqrt{(d+1)/n}$ bound, so the infinite-parameter scaling law would not follow from the proof as written.","tokens_in":25447,"feed_emoji":"📏","tokens_out":7065,"duration_ms":64712,"temperature":0.7,"pith_summary":"The paper asks when it is safe to use the same dataset for tuning and calibration in conformal prediction, a practice that breaks the exchangeability assumption behind coverage guarantees. Its central claim is that the resulting coverage gap, called the tuning bias, obeys a parametric scaling law: it grows with the complexity of the parameter space and shrinks with the size of the calibration set. For a finite set of $|\\Lambda|$ parameter choices the paper proves a bound of order $\\sqrt{\\log(2|\\Lambda|)/(2n)}$, and for a $d$-dimensional parameter space a bound of order $\\sqrt{(d+1)/n}$. If the claim holds, practitioners in data-scarce settings can reuse one dataset for simple tuning at small coverage cost, and the main danger is high-dimensional parameter spaces such as vector scaling or full-layer fine-tuning.","feed_headline":"Tuning bias in conformal prediction follows a parametric scaling law","feed_subtitle":"New bounds show the coverage gap grows with parameter count and shrinks with calibration size.","key_machinery":"The load-bearing object is the threshold class $H_\\Lambda = \\{1\\{S_\\lambda(x,y)\\le t\\}: \\lambda\\in\\Lambda,\\ t\\in\\mathbb R\\}$, the indicator functions of events that a tuned score falls below a threshold. Tuning bias is bounded by the expected supremum of the empirical process indexed by this class, $E\\sup_{g\\in H_\\Lambda} |(1/n)\\sum_i g(x_i,y_i) - E[g(x_{\\text{test}},y_{\\text{test}})\\mid D_{\\text{cal}}]|$. The finite-parameter case is controlled by the Dvoretzky–Kiefer–Wolfowitz inequality plus a union bound over $|\\Lambda|$; the infinite-parameter case is controlled by the VC dimension of $H_\\Lambda$, with the paper claiming $\\mathrm{VC}(H_\\Lambda)\\le d+1$ for $\\Lambda\\subset\\mathbb R^d$ under continuity and boundedness. The same machinery yields corollaries for RAPS, score selection, temperature versus vector scaling, and order-preserving regularization, which shrinks the effective parameter space.","core_discovery":"The discovery is a formal quantification of tuning bias: when the tuning parameter $\\hat\\lambda$ is picked and the threshold $\\hat t$ is calibrated on the same i.i.d. sample, the extra coverage gap is bounded by the supremum of empirical-process fluctuations over the class $H_\\Lambda=\\{1\\{S_\\lambda(x,y)\\le t\\}:\\lambda\\in\\Lambda,\\ t\\in\\mathbb R\\}$. For finite $\\Lambda$, a union bound with the Dvoretzky–Kiefer–Wolfowitz inequality gives $\\text{TuningBias}(C) \\le \\sqrt{\\log(2|\\Lambda|)/(2n)} + 1/(\\sqrt{2n}\\sqrt{\\log(2|\\Lambda|)})$. For infinite $\\Lambda\\subset\\mathbb R^d$, a VC-dimension bound gives $\\text{TuningBias}(C) \\le C\\sqrt{(d+1)/n}$. The proof identifies parameter-space complexity, not the number of parameters per se, as the driver of tuning bias, and the applications show that temperature scaling has a smaller bound than vector scaling, with zero tuning bias in binary classification. The authors take these bounds as rigorous support for the scaling law they observe empirically across RAPS, SAPS, score aggregation, C-Adapter, and confidence calibration methods.","pith_inferences":["Implicit consequence: the proved bounds are uniform over score distributions and likely loose for typical neural-network scores, so a data-dependent or Rademacher-style bound could show same-set reuse is safe in even more cases, or reveal settings where the empirical process is larger than $\\sqrt{d/n}$.","Testable design rule: compare the estimated tuning-bias bound for a specific method and calibration size against the coverage slack $\\varepsilon_{\\alpha,n}$ already tolerated; reuse is justified when the bound is a small fraction of that slack.","Testable extension: the theory should extend to early stopping and model selection in regression, which the paper treats only empirically; verifying the same scaling there would test whether the law is universal across conformal prediction pipelines.","Computational check: for random low-dimensional score families, search for the largest shattered set; a counterexample to the $d+1$ VC-dimension lemma would confine the scaling law to the finite-parameter regime."],"forward_implications":["Same-set tuning is safe only when the parameter space is simple: the bound is practically small for a handful of candidates and thousands of calibration points.","High-dimensional tuning methods, such as vector scaling with $2K$ parameters, carry a coverage penalty that grows like $\\sqrt{(d+1)/n}$, matching the empirical failures the paper reports.","Temperature scaling dominates vector scaling in tuning bias; in binary classification temperature scaling has exactly zero tuning bias, so no hold-out split is needed for that case.","Order-preserving regularization reduces the effective parameter dimension (from $2K$ to $2$ for vector scaling, and from $K^2+K$ to $K+2$ for matrix scaling) and therefore lowers the tuning bias bound.","Increasing calibration size $n$ reduces the bias at rate $1/\\sqrt n$; the settings where data reuse is most tempting are exactly the settings where the bound is largest."],"supporting_citations":[{"why":"Supplies the Dvoretzky–Kiefer–Wolfowitz inequality used to union-bound the finite-parameter empirical process.","marker":"Dvoretzky et al. (1956)"},{"why":"Provides the tight constant version of the Dvoretzky–Kiefer–Wolfowitz inequality used in the tail bound.","marker":"Massart (1990)"},{"why":"Defines VC dimension and the uniform-convergence machinery used for the infinite-parameter threshold class.","marker":"Vapnik & Chervonenkis (1971)"},{"why":"Frames the quantity $R_\\Lambda$ as an empirical process indexed by the class $H_\\Lambda$.","marker":"van der Vaart & Wellner (1996)"},{"why":"Supplies the theorem converting VC dimension into the empirical-process bound used in Lemma D.6.","marker":"Vershynin (2018)"},{"why":"Provides the exchangeable coverage-gap baseline and the non-exchangeable context that motivates the tuning bias decomposition.","marker":"Barber et al. (2023)"},{"why":"Defines the RAPS method whose finite-parameter tuning bias is bounded in Corollary 4.3.","marker":"Angelopoulos et al. (2021)"},{"why":"Defines temperature scaling and vector scaling, the methods whose infinite-parameter bounds are compared in Corollary 4.7.","marker":"Guo et al. (2017)"},{"why":"Provides the score-selection framework whose coverage-gap bound appears as a special case in Corollary 4.4.","marker":"Yang & Kuchibhotla (2024)"}],"fun_headline_variants":["Scaling law bounds tuning bias in conformal prediction","Tuning bias grows with complexity, shrinks with calibration size","Tuning bias follows parametric scaling law","Parametric scaling law for tuning bias proven"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The infinite-parameter bound relies on the assumption that a continuous family of score functions cannot fit arbitrary label patterns on more than $d+1$ points; oscillatory families can violate this, and the paper's proof of this assumption is not valid in general.","fun_headline_variants_meta":{"raw":{"variants":["Scaling law bounds tuning bias in conformal prediction","Tuning bias grows with complexity, shrinks with calibration size","Tuning bias follows parametric scaling law","Parametric scaling law for tuning bias proven"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000696,"raw_usage":{"total_tokens":3150,"prompt_tokens":953,"completion_tokens":2197,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":569,"completion_tokens_details":{"reasoning_tokens":2137}},"tokens_in":569,"tokens_out":2197,"duration_ms":15916,"temperature":1.0,"reasoning_tokens":2137,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T10:09:43.783884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Find a score function $S_\\lambda(x,y)$ that is continuous and bounded in $\\lambda\\in\\mathbb R$ and three fixed samples whose eight label patterns are all realized by varying $\\lambda$ and the threshold $t$ (for instance $S_\\lambda(x_i)=\\cos(a_i\\lambda)$ with well-separated frequencies $a_i$). Exhibiting such a triple refutes the lemma behind the $\\sqrt{(d+1)/n}$ bound, so the infinite-parameter scaling law would not follow from the proof as written.","supporting_citations":[{"cited_title":"Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator","cited_arxiv_id":null,"evidence_quote":"Supplies the Dvoretzky–Kiefer–Wolfowitz inequality used to union-bound the finite-parameter empirical process."},{"cited_title":"The tight constant in the Dvoretzky-Kiefer-Wolfowitz inequality","cited_arxiv_id":null,"evidence_quote":"Provides the tight constant version of the Dvoretzky–Kiefer–Wolfowitz inequality used in the tail bound."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines VC dimension and the uniform-convergence machinery used for the infinite-parameter threshold class."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames the quantity $R_\\Lambda$ as an empirical process indexed by the class $H_\\Lambda$."},{"cited_title":"F., Cand \\`e s, E","cited_arxiv_id":null,"evidence_quote":"Provides the exchangeable coverage-gap baseline and the non-exchangeable context that motivates the tuning bias decomposition."},{"cited_title":"N., Bates, S., Jordan, M., and Malik, J","cited_arxiv_id":null,"evidence_quote":"Defines the RAPS method whose finite-parameter tuning bias is bounded in Corollary 4.3."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines temperature scaling and vector scaling, the methods whose infinite-parameter bounds are compared in Corollary 4.7."},{"cited_title":"and Kuchibhotla, A","cited_arxiv_id":null,"evidence_quote":"Provides the score-selection framework whose coverage-gap bound appears as a special case in Corollary 4.4."}],"review_version":1}