{"id":"bff65110-43af-43f4-a162-14d1762fdc91","arxiv_id":"2502.00818","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"ECI adds a smoothed error-quantification term to the online conformal update rule and proves distribution-free long-run coverage bounds, yielding tighter prediction sets on real time-series benchmarks.","lead":"This paper introduces a new online conformal prediction method, ECI, that adjusts prediction interval thresholds using both the binary miss indicator and the continuous distance between a new score and the threshold. The method aims to react faster to distribution shifts so that prediction intervals stay at the target coverage while being narrower than existing approaches.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's coverage guarantee is proven for a projected update q_t=max(q_t,0), not for the ECI update in Eq. (5); since negative thresholds are possible under Eq. (5), the central guarantee is not established for the algorithm as defined.","rationale":"The reader's weakest assumption is exactly where the paper's core claim is least secure. Theorem 1 is the only result giving the advertised blockwise 'miscoverage then N-1 coverage' dynamics, and it is the stated basis for the long-run guarantee in Eq. (2). The proof in Appendix B.2 silently imposes q_t = max(q_t,0), while the algorithm defined in Eq. (5) does not. Since Proposition 1 explicitly allows q_t < 0, and a negative threshold forces miscoverage for nonnegative scores, this is a substantive change to the feedback path, not a cosmetic rewrite. The additional inconsistencies in N (floor vs ceiling) and the reversed inequality in the proof's induction reinforce that Theorem 1, as written, cannot be checked line-by-line by a reader. I do not take this as evidence of bad faith: the empirical results are plausible, code is released, and Theorem 2 is derived from the unprojected update (5). But Theorem 2 only gives an approximate finite-sample bound whose second term is a constant controlled by c; it does not imply the exact blockwise guarantee or the equality claimed in Eq. (2). Thus the precise theoretical contribution currently applies to a variant that is not defined, not implemented, and not evaluated. The likely fix is either to prove the same block bound for the unprojected recurrence using the sharper lower bound q_t >= -(alpha+lambda)eta, or to state explicitly that the implemented algorithm clips thresholds and prove the guarantee for that version. Both are within reach, so conditional acceptance with requested revisions remains the appropriate verdict.","tokens_in":27438,"tokens_out":9393,"duration_ms":99278,"concrete_test":"With N=10, alpha=0.1, B=1, eta=21, and c=0.001 (satisfying Theorem 1's stated conditions), simulate Eq. (5) exactly from q_1=0 and choose scores adversarially to drive q_t negative--for example, set s_t=0 whenever q_t is small or negative--then record all miscoverage indicators in sliding 10-step windows. If any window contains two or more miscoverage events, Theorem 1 as stated for the unprojected update is false; if no such window appears over an exhaustive or Monte-Carlo search of feasible score sequences, the clipping may be a removable proof artifact. Separately, inspect the released code to determine whether q_t is clipped in the reported experiments; if it is, the paper must state this and the guarantee should be attributed to the clipped variant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim is Theorem 1, which promises that the ECI update (5) has at most one miscoverage in every N consecutive steps. The proof in Appendix B.2 begins: 'we set q_t to be max{q_t,0} after each update (which does not affect the validity of our proof)'. This is not a no-op. Proposition 1, proved for the stated update (5), only gives q_t >= -(alpha+lambda)M_{t-1}, so q_t can be negative under (5). For nonnegative scores, a negative threshold makes the prediction set empty and forces a miscoverage, changing the feedback dynamics that the theorem analyzes. No clipping is stated in Eq. (5), in the algorithm descriptions, or in the experimental setup, so the guarantee is proven for a different algorithm than the one implemented and evaluated. The proof also has internal inconsistencies: the appendix statement uses N = ceil(1/alpha) while the main text uses N = floor(1/alpha), and the proof line 'k <= N-1, alpha >= 1/N' has the inequality reversed (the needed condition is alpha <= 1/N, which holds for the floor but not the ceiling definition). These matter because the key lower bound eta(1-k alpha) >= eta/N depends on which N is used. Until Theorem 1 is re-derived for the unprojected update (5), the advertised long-term miscoverage control is unproven for the method as presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Error-quantified Conformal Inference (ECI), an online conformal method that updates the threshold q_t via q_{t+1} = q_t + eta[err_t - alpha + (s_t - q_t) nabla f(s_t - q_t)], adding a smooth error-quantification term to the binary feedback used by ACI/OGD. The authors state two distribution-free results: Theorem 1, claiming that with a fixed learning rate every miscoverage step is followed by N-1 coverage steps (yielding a long-run miscoverage bound), and Theorem 2, a finite-sample averaged-miscoverage bound for arbitrary adaptive learning rates. Experiments on stock, electricity, Delhi temperature, and synthetic changepoint data compare ECI and its cutoff/integral variants against ACI, OGD, SF-OGD, decay-OGD, and PID, reporting comparable coverage with generally shorter prediction intervals. The main caveat, detailed below, is that the proof of Theorem 1 is carried out for a projected update q_t <- max(q_t,0), not for the update in Eq. (5) that is implemented and evaluated.","tokens_in":27714,"tokens_out":10932,"duration_ms":113778,"significance":"The core idea is practically appealing: using the signed distance between the score and the threshold to modulate the update can plausibly yield faster adaptation and tighter sets than binary-only feedback. If Theorem 1 can be established for the actual update rule, the paper would be a useful contribution to online conformal inference, related to but distinct from Conformal PID. The paper should be credited for releasing code, testing multiple datasets and base predictors, including ablations on the scale parameter c and window length w, and attempting distribution-free guarantees for both fixed and adaptive learning rates. At present, the central theoretical guarantee is not proved for the algorithm that is actually implemented and evaluated, so the strength of the contribution depends on repairing the proof or adjusting the algorithm.","major_comments":[{"comment":"The proof of Theorem 1 begins in Appendix B.2 by declaring 'we set q_t to be max{q_t,0} after each update (which does not affect the validity of our proof)'. This is not a harmless convention. Proposition 1 in Appendix B.1 explicitly permits q_t < 0, and for nonnegative scores a negative threshold makes the prediction set empty and sets err_t = 1, which changes the future updates. Neither Eq. (5) nor the experimental description contains this projection. Therefore the statement that Theorem 1 applies to 'the prediction sets generated by (5)' is not what is proved; the long-term miscoverage guarantee is established only for a clipped variant. This is the central load-bearing issue and should be fixed, either by adding the projection to the algorithm and experiments or by re-deriving the bound for the unprojected update.","section":"Section 3.3, Appendix B.2, Eq. (5)"},{"comment":"The main text in Section 3.3 defines N = floor(1/alpha), while the Appendix B.2 statement uses N = ceil(1/alpha) and concludes with a limsup bound rather than Eq. (8). Within the proof, the line 'k <= N-1, alpha >= 1/N' has the inequality reversed: the step (1 - k alpha) >= 1/N requires alpha <= 1/N. This condition holds for the floor definition but is false for the ceiling definition in general (e.g., alpha = 0.12 gives ceil = 9 and alpha > 1/9). Because this inequality produces the lower bound eta/N used in the final positivity argument, the proof does not currently support either version of the theorem as written.","section":"Appendix B.2, Theorem 1"},{"comment":"The displayed identity in the proof of Theorem 2 has a sign error. From Eq. (5), eta_t(err_t - alpha) = q_{t+1} - q_t - eta_t (s_t - q_t) nabla f(s_t - q_t), so the sum from t = r to T equals q_{T+1} - q_r minus the sum of eta_t g_t, not plus. The subsequent absolute-value steps may be repairable because taking absolute values makes the sign immaterial, but the proof as printed is not a valid derivation of the bound in Eq. (13).","section":"Appendix B.3, Theorem 2"},{"comment":"The theoretical guarantees depend on a known bound B in Assumption 1, and Theorem 1 requires eta > 2NB and c < min{eta,N^2}/(2N^2[B+(1-alpha+lambda)eta]). The experimental section does not state a value of B or verify these inequalities, and the implemented adaptive rates eta_t = eta*(max - min over a window) with eta in {1, 0.5, 0.1, 0.05} will typically violate eta > 2NB for any plausible B on the real datasets. The paper should clarify which theorem is intended to cover the experimental configuration and discuss how B would be obtained in practice; as it stands, the empirical demonstration does not instantiate the conditions of the main theorem.","section":"Section 3.3, Section 4.1, Section G.2"}],"minor_comments":[{"comment":"In Step 5 of Algorithm 5, the adaptive learning rate is defined as eta*(max{s_{t-w+1},...,s_t} - max{s_{t-w+1},...,s_t}), which is identically zero; the second maximum should presumably be a minimum.","section":"Algorithm 5"},{"comment":"In Algorithm 3, the loop reads 'Observe input X_{t+1}' and returns a prediction set using q_{t+1}, which is inconsistent with the sequential convention used in the other algorithms; it should be X_t and q_t.","section":"Algorithm 3"},{"comment":"The phrase 'degree of miscovery' should be 'degree of miscoverage'.","section":"Section 3.2"},{"comment":"The notation h_t is used both as a fixed cutoff scaled by h and as the window range of the scores; please state the domain of h and clarify the relation between h and h_t explicitly.","section":"Eq. (6) and Section 4.1"},{"comment":"The sentence stating that the EQ term 'tends to decrease as s_t - q_t grows' is ambiguous, because for the sigmoid the EQ function increases on small positive x and then decreases; consider describing the non-monotone shape shown in Figure 2.","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The technical issues are localized but load-bearing: the proof of Theorem 1 is for a projected update, the definition of N is inconsistent between the main text and the appendix, the key inequality is reversed, and the proof of Theorem 2 has a sign error. All of these are fixable either by changing the algorithm to include the projection or by re-deriving the bounds for the unprojected update. I recommend requiring a major revision and a careful re-verification of both theorems before reconsidering the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ECI is a real, modest extension of OGD/ACI: replace binary miscoverage feedback with an error-quantified term, and analyze the update. The empirical story is credible: coverage near nominal and widths mostly smaller than baselines across several datasets. The variants (cutoff, integral) are reasonable, and code is provided. I think the method will interest people in online conformal prediction.\n\nThe problem is the paper's central guarantee. Theorem 1, as stated for update (5), is proved in Appendix B.2 after imposing q_t = max{q_t,0} at every step. That is not a no-op under the stated update: Proposition 1 only gives q_t >= -(alpha+lambda)M_{t-1}, which can be negative, and a negative threshold yields empty prediction sets and changes the feedback dynamics. The proof's inequality 'alpha >= 1/N' is also reversed (should be alpha <= 1/N), and N is floor in the main text but ceil in the appendix. These are not just typos; the lower bound eta/N and the whole 'N-1 coverage steps' argument depends on the definition. So as written, the promised long-term miscoverage control is unproven for the algorithm that is actually implemented and evaluated. This is a load-bearing gap, but it is repairable: either prove the bound for the unprojected update, or change the algorithm to include the projection and update the text.\n\nSecondary issues: baselines get best-of-many learning rates, which can overstate improvements, and the Transformer tables lack variance estimates. Assumption 1 (scores in [0,B]) is standard in this literature, even if real stock returns don't literally satisfy it, so I won't hold that against the paper.\n\nBottom line: the idea is worth engaging with, and the gap is fixable, but the paper should not be accepted with the current theorem/proof mismatch. It deserves a serious referee and a major revision.","headline":"Useful new update rule for online conformal inference, but the central coverage theorem is proven for a projected variant; the gap is fixable and worth a major revision.","tokens_in":28268,"tokens_out":1943,"would_cite":false,"duration_ms":19920,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new online conformal method, ECI, adds the size of the miss to hit/miss feedback and claims to hold long-run miscoverage at its target while shrinking prediction sets.","keywords":["conformal prediction","online learning","time series","uncertainty quantification","miscoverage control","distribution shift","prediction intervals","error quantification"],"falsifier":"Run the unclipped update in Equation (5) on a simulated stream with $\\alpha=0.1$ and scores in $[0,1]$, and check whether every miscoverage step is followed by at least $N-1$ coverage steps; one violating window would falsify Theorem 1's finite-window bound for the implemented algorithm.","tokens_in":27200,"feed_emoji":"🎯","tokens_out":13752,"duration_ms":119524,"temperature":0.7,"pith_summary":"Online conformal prediction for time series usually updates a per-step threshold using only a binary indicator of whether the true label fell inside the prediction set. The paper argues that this throws away useful information: a miss by a little and a miss by a lot get the same feedback, so recovery from distribution shifts is slow. ECI adds an error-quantification (EQ) term, roughly the signed distance between the non-conformity score and the current threshold multiplied by the derivative of a smooth approximation to the sign function, to the usual subgradient update. The paper claims that with this extra feedback the long-run average miscoverage still converges to the target level $\\alpha$ under arbitrary dependence and distribution shift, and that the resulting prediction sets are narrower than those of existing online conformal methods on finance, energy, and climate datasets. The payoff would be distribution-free prediction intervals that remain calibrated and informative when exchangeability fails.","feed_headline":"ECI holds long-run miscoverage at target and shrinks prediction sets","feed_subtitle":"Feeding the size of the miss into the threshold update yields distribution-free coverage guarantees in non-stationary streams","key_machinery":"The central object is the error-quantification (EQ) term $(s_t-q_t)\\nabla f(s_t-q_t)$ added to the online gradient update. It is the signed distance between the revealed non-conformity score and the current threshold, scaled by the local slope of a smooth surrogate for the indicator function, such as the sigmoid. This makes the feedback continuous and adaptive: small misses are corrected gently, large deviations produce a larger but damped correction, and the proof uses the boundedness of the term, together with bounded scores, to show that one miss forces the next $N-1$ steps to be hits.","core_discovery":"ECI's central claim is that quantile tracking in online conformal inference can be improved by replacing purely binary feedback with partially smoothed feedback. The update is $q_{t+1}=q_t+\\eta(\\mathrm{err}_t-\\alpha+(s_t-q_t)\\nabla f(s_t-q_t))$, where $\\mathrm{err}_t$ is the miscoverage indicator, $s_t$ is the non-conformity score, and $f$ is a smooth approximation to the indicator of $x>0$, typically the sigmoid $\\sigma(cx)$. Under the assumption that scores $s_t$ lie in $[0,B]$ and $|x\\nabla f(x)|\\leq\\lambda$, Theorem 1 proves a dynamic miscoverage bound: for a fixed learning rate satisfying $\\eta>2NB$ and a small smoothing scale $c$, every miscoverage step is followed by at least $N-1$ coverage steps, where $N=\\lfloor 1/\\alpha\\rfloor$, so $(1/N)\\sum_{t=T+1}^{T+N}\\mathbf{1}\\{Y_t\\notin\\hat{C}_t\\}\\leq 1/N$; when $\\alpha=1/N$ this gives the long-run guarantee $(1/T)\\sum_{t=1}^{T}\\mathbf{1}\\{Y_t\\notin\\hat{C}_t\\}\\to\\alpha$. Theorem 2 gives a finite-sample bound for arbitrary positive learning rates. Empirically, ECI and its cutoff and integral variants hold coverage near the nominal level while reporting shorter average and median prediction-set widths than the baselines on Amazon and Google stock prices, electricity demand, Delhi temperature, and a synthetic changepoint setting. The proof of Theorem 1 works with thresholds clipped at zero after each update, as stated in Appendix B.2.","pith_inferences":["A fair test of the guarantee should implement the threshold-clipped update used in the proof; the unclipped update in Equation (5) may behave differently when the optimal threshold would go negative.","For unbounded real scores, a practical route to satisfy the bounded-score assumption is to transform scores first; whether ECI retains its tighter-width advantage under such transformations is not tested in the paper.","The one-miss-then-$N-1$-hits pattern suggests ECI could double as a changepoint detector, since a cluster of misses inside a short window signals that the learning rate or base forecaster needs resetting.","Because the EQ term damps very large deviations, ECI may be less vulnerable than binary-feedback methods to single outliers; a heavy-tailed synthetic experiment would separate this robustness from the distribution-shift benefit."],"forward_implications":["A user can run ECI with a single fixed learning rate and still expect long-run miscoverage at level $\\alpha$ without any exchangeability or stationarity assumption on the time series.","The proof structure implies a finite-window guarantee: after any miscoverage step, at least $N-1$ of the next $N$ steps cover the true label, which is stronger than an asymptotic average.","The empirical widths imply practitioners can shrink prediction intervals without sacrificing calibration on these datasets, which reduces the cost of decisions based on those intervals.","The cutoff and integral variants show the same feedback idea can be tuned to avoid over-correction for small errors and to stabilize coverage by averaging over past errors.","Combining ECI with a scorecaster can beat conformal PID using the same scorecaster, indicating that the EQ update is compatible with residualization of systematic forecast error."],"supporting_citations":[{"why":"introduces the ACI online conformal framework and the long-term miscoverage objective that ECI inherits.","marker":"Gibbs & Candès (2021)"},{"why":"supplies the conformal PID baseline with adaptive learning rates and scorecasting, the main comparator in the experiments.","marker":"Angelopoulos et al. (2023b)"},{"why":"gives the decay-OGD baseline and the adaptive-learning-rate analysis that Theorem 2 extends.","marker":"Angelopoulos et al. (2024)"},{"why":"provides the strongly adaptive SF-OGD baseline against which ECI is compared.","marker":"Bhatnagar et al. (2023)"},{"why":"supplies the non-exchangeable conformal theory and the synthetic changepoint data-generating process used in the experiments.","marker":"Barber et al. (2023)"},{"why":"establishes the conformal prediction framework and exchangeability-based prediction sets that online methods generalize.","marker":"Vovk et al. (2005)"}],"fun_headline_variants":["Quantify the miss: ECI beats binary feedback for conformal sets","Beyond misses: ECI uses error size to shrink prediction sets","Error-quantified updates: tighter intervals with long-run coverage","Smoothed error feedback tightens conformal prediction sets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The long-term coverage guarantee rests on the proof clipping thresholds at zero after every update, while the algorithm as implemented and evaluated uses the unclipped update, and on all scores lying inside a known bound $B$; if either condition fails for the implemented procedure, the advertised guarantee may not hold.","fun_headline_variants_meta":{"raw":{"variants":["Quantify the miss: ECI beats binary feedback for conformal sets","Beyond misses: ECI uses error size to shrink prediction sets","Error-quantified updates: tighter intervals with long-run coverage","Smoothed error feedback tightens conformal prediction sets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000509,"raw_usage":{"total_tokens":2561,"prompt_tokens":1109,"completion_tokens":1452,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":725,"completion_tokens_details":{"reasoning_tokens":1380}},"tokens_in":725,"tokens_out":1452,"duration_ms":11665,"temperature":1.0,"reasoning_tokens":1380,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-09T17:36:50.212238+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the unclipped update in Equation (5) on a simulated stream with $\\alpha=0.1$ and scores in $[0,1]$, and check whether every miscoverage step is followed by at least $N-1$ coverage steps; one violating window would falsify Theorem 1's finite-window bound for the implemented algorithm.","supporting_citations":[{"cited_title":"Improved online conformal prediction via strongly adaptive online learning","cited_arxiv_id":null,"evidence_quote":"provides the strongly adaptive SF-OGD baseline against which ECI is compared."}],"review_version":1}