{"id":"99caab7a-fd35-46d9-9805-c7d47edffe4e","arxiv_id":"2510.02809","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Replacing the binary inside/outside error in PID and ECI online conformal prediction with smooth relevance functions can shrink prediction intervals while keeping long-run coverage on several time-series benchmarks.","lead":"The paper changes how online conformal prediction adjusts interval thresholds: instead of only noting whether the true value landed inside the forecast interval, it also uses the distance from the true value to the interval boundary. On stock and temperature datasets, the modified PID algorithm mostly produces narrower intervals at similar coverage, while the modified ECI variant trades some width for better coverage.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Modified ECI coverage guarantee is not established: Theorem 3 assumes bounded ∇f but μ_t in Eq. (6) can vanish, making |∇f| unbounded.","rationale":"The reader's weakest_assumption focused on Theorem 2's dominance condition, which affects the half/full PID variants (Eqs. 7, 8) that are secondary to the main PID result (Eq. 9). My concern targets Theorem 3 and the modified ECI update (Eq. 10), which is one of the two headline methods. This is a more load-bearing gap because the central claim explicitly includes ECI maintaining long-run coverage. The issue is concrete and addressable (e.g., clipping μ_t away from zero or proving a lower bound), so the paper should remain CONDITIONAL rather than be rejected. Since the reader's own rationale already notes the μ_t issue in Theorem 3, there is partial agreement, but the reader's primary flagged assumption was Theorem 2. Thus the recommended verdict is unchanged.","tokens_in":13248,"tokens_out":7611,"duration_ms":60830,"concrete_test":"Construct a synthetic sequence with constant score s_t = c and initialize q_1 = c, using a very small η so q_t remains close to c. Then μ_t → 0 for t ≥ T_w. Compute max_x |∇f_{ω,v,μ}(x)| for μ = 10^{-1}, 10^{-2}, 10^{-3} with v = 1; if the maximum grows roughly as 1/μ, the uniform-M assumption fails. Then run update (10) on this sequence (adding a tiny ε to μ_t to avoid division by zero) and record the empirical coverage over a long horizon; if coverage deviates substantially from α, the theorem is empirically falsified. Alternatively, re-derive Theorem 3 without invoking the boundedness of ∇f and check whether the argument still goes through.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the modified ECI update (Eq. 10) maintains long-run coverage rests on Theorem 3, which reuses the ECI proof. That proof requires uniform constants M, U with |∇f| ≤ M and |x·∇f(x)| ≤ U for all x and t. For the proposed family in Eq. (6), ∂f/∂x = Σ_i ω_i (v_i/μ_t) σ'( (v_i/μ_t)(s_t−q_t) − ln((1−α)/α) ), so |∇f| scales as 1/μ_t. μ_t is a moving average of |s_i − q_i| over the previous T_w steps; nothing in the updates prevents q_i from tracking s_i so closely that μ_t → 0 (e.g., when the interval boundary repeatedly equals the ground-truth score). In that regime no finite M exists, and Theorem 3's assumptions cannot be satisfied. The paper does not address this, and its own Appendix B.2 questions the original ECI proof's strong assumption η > N·B, further weakening the transfer. Thus the modified ECI's long-run coverage guarantee is unsupported; the empirical coverage (≈0.90) is reported without error bars and with v adjusted per dataset, which is insufficient to substitute for a proof.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes replacing the binary in/out error indicator in online conformal prediction (OCP) threshold updates with a family of smooth \"relevance\" functions f_{ω,v,μ_t}(s_t−q_t) (Eq. 6), which take values in [0,1], satisfy f(0)=α, and are scale-adaptive through the moving-average normalizer μ_t. The modified PID updates (Eqs. 7–9) and the modified ECI update (Eq. 10) are analyzed for long-run coverage (Eq. 5) and evaluated on four time-series datasets with AR and Theta regressors. Theorems 1 and 2 give coverage guarantees for the PID variants, Theorem 2 conditionally on an unproven dominance assumption; Theorem 3 claims coverage for modified ECI by transferring the original ECI proof. Experiments show reduced interval widths for the modified PID update (Eq. 9) in several settings, while modified ECI achieves nominal coverage but with larger interval widths than the under-covering original ECI in the same settings.","tokens_in":13600,"tokens_out":2694,"duration_ms":76412,"significance":"If the theoretical guarantees were fully established, the proposed relevance-aware thresholding would be a useful and broadly applicable idea for OCP, potentially improving interval efficiency without sacrificing long-run coverage. The paper is clearly written, builds on state-of-the-art methods (PID, ECI), and provides code. However, the two load-bearing theoretical claims—Theorem 2's conditional guarantee and Theorem 3's boundedness assumption—are not adequately justified, and the experimental evidence is limited to point estimates without error bars and with per-dataset parameter tuning. The contribution is therefore promising but not yet at the standard required for publication as a definitive advance.","major_comments":[{"comment":"The proof of Theorem 3 simply states that the original ECI proof holds because f_{ω,v,μ_t} satisfies the boundedness conditions. But for f defined in Eq. (6), ∂f/∂x = Σ ω_i (v_i/μ_t) σ'( (v_i/μ_t)(s_t−q_t) − ln((1−α)/α) ), so |∇f| scales as 1/μ_t. μ_t is a moving average of |s_i−q_i| over the previous T_w steps; nothing in the updates prevents μ_t → 0 (e.g., when q_i exactly equals s_i for many steps). In that regime no finite M exists. Since the boundedness assumption is not proven and can be violated, the long-run coverage guarantee for modified ECI is unsupported. The authors themselves cast doubt on the original ECI proof's strong assumption η > N·B in Appendix B.2, further weakening the transfer.","section":"Theorem 3 / Eq. (10), Appendix A"},{"comment":"The coverage guarantee for the two PID variants in Eqs. (7) and (8) depends on the existence of T′ such that |Σ(1{Y_i∉C_i}−α)| ≤ |Σ(f_i−α)| for all T≥T′. The authors explicitly note this \"could be pretty strong\" because f is often close to α. No argument or experiment is given to support this dominance condition. Since these two variants are presented as contributions and evaluated in Tables 7–8, their validity claim is conditional on an unverified property. At minimum, the condition should be proved for some nontrivial parameter regime or empirically checked on the tested datasets; otherwise Eqs. (7) and (8) should be presented only as heuristics.","section":"Theorem 2, Section 3 (after Eq. 9), Appendix D.1"},{"comment":"The experimental support for the central claim of narrower intervals is mixed. For modified PID (Eq. 9), results are mostly favorable, but for modified ECI, Tables 2–3 and 9–10 show that the interval widths are systematically larger than those of the original ECI in the same setting; the favorable comparison to the original ECI only appears when using the results reported in [18] (Table 4), which were obtained with different settings. Moreover, all results are point estimates without error bars or multiple runs, and parameters (especially v for modified ECI and the learning rate for ECI) are tuned per dataset. This is insufficient to substantiate the claim that relevance-aware thresholding \"potentially result[s] in narrower prediction intervals\" while maintaining coverage.","section":"Section 4, Tables 1–3, 5–10"}],"minor_comments":[{"comment":"The saturation function r_t(x) = KI·tan(x·log(t)) / (t·C_sat) appears to contain a typo: the standard PID saturation uses tanh, not tan. If tan is intended, the behavior at ±π/2 is undefined and the stated property in Eq. (3) needs clarification.","section":"Eq. (15)"},{"comment":"In Eq. (6), μ_t = (1/T_w) |Σ_{i=max(t−T_w,1)}^{t−1} (s_i−q_i)| uses only past values, which is fine, but the indexing is inconsistent with the text \"previous T_w steps\" and with the upper limit t−1. Please align notation.","section":"Notation for μ_t"},{"comment":"The induction proof for Eq. (13) is sketched for the upper bound and says \"Doing the same thing with the lower bound,\" but the lower-bound case is not written out. This is acceptable for an appendix, but the symmetric argument should at least mention that f takes values in [α,1] and that the saturation property is used symmetrically.","section":"Proof of Theorem 2, Appendix A"},{"comment":"The comparison with ECI (Appendix B.2) is useful, but the discussion of the original ECI proof's assumption η > N·B is presented as a remark without a citation. This is an important correctness concern and should be integrated into the main text or properly supported.","section":"Related work"}],"recommendation":"major_revision","confidential_remarks":"The paper has a sound core idea and one solid result (Theorem 1 for Eq. (9)), but the two other theoretical guarantees rest on assumptions that are either unproven (Theorem 2) or likely violated (Theorem 3). The experimental section needs reruns with error bars and a fair comparison protocol, especially for ECI. Given these are fixable—by adding assumptions, proving boundedness, or repositioning claims—I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know this paper is more interesting than its abstract promises, and more honest than most. The family of sigmoid-based relevance functions in Eq. (6) — with f(0)=α and the scale normalizer μ_t — is a genuine, clearly described modification to existing online conformal prediction updates. The code is available, and the authors are upfront about the weak spots: they explicitly flag the dominance assumption in Theorem 2 as \"pretty strong,\" and Appendix B.2 questions the original ECI proof's reliance on η > N·B. That candor is real and worth crediting.\n\nWhat actually works: the modified PID update in Eq. (9) inherits the PID coverage guarantee, and the experiments suggest it does narrow intervals on several datasets while keeping coverage near 0.90. That is a useful, practical gain for people doing time-series forecasting.\n\nWhere it gets soft: the other three variants have serious theoretical gaps. Theorem 2's coverage guarantee for Eqs. (7) and (8) depends on an unproven dominance condition that the authors admit may fail — and no empirical check is offered. Theorem 3, which is supposed to underwrite the modified ECI update in Eq. (10), assumes uniform bounds on |∇f| and |x·∇f(x)|. But with μ_t as a moving average of |s_i − q_i|, nothing stops the interval boundary from tracking the score closely enough that μ_t → 0, making |∇f| unbounded. The paper hand-waves this by citing the original ECI proof, which is not a sound transfer. So the modified ECI's coverage guarantee is not established.\n\nThe experiments reflect that gap: for ECI, the modified version fixes the baseline's undercoverage (0.90 vs 0.78–0.84) but produces wider intervals, not narrower ones. That undermines the paper's central claim as stated. The authors note this but frame the width comparison as unfair because the baseline undercovers. There are no error bars, and the v parameter is tuned per dataset, so we don't know how robust the results are.\n\nThis is not a reject-with-prejudice. The idea is plausible, the writing is clear, and the honest discussion suggests the authors understand the limitations. But as it stands, the theory supports only Eq. (9), and the ECI empirical story is mixed. A serious referee could push them to validate or drop the dominance assumption, fix the boundedness issue for μ_t, add error bars, and report parameter sensitivity. I'd send it to review, but I'd expect major revision.","headline":"Useful empirical idea and an honest paper, but the theory only covers one of the four variants and the ECI experiments don't show the promised narrower intervals.","tokens_in":14058,"tokens_out":1524,"would_cite":false,"duration_ms":21785,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that replacing the binary inside/outside error in online conformal prediction threshold updates with smooth, distance-aware relevance functions can produce tighter prediction intervals while preserving long-run coverage.","keywords":["online conformal prediction","time series","prediction intervals","long-run coverage","relevance function","smooth threshold update","PID control","sigmoid smoothing"],"falsifier":"On any time-series dataset, compute A_T=|Σ_{i≤T}(1{Y_i∉C_i}−α)| and B_T=|Σ_{i≤T}(f_i−α)| for the modified PID update (7). If A_T>B_T at all large T, the Theorem 2 condition is violated; a run where coverage then drifts materially below α would refute the practical usefulness of that variant, while a run where coverage still holds would suggest a weaker condition suffices.","tokens_in":13128,"feed_emoji":"📉","tokens_out":6416,"duration_ms":51669,"temperature":0.7,"pith_summary":"Online conformal prediction for time series updates a threshold each step based on whether the last ground truth fell inside the forecast interval. This paper argues that the binary pass/fail signal throws away usable information: a miss by an inch and a miss by a mile are treated identically. It proposes a family of smooth relevance functions, built from scaled sigmoids, that grade the interval by the distance from the ground truth to its boundary. Grafted onto two existing online update rules, these functions are shown — under explicit conditions — to keep the long-run miscoverage at the target α while often shrinking the intervals. On stock-price and temperature datasets, the modified PID-style rule usually narrows intervals at fixed coverage, and the modified smoothed rule reaches nominal coverage where its original version undercovered. If the claim holds, forecasters get narrower uncertainty bands without relaxing the coverage guarantee.","feed_headline":"Graded misses shrink forecast intervals, keep coverage","feed_subtitle":"Replacing pass/fail updates with distance-aware smooth scores tightens forecast bands in online prediction.","key_machinery":"The key object is the smooth relevance family f^{ω,v,µ_t}(x)=Σ_i ω_i σ( (v_i/µ_t) x − ln((1−α)/α) ), with x=s_t−q_t measuring signed distance from ground truth to interval boundary, µ_t the mean absolute distance over a trailing window, and ω, v tunable weights and sharpnesses. It satisfies three design constraints: output in [0,1] like an indicator, scale independence via µ_t, and the static state f(0)=α so a boundary hit causes no update. This function replaces the binary error inside PID and ECI updates, converting the threshold update from a step function into a graded, saturating response.","core_discovery":"The paper's central discovery is that the threshold update in online conformal prediction does not have to treat every prediction interval as simply right or wrong. By replacing the indicator 1{Y_t∉C_t} with a smooth function f^{ω,v,µ_t}(s_t−q_t) that rises with the ground truth's distance outside the interval and falls when the ground truth is comfortably inside, the update can react proportionately instead of abruptly. The authors construct f as a weighted sum of sigmoids whose input is scaled by the windowed mean absolute distance µ_t, making it scale-invariant, and they fix the intercept so f(0)=α, meaning a ground truth exactly on the boundary leaves the threshold unchanged. They prove","pith_inferences":["The authors chose PID and ECI as carriers, but nothing in the construction is specific to them; replacing the indicator in other online update rules (e.g., simple online gradient descent on quantile loss) with the same relevance family is a direct, testable extension.","Because µ_t is a windowed mean of absolute distances, the functions may be sensitive to outliers inflating µ_t; a quantile-based scale estimator would be a natural robustness extension.","The experimental comparison tilts in favor of the modified ECI partly because the original ECI undercovered on these datasets; a fairer width comparison would tune both methods to the same achieved coverage level before comparing interval sizes."],"forward_implications":["Modified PID update (9) preserves long-run coverage without extra assumptions, and in experiments often produces narrower average and median intervals at fixed coverage.","Modified ECI update (10) reaches 90% coverage on the tested datasets where the original ECI update undercovered, though with wider intervals.","The family's parameters v and ω give decision-makers a continuous knob: large v recovers indicator-like behavior, small v smooths aggressively; time-varying choices are possible.","The coverage guarantee for updates (7) and (8) is conditional on a cumulative-deviation dominance; when that condition holds, the full-relevance variants also keep long-run coverage.","Since f(0)=α, the update leaves the threshold unchanged when the ground truth lands exactly on the boundary, a property the original binary/ECI smoothing lacks."],"fun_headline_variants":["Smooth scores replace pass/fail for tighter bands","Relevance-aware updates slim prediction intervals","Distance-based thresholding trims forecast width","Graded coverage: maintain validity, cut interval width","Online conformal prediction with smooth relevance updates"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"For two of the modified PID updates, the long-run coverage proof assumes that the cumulative absolute deviation of the smooth relevance scores from α eventually stays at least as large as the cumulative absolute deviation of the binary errors — an inequality the authors acknowledge can fail because the smooth scores often sit close to α.","fun_headline_variants_meta":{"raw":{"variants":["Smooth scores replace pass/fail for tighter bands","Relevance-aware updates slim prediction intervals","Distance-based thresholding trims forecast width","Graded coverage: maintain validity, cut interval width","Online conformal prediction with smooth relevance updates"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1529,"prompt_tokens":779,"completion_tokens":750,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":523,"completion_tokens_details":{"reasoning_tokens":681}},"tokens_in":523,"tokens_out":750,"duration_ms":7017,"temperature":1.0,"reasoning_tokens":681,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T12:37:35.125787+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On any time-series dataset, compute A_T=|Σ_{i≤T}(1{Y_i∉C_i}−α)| and B_T=|Σ_{i≤T}(f_i−α)| for the modified PID update (7). If A_T>B_T at all large T, the Theorem 2 condition is violated; a run where coverage then drifts materially below α would refute the practical usefulness of that variant, while a run where coverage still holds would suggest a weaker condition suffices.","supporting_citations":[],"review_version":1}