Pith. sign in

REVIEW 3 major objections 4 minor 25 references

Relevance-Aware Thresholding in Online Conformal Prediction for Time Series

T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Pith's one-line read This paper claims that replacing the binary inside/outside error in online conformal prediction threshold updates with smooth, distance-aware relevance functions can produce tighter prediction intervals while preserving long-run coverage.

desk verdict Useful empirical idea and an honest paper, but the theory only covers one of the four variants and the ECI experiments don't show the promised narrower intervals. read the letter →

arxiv 2510.02809 v2 pith:OQLRVLNM submitted 2025-10-03 cs.LG cs.AI

classification cs.LGcs.AI
keywords onlineconformalpredictiontimeseriesintervalslong-runcoveragerelevancefunctionsmooththresholdupdatePIDcontrolsigmoidsmoothing
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Online conformal prediction for time series updates a threshold each step based on whether the last ground truth fell inside the forecast interval. This paper argues that the binary pass/fail signal throws away usable information: a miss by an inch and a miss by a mile are treated identically. It proposes a family of smooth relevance functions, built from scaled sigmoids, that grade the interval by the distance from the ground truth to its boundary. Grafted onto two existing online update rules, these functions are shown — under explicit conditions — to keep the long-run miscoverage at the target α while often shrinking the intervals. On stock-price and temperature datasets, the modified PID-style rule usually narrows intervals at fixed coverage, and the modified smoothed rule reaches nominal coverage where its original version undercovered. If the claim holds, forecasters get narrower uncertainty bands without relaxing the coverage guarantee.

What carries the argument

The key object is the smooth relevance family f^{ω,v,µ_t}(x)=Σ_i ω_i σ( (v_i/µ_t) x − ln((1−α)/α) ), with x=s_t−q_t measuring signed distance from ground truth to interval boundary, µ_t the mean absolute distance over a trailing window, and ω, v tunable weights and sharpnesses. It satisfies three design constraints: output in [0,1] like an indicator, scale independence via µ_t, and the static state f(0)=α so a boundary hit causes no update. This function replaces the binary error inside PID and ECI updates, converting the threshold update from a step function into a graded, saturating response.

What would settle it

On any time-series dataset, compute A_T=|Σ_{i≤T}(1{Y_i∉C_i}−α)| and B_T=|Σ_{i≤T}(f_i−α)| for the modified PID update (7). If A_T>B_T at all large T, the Theorem 2 condition is violated; a run where coverage then drifts materially below α would refute the practical usefulness of that variant, while a run where coverage still holds would suggest a weaker condition suffices.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the threshold update in online conformal prediction does not have to treat every prediction interval as simply right or wrong. By replacing the indicator 1{Y_t∉C_t} with a smooth function f^{ω,v,µ_t}(s_t−q_t) that rises with the ground truth's distance outside the interval and falls when the ground truth is comfortably inside, the update can react proportionately instead of abruptly. The authors construct f as a weighted sum of sigmoids whose input is scaled by the windowed mean absolute distance µ_t, making it scale-invariant, and they fix the intercept so f(0)=α, meaning a ground truth exactly on the boundary leaves the threshold unchanged. They prove

Load-bearing premise

For two of the modified PID updates, the long-run coverage proof assumes that the cumulative absolute deviation of the smooth relevance scores from α eventually stays at least as large as the cumulative absolute deviation of the binary errors — an inequality the authors acknowledge can fail because the smooth scores often sit close to α.

Editorial extensions

If this is right

  • Modified PID update (9) preserves long-run coverage without extra assumptions, and in experiments often produces narrower average and median intervals at fixed coverage.
  • Modified ECI update (10) reaches 90% coverage on the tested datasets where the original ECI update undercovered, though with wider intervals.
  • The family's parameters v and ω give decision-makers a continuous knob: large v recovers indicator-like behavior, small v smooths aggressively; time-varying choices are possible.
  • The coverage guarantee for updates (7) and (8) is conditional on a cumulative-deviation dominance; when that condition holds, the full-relevance variants also keep long-run coverage.
  • Since f(0)=α, the update leaves the threshold unchanged when the ground truth lands exactly on the boundary, a property the original binary/ECI smoothing lacks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The authors chose PID and ECI as carriers, but nothing in the construction is specific to them; replacing the indicator in other online update rules (e.g., simple online gradient descent on quantile loss) with the same relevance family is a direct, testable extension.
  • Because µ_t is a windowed mean of absolute distances, the functions may be sensitive to outliers inflating µ_t; a quantile-based scale estimator would be a natural robustness extension.
  • The experimental comparison tilts in favor of the modified ECI partly because the original ECI undercovered on these datasets; a fairer width comparison would tune both methods to the same achieved coverage level before comparing interval sizes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes replacing the binary in/out error indicator in online conformal prediction (OCP) threshold updates with a family of smooth "relevance" functions f_{ω,v,μ_t}(s_t−q_t) (Eq. 6), which take values in [0,1], satisfy f(0)=α, and are scale-adaptive through the moving-average normalizer μ_t. The modified PID updates (Eqs. 7–9) and the modified ECI update (Eq. 10) are analyzed for long-run coverage (Eq. 5) and evaluated on four time-series datasets with AR and Theta regressors. Theorems 1 and 2 give coverage guarantees for the PID variants, Theorem 2 conditionally on an unproven dominance assumption; Theorem 3 claims coverage for modified ECI by transferring the original ECI proof. Experiments show reduced interval widths for the modified PID update (Eq. 9) in several settings, while modified ECI achieves nominal coverage but with larger interval widths than the under-covering original ECI in the same settings.

Significance. If the theoretical guarantees were fully established, the proposed relevance-aware thresholding would be a useful and broadly applicable idea for OCP, potentially improving interval efficiency without sacrificing long-run coverage. The paper is clearly written, builds on state-of-the-art methods (PID, ECI), and provides code. However, the two load-bearing theoretical claims—Theorem 2's conditional guarantee and Theorem 3's boundedness assumption—are not adequately justified, and the experimental evidence is limited to point estimates without error bars and with per-dataset parameter tuning. The contribution is therefore promising but not yet at the standard required for publication as a definitive advance.

major comments (3)
  1. [Theorem 3 / Eq. (10), Appendix A] The proof of Theorem 3 simply states that the original ECI proof holds because f_{ω,v,μ_t} satisfies the boundedness conditions. But for f defined in Eq. (6), ∂f/∂x = Σ ω_i (v_i/μ_t) σ'( (v_i/μ_t)(s_t−q_t) − ln((1−α)/α) ), so |∇f| scales as 1/μ_t. μ_t is a moving average of |s_i−q_i| over the previous T_w steps; nothing in the updates prevents μ_t → 0 (e.g., when q_i exactly equals s_i for many steps). In that regime no finite M exists. Since the boundedness assumption is not proven and can be violated, the long-run coverage guarantee for modified ECI is unsupported. The authors themselves cast doubt on the original ECI proof's strong assumption η > N·B in Appendix B.2, further weakening the transfer.
  2. [Theorem 2, Section 3 (after Eq. 9), Appendix D.1] The coverage guarantee for the two PID variants in Eqs. (7) and (8) depends on the existence of T′ such that |Σ(1{Y_i∉C_i}−α)| ≤ |Σ(f_i−α)| for all T≥T′. The authors explicitly note this "could be pretty strong" because f is often close to α. No argument or experiment is given to support this dominance condition. Since these two variants are presented as contributions and evaluated in Tables 7–8, their validity claim is conditional on an unverified property. At minimum, the condition should be proved for some nontrivial parameter regime or empirically checked on the tested datasets; otherwise Eqs. (7) and (8) should be presented only as heuristics.
  3. [Section 4, Tables 1–3, 5–10] The experimental support for the central claim of narrower intervals is mixed. For modified PID (Eq. 9), results are mostly favorable, but for modified ECI, Tables 2–3 and 9–10 show that the interval widths are systematically larger than those of the original ECI in the same setting; the favorable comparison to the original ECI only appears when using the results reported in [18] (Table 4), which were obtained with different settings. Moreover, all results are point estimates without error bars or multiple runs, and parameters (especially v for modified ECI and the learning rate for ECI) are tuned per dataset. This is insufficient to substantiate the claim that relevance-aware thresholding "potentially result[s] in narrower prediction intervals" while maintaining coverage.
minor comments (4)
  1. [Eq. (15)] The saturation function r_t(x) = KI·tan(x·log(t)) / (t·C_sat) appears to contain a typo: the standard PID saturation uses tanh, not tan. If tan is intended, the behavior at ±π/2 is undefined and the stated property in Eq. (3) needs clarification.
  2. [Notation for μ_t] In Eq. (6), μ_t = (1/T_w) |Σ_{i=max(t−T_w,1)}^{t−1} (s_i−q_i)| uses only past values, which is fine, but the indexing is inconsistent with the text "previous T_w steps" and with the upper limit t−1. Please align notation.
  3. [Proof of Theorem 2, Appendix A] The induction proof for Eq. (13) is sketched for the upper bound and says "Doing the same thing with the lower bound," but the lower-bound case is not written out. This is acceptable for an appendix, but the symmetric argument should at least mention that f takes values in [α,1] and that the saturation property is used symmetrically.
  4. [Related work] The comparison with ECI (Appendix B.2) is useful, but the discussion of the original ECI proof's assumption η > N·B is presented as a remark without a citation. This is an important correctness concern and should be integrated into the main text or properly supported.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the few load-bearing guarantees are inherited from external PID/ECI proofs, and the paper's own caveats are flagged as strong assumptions rather than hidden inputs.

full rationale

The paper's derivation chain is not circular in the sense of reducing a prediction to its own fitted inputs. The modified PID coverage statements (Theorems 1 and 2) reuse the external PID saturation-function argument from [1]; Theorem 2 adds an explicit eventual-dominance condition on the cumulative modified errors rather than assuming the desired coverage conclusion. The paper itself warns that this condition 'could be pretty strong,' which is a rigor caveat, not a circular step. The modified ECI result (Theorem 3) is likewise imported from the external ECI proof [18]; the assertion that |∇f| and |x·∇f| are bounded is not derived from the paper's own outputs, and the dependence of ∇f on 1/μ_t means the premise can fail if μ_t → 0. That is an unsupported or possibly false hypothesis, and Appendix B.2 even questions the original ECI proof's strong assumption η > N·B, but this is a correctness gap, not a circular reduction: the cited proof is independent prior work and the boundedness condition is not equivalent to the target long-run coverage. The reported interval-width improvements are empirical comparisons, not quantities forced by construction. The only self-citation, [11], appears in a non-technical remark about decision makers incorporating preferences and carries no mathematical weight. Therefore the appropriate finding is a low non-circularity score, with the main concerns centered on unverified assumptions rather than circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The central theoretical guarantees rest on inherited PID/ECI proofs plus a conditional dominance assumption that the authors admit may fail. Experimental gains depend on empirically selected v and the ECI learning rate. No new physical or conceptual entities are introduced.

free parameters (5)
  • slope parameter v = Not reported in main text; experiments adjust v (Fig. 1 shows v=4 for illustration)
    Controls how quickly f approaches 0/1. Authors state parameters are chosen empirically (Appendix C.3), so results depend on this tuning.
  • learning rate η (ECI) = Selected from a predefined set to give best coverage
    Section 4.2 states they use the learning rate that yields the best coverage in their setting, making the coverage comparison partly a product of tuning.
  • learning rate η (PID) = 0.005
    Fixed to a classical value; the authors note learning rate and q1 initialization play a crucial role in convergence.
  • window size T_w for μ_t = 100
    Chosen for the scale-normalization term μ_t; the authors note in the conclusion that this choice is not unique.
  • weights ω = ω=1, l=1 in experiments
    For l>1, ω shapes the plateau of the function. Authors state these are chosen empirically.
assumptions (4)
  • domain assumption Bounded non-conformity scores: s_i∈[-b/2,b/2] in Theorems 1-2, and s_i∈[0,B] in Theorem 3
    The proofs inherit boundedness assumptions from PID and ECI; no evidence is given that these bounds hold for the stock and temperature datasets.
  • domain assumption Saturation function r_t satisfies Equation (3)
    The PID-based proofs rely on the saturation function properties from [1]; the paper uses the same class but does not verify it in the experiments.
  • ad hoc to paper Theorem 2 dominance assumption: eventually |Σ(1{Y_i∉C_i}−α)| ≤ |Σ(f_i−α)|
    This unproven assumption is required for the coverage guarantee of the full and half PID variants. The authors explicitly state it could be strong.
  • ad hoc to paper Uniform boundedness of |∇f| and |x∇f(x)| in Theorem 3
    The paper asserts such M and U exist, but f's derivative scales as v/μ_t, so the bound is not uniform if μ_t can approach zero. No lower bound on μ_t is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Relevance-Aware Thresholding in Online Conformal Prediction for Time Series." pith.science (2026). https://pith.science/paper/OQLRVLNM

@misc{pith2026251002809,
  author       = {Pith},
  title        = {Pith review of: Relevance-Aware Thresholding in Online Conformal Prediction for Time Series},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OQLRVLNM}},
  note         = {Machine review of arXiv:2510.02809}
}
read the original abstract

Uncertainty quantification has received considerable interest in recent works in Machine Learning. In particular, Conformal Prediction (CP) gains ground in this field. For the case of time series, Online Conformal Prediction (OCP) becomes an option to address the problem of data distribution shift over time. Indeed, the idea of OCP is to update a threshold of some quantity (whether the miscoverage level or the quantile) based on the distribution observation. To evaluate the performance of OCP methods, two key aspects are typically considered: the coverage validity and the prediction interval width minimization. Recently, new OCP methods have emerged, offering long-run coverage guarantees and producing more informative intervals. However, during the threshold update step, most of these methods focus solely on the validity of the prediction intervals~--~that is, whether the ground truth falls inside or outside the interval~--~without accounting for their relevance. In this paper, we aim to leverage this overlooked aspect. Specifically, we propose enhancing the threshold update step by replacing the binary evaluation (inside/outside) with a broader class of functions that quantify the relevance of the prediction interval using the ground truth. This approach helps prevent abrupt threshold changes, potentially resulting in narrower prediction intervals. Indeed, experimental results on real-world datasets suggest that these functions can produce tighter intervals compared to existing OCP methods while maintaining coverage validity.

Figures

Figures reproduced from arXiv: 2510.02809 by the authors.

Figure 1
Figure 1. fω,v,µt with α = 0.1, ω = 1, v = 4 (l = 1) and µt = 20 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Plots of fω,v,µt with α = 0.1, ω = 1 (l = 1) and µt = 10 [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Plots of fω,v,µt with α = 0.1, v1 = 1, v2 = 10 and µt = 10. In [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: PID vs modified PID performances on the Amazon stock price dataset. The plots show Theta as the regressor. In the table of [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: ECI vs modified ECI on the Microsoft stock price dataset (AR regressor). [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: Evolution of Amazon stock price over time We observe a distribution shift around time step 2300. The increase in interval size appears around time step 2000 in [PITH_FULL_IMAGE:figures/full_fig_p022_6.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

25 extracted references · 5 linked inside Pith

  1. [18]

    https://doi.org/10.48550/arXiv.2502.00818, arXiv:2502.00818 [stat]

    Wu, J., Hu, D., Bao, Y., Xia, S.T., Zou, C.: Error-quantified Conformal In- ference for Time Series (Feb 2025). https://doi.org/10.48550/arXiv.2502.00818, arXiv:2502.00818 [stat]

  2. [1]

    In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S

    Angelopoulos, A., Candes, E., Tibshirani, R.J.: Conformal PID Control for Time Series Prediction. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems. vol. 36, pp. 23047–23074. Curran Associates, Inc. (2023)

  3. [2]

    https://doi.org/10.48550/arXiv.2402.01139 14 T

    Angelopoulos, A.N., Barber, R.F., Bates, S.: Online conformal prediction with decaying step sizes (May 2024). https://doi.org/10.48550/arXiv.2402.01139 14 T. Dupuy et al

  4. [3]

    https://doi.org/10

    Angelopoulos, A.N., Bates, S.: A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (Dec 2022). https://doi.org/10. 48550/arXiv.2107.07511, arXiv:2107.07511

  5. [4]

    International Journal of Forecasting16(4), 521–530 (2000)

    Assimakopoulos,V.,Nikolopoulos,K.:Thethetamodel:Adecompositionapproach to forecasting. International Journal of Forecasting16(4), 521–530 (2000)

  6. [5]

    https://doi.org/10.48550/arXiv.1905.02928

    Barber, R.F., Candes, E.J., Ramdas, A., Tibshirani, R.J.: Predictive inference with the jackknife+ (May 2020). https://doi.org/10.48550/arXiv.1905.02928

  7. [6]

    https://doi.org/10.48550/arXiv.2202.13415, http://arxiv.org/abs/2202.13415, arXiv:2202.13415 [stat]

    Barber, R.F., Candes, E.J., Ramdas, A., Tibshirani, R.J.: Conformal prediction beyond exchangeability (Mar 2023). https://doi.org/10.48550/arXiv.2202.13415, http://arxiv.org/abs/2202.13415, arXiv:2202.13415 [stat]

  8. [7]

    Holden-Day, San Francisco, 1st edn

    Box, G.E.P., Jenkins, G.M.: Time Series Analysis: Forecasting and Control. Holden-Day, San Francisco, 1st edn. (1970)

Show all 25 references
  1. [8]

    Mathe- matics of control, signals and systems2(4) (1989)

    Cybenko, G.: Approximation by superpositions of a sigmoidal function. Mathe- matics of control, signals and systems2(4) (1989)

  2. [9]

    In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence

    Gammerman, A., Vapnik, V., Vovk, V.: Learning by transduction. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann, San Francisco, CA (1998)

  3. [10]

    In: Advances in Neural Information Processing Systems

    Gibbs, I., Candes, E.: Adaptive Conformal Inference Under Distribution Shift. In: Advances in Neural Information Processing Systems. vol. 34, pp. 1660–1672. Curran Associates, Inc. (2021)

  4. [11]

    Resources, Conservation and Recycling168, art

    Jacquin, L., Imoussaten, A., Trousset, F., Didier, P., Montmain, J.: Control of waste fragment sorting process based on MIR imaging coupled with cautious clas- sification. Resources, Conservation and Recycling168, art. 105258 (May 2021). https://doi.org/10.1016/j.resconrec.2020.105258

  5. [12]

    https://www.kaggle.com/datasets/camnugent/ sandp500 (2018)

    Nguyen, C.: S&p 500 stock data. https://www.kaggle.com/datasets/camnugent/ sandp500 (2018)

  6. [13]

    In: Elomaa, T., Mannila, H., Toivonen, H

    Papadopoulos, H., Proedrou, K., Vovk, V., Gammerman, A.: Inductive Confidence Machines for Regression. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) Machine Learning: ECML 2002. pp. 345–356. Springer, Berlin, Heidelberg (2002). https: //doi.org/10.1007/3-540-36755-1_29

  7. [14]

    Prinster, D., Stanton, S., Liu, A., Saria, S.: Conformal validity guarantees exist for any data distribution (and how to find them) (2024), https://arxiv.org/abs/2405. 06627

  8. [15]

    In: Advances in Neural Information Processing Systems

    Tibshirani, R.J., Foygel Barber, R., Candes, E., Ramdas, A.: Conformal Pre- diction Under Covariate Shift. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/ paper/2019/hash/8fb21ee7a2207526da55a679f0...

  9. [16]

    Springer, New York (2005)

    Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005). https://doi.org/10.1007/978-3-031-06649-8

  10. [17]

    https://www.kaggle.com/datasets/ sumanthvrao/daily-climate-time-series-data (2017)

    Vrao, S.: Daily climate time series data. https://www.kaggle.com/datasets/ sumanthvrao/daily-climate-time-series-data (2017)

  11. [19]

    In: Proceedings of the 40th International Conference on Machine Learning

    Xu, C., Xie, Y.: Sequential Predictive Conformal Inference for Time Series. In: Proceedings of the 40th International Conference on Machine Learning. pp. 38707– 38727. PMLR (Jul 2023), iSSN: 2640-3498

  12. [20]

    In: Proceedings of the 39th International Conference on Machine Learning

    Zaffran, M., Feron, O., Goude, Y., Josse, J., Dieuleveut, A.: Adaptive Conformal Predictions for Time Series. In: Proceedings of the 39th International Conference on Machine Learning. pp. 25834–25866. PMLR (Jun 2022), iSSN: 2640-3498 Relevance-Aware Thresholding in OCP for Tim...

  13. [21]

    we prove the long-run coverage for the update in equation (11) with(si)i∈N∗ being any sequence of numbers in[−b, b]

  14. [22]

    This leads to the fact that the update in equation (12) satisfies the long-run coverage

    we apply the point 1 onq′ t+1 =q t+1 −ˆqt+1 with(s i)i∈N∗ being any sequence of numbers in[−b 2 , b 2 ]and(ˆq i)i∈N∗ any sequence of numbers in[−b 2 , b 2 ]. This leads to the fact that the update in equation (12) satisfies the long-run coverage. Thus, proving the point 1 is e...

  15. [23]

    First of all, we propose a general idea that could be generalized to all the OCP methods

  16. [24]

    They set, in almost all the experiments,λ= 1

    In ECI, the authors use the sigmoid functionσ(λ·x). They set, in almost all the experiments,λ= 1. With this function they haveσ(0) = 0.5whereas, we propose a family of functions for whichf ω,v,µ t (0) =α. Indeed, when st =q t (i.e. the ground truth is equal to one of the inter...

  17. [25]

    The functionsfω,v,µ t try to consider the scale of the dataset which is a crucial aspect. Indeed, let’s take the two following cases: –First,wehaveadatasetwhere,usually,thetargetedvaluesrangebetween 0and10.Let’ssayattimet,thealgorithmoutputstheinterval cCt = [5,7] whereas the ...

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.