REVIEW 3 major objections 4 minor 25 references
Relevance-Aware Thresholding in Online Conformal Prediction for Time Series
T0 review · 3 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash
Pith's one-line read This paper claims that replacing the binary inside/outside error in online conformal prediction threshold updates with smooth, distance-aware relevance functions can produce tighter prediction intervals while preserving long-run coverage.
desk verdict Useful empirical idea and an honest paper, but the theory only covers one of the four variants and the ECI experiments don't show the promised narrower intervals. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key object is the smooth relevance family f^{ω,v,µ_t}(x)=Σ_i ω_i σ( (v_i/µ_t) x − ln((1−α)/α) ), with x=s_t−q_t measuring signed distance from ground truth to interval boundary, µ_t the mean absolute distance over a trailing window, and ω, v tunable weights and sharpnesses. It satisfies three design constraints: output in [0,1] like an indicator, scale independence via µ_t, and the static state f(0)=α so a boundary hit causes no update. This function replaces the binary error inside PID and ECI updates, converting the threshold update from a step function into a graded, saturating response.
What would settle it
On any time-series dataset, compute A_T=|Σ_{i≤T}(1{Y_i∉C_i}−α)| and B_T=|Σ_{i≤T}(f_i−α)| for the modified PID update (7). If A_T>B_T at all large T, the Theorem 2 condition is violated; a run where coverage then drifts materially below α would refute the practical usefulness of that variant, while a run where coverage still holds would suggest a weaker condition suffices.
Extended reading notes
Core claim
The paper's central discovery is that the threshold update in online conformal prediction does not have to treat every prediction interval as simply right or wrong. By replacing the indicator 1{Y_t∉C_t} with a smooth function f^{ω,v,µ_t}(s_t−q_t) that rises with the ground truth's distance outside the interval and falls when the ground truth is comfortably inside, the update can react proportionately instead of abruptly. The authors construct f as a weighted sum of sigmoids whose input is scaled by the windowed mean absolute distance µ_t, making it scale-invariant, and they fix the intercept so f(0)=α, meaning a ground truth exactly on the boundary leaves the threshold unchanged. They prove
Load-bearing premise
For two of the modified PID updates, the long-run coverage proof assumes that the cumulative absolute deviation of the smooth relevance scores from α eventually stays at least as large as the cumulative absolute deviation of the binary errors — an inequality the authors acknowledge can fail because the smooth scores often sit close to α.
Editorial extensions
If this is right
- Modified PID update (9) preserves long-run coverage without extra assumptions, and in experiments often produces narrower average and median intervals at fixed coverage.
- Modified ECI update (10) reaches 90% coverage on the tested datasets where the original ECI update undercovered, though with wider intervals.
- The family's parameters v and ω give decision-makers a continuous knob: large v recovers indicator-like behavior, small v smooths aggressively; time-varying choices are possible.
- The coverage guarantee for updates (7) and (8) is conditional on a cumulative-deviation dominance; when that condition holds, the full-relevance variants also keep long-run coverage.
- Since f(0)=α, the update leaves the threshold unchanged when the ground truth lands exactly on the boundary, a property the original binary/ECI smoothing lacks.
Reading between the lines
- The authors chose PID and ECI as carriers, but nothing in the construction is specific to them; replacing the indicator in other online update rules (e.g., simple online gradient descent on quantile loss) with the same relevance family is a direct, testable extension.
- Because µ_t is a windowed mean of absolute distances, the functions may be sensitive to outliers inflating µ_t; a quantile-based scale estimator would be a natural robustness extension.
- The experimental comparison tilts in favor of the modified ECI partly because the original ECI undercovered on these datasets; a fairer width comparison would tune both methods to the same achieved coverage level before comparing interval sizes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes replacing the binary in/out error indicator in online conformal prediction (OCP) threshold updates with a family of smooth "relevance" functions f_{ω,v,μ_t}(s_t−q_t) (Eq. 6), which take values in [0,1], satisfy f(0)=α, and are scale-adaptive through the moving-average normalizer μ_t. The modified PID updates (Eqs. 7–9) and the modified ECI update (Eq. 10) are analyzed for long-run coverage (Eq. 5) and evaluated on four time-series datasets with AR and Theta regressors. Theorems 1 and 2 give coverage guarantees for the PID variants, Theorem 2 conditionally on an unproven dominance assumption; Theorem 3 claims coverage for modified ECI by transferring the original ECI proof. Experiments show reduced interval widths for the modified PID update (Eq. 9) in several settings, while modified ECI achieves nominal coverage but with larger interval widths than the under-covering original ECI in the same settings.
Significance. If the theoretical guarantees were fully established, the proposed relevance-aware thresholding would be a useful and broadly applicable idea for OCP, potentially improving interval efficiency without sacrificing long-run coverage. The paper is clearly written, builds on state-of-the-art methods (PID, ECI), and provides code. However, the two load-bearing theoretical claims—Theorem 2's conditional guarantee and Theorem 3's boundedness assumption—are not adequately justified, and the experimental evidence is limited to point estimates without error bars and with per-dataset parameter tuning. The contribution is therefore promising but not yet at the standard required for publication as a definitive advance.
major comments (3)
- [Theorem 3 / Eq. (10), Appendix A] The proof of Theorem 3 simply states that the original ECI proof holds because f_{ω,v,μ_t} satisfies the boundedness conditions. But for f defined in Eq. (6), ∂f/∂x = Σ ω_i (v_i/μ_t) σ'( (v_i/μ_t)(s_t−q_t) − ln((1−α)/α) ), so |∇f| scales as 1/μ_t. μ_t is a moving average of |s_i−q_i| over the previous T_w steps; nothing in the updates prevents μ_t → 0 (e.g., when q_i exactly equals s_i for many steps). In that regime no finite M exists. Since the boundedness assumption is not proven and can be violated, the long-run coverage guarantee for modified ECI is unsupported. The authors themselves cast doubt on the original ECI proof's strong assumption η > N·B in Appendix B.2, further weakening the transfer.
- [Theorem 2, Section 3 (after Eq. 9), Appendix D.1] The coverage guarantee for the two PID variants in Eqs. (7) and (8) depends on the existence of T′ such that |Σ(1{Y_i∉C_i}−α)| ≤ |Σ(f_i−α)| for all T≥T′. The authors explicitly note this "could be pretty strong" because f is often close to α. No argument or experiment is given to support this dominance condition. Since these two variants are presented as contributions and evaluated in Tables 7–8, their validity claim is conditional on an unverified property. At minimum, the condition should be proved for some nontrivial parameter regime or empirically checked on the tested datasets; otherwise Eqs. (7) and (8) should be presented only as heuristics.
- [Section 4, Tables 1–3, 5–10] The experimental support for the central claim of narrower intervals is mixed. For modified PID (Eq. 9), results are mostly favorable, but for modified ECI, Tables 2–3 and 9–10 show that the interval widths are systematically larger than those of the original ECI in the same setting; the favorable comparison to the original ECI only appears when using the results reported in [18] (Table 4), which were obtained with different settings. Moreover, all results are point estimates without error bars or multiple runs, and parameters (especially v for modified ECI and the learning rate for ECI) are tuned per dataset. This is insufficient to substantiate the claim that relevance-aware thresholding "potentially result[s] in narrower prediction intervals" while maintaining coverage.
minor comments (4)
- [Eq. (15)] The saturation function r_t(x) = KI·tan(x·log(t)) / (t·C_sat) appears to contain a typo: the standard PID saturation uses tanh, not tan. If tan is intended, the behavior at ±π/2 is undefined and the stated property in Eq. (3) needs clarification.
- [Notation for μ_t] In Eq. (6), μ_t = (1/T_w) |Σ_{i=max(t−T_w,1)}^{t−1} (s_i−q_i)| uses only past values, which is fine, but the indexing is inconsistent with the text "previous T_w steps" and with the upper limit t−1. Please align notation.
- [Proof of Theorem 2, Appendix A] The induction proof for Eq. (13) is sketched for the upper bound and says "Doing the same thing with the lower bound," but the lower-bound case is not written out. This is acceptable for an appendix, but the symmetric argument should at least mention that f takes values in [α,1] and that the saturation property is used symmetrically.
- [Related work] The comparison with ECI (Appendix B.2) is useful, but the discussion of the original ECI proof's assumption η > N·B is presented as a remark without a citation. This is an important correctness concern and should be integrated into the main text or properly supported.
Circularity Check
No significant circularity: the few load-bearing guarantees are inherited from external PID/ECI proofs, and the paper's own caveats are flagged as strong assumptions rather than hidden inputs.
full rationale
The paper's derivation chain is not circular in the sense of reducing a prediction to its own fitted inputs. The modified PID coverage statements (Theorems 1 and 2) reuse the external PID saturation-function argument from [1]; Theorem 2 adds an explicit eventual-dominance condition on the cumulative modified errors rather than assuming the desired coverage conclusion. The paper itself warns that this condition 'could be pretty strong,' which is a rigor caveat, not a circular step. The modified ECI result (Theorem 3) is likewise imported from the external ECI proof [18]; the assertion that |∇f| and |x·∇f| are bounded is not derived from the paper's own outputs, and the dependence of ∇f on 1/μ_t means the premise can fail if μ_t → 0. That is an unsupported or possibly false hypothesis, and Appendix B.2 even questions the original ECI proof's strong assumption η > N·B, but this is a correctness gap, not a circular reduction: the cited proof is independent prior work and the boundedness condition is not equivalent to the target long-run coverage. The reported interval-width improvements are empirical comparisons, not quantities forced by construction. The only self-citation, [11], appears in a non-technical remark about decision makers incorporating preferences and carries no mathematical weight. Therefore the appropriate finding is a low non-circularity score, with the main concerns centered on unverified assumptions rather than circularity.
Assumptions & free parameters
free parameters (5)
- slope parameter v =
Not reported in main text; experiments adjust v (Fig. 1 shows v=4 for illustration)
- learning rate η (ECI) =
Selected from a predefined set to give best coverage
- learning rate η (PID) =
0.005
- window size T_w for μ_t =
100
- weights ω =
ω=1, l=1 in experiments
assumptions (4)
- domain assumption Bounded non-conformity scores: s_i∈[-b/2,b/2] in Theorems 1-2, and s_i∈[0,B] in Theorem 3
- domain assumption Saturation function r_t satisfies Equation (3)
- ad hoc to paper Theorem 2 dominance assumption: eventually |Σ(1{Y_i∉C_i}−α)| ≤ |Σ(f_i−α)|
- ad hoc to paper Uniform boundedness of |∇f| and |x∇f(x)| in Theorem 3
Cite this review
Pith. "Pith review of Relevance-Aware Thresholding in Online Conformal Prediction for Time Series." pith.science (2026). https://pith.science/paper/OQLRVLNM
@misc{pith2026251002809,
author = {Pith},
title = {Pith review of: Relevance-Aware Thresholding in Online Conformal Prediction for Time Series},
year = {2026},
howpublished = {\url{https://pith.science/paper/OQLRVLNM}},
note = {Machine review of arXiv:2510.02809}
}
read the original abstract
Uncertainty quantification has received considerable interest in recent works in Machine Learning. In particular, Conformal Prediction (CP) gains ground in this field. For the case of time series, Online Conformal Prediction (OCP) becomes an option to address the problem of data distribution shift over time. Indeed, the idea of OCP is to update a threshold of some quantity (whether the miscoverage level or the quantile) based on the distribution observation. To evaluate the performance of OCP methods, two key aspects are typically considered: the coverage validity and the prediction interval width minimization. Recently, new OCP methods have emerged, offering long-run coverage guarantees and producing more informative intervals. However, during the threshold update step, most of these methods focus solely on the validity of the prediction intervals~--~that is, whether the ground truth falls inside or outside the interval~--~without accounting for their relevance. In this paper, we aim to leverage this overlooked aspect. Specifically, we propose enhancing the threshold update step by replacing the binary evaluation (inside/outside) with a broader class of functions that quantify the relevance of the prediction interval using the ground truth. This approach helps prevent abrupt threshold changes, potentially resulting in narrower prediction intervals. Indeed, experimental results on real-world datasets suggest that these functions can produce tighter intervals compared to existing OCP methods while maintaining coverage validity.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[18]
https://doi.org/10.48550/arXiv.2502.00818, arXiv:2502.00818 [stat]
Wu, J., Hu, D., Bao, Y., Xia, S.T., Zou, C.: Error-quantified Conformal In- ference for Time Series (Feb 2025). https://doi.org/10.48550/arXiv.2502.00818, arXiv:2502.00818 [stat]
-
[1]
In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S
Angelopoulos, A., Candes, E., Tibshirani, R.J.: Conformal PID Control for Time Series Prediction. In: Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., Levine, S. (eds.) Advances in Neural Information Processing Systems. vol. 36, pp. 23047–23074. Curran Associates, Inc. (2023)
2023
-
[2]
https://doi.org/10.48550/arXiv.2402.01139 14 T
Angelopoulos, A.N., Barber, R.F., Bates, S.: Online conformal prediction with decaying step sizes (May 2024). https://doi.org/10.48550/arXiv.2402.01139 14 T. Dupuy et al
-
[3]
Angelopoulos, A.N., Bates, S.: A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification (Dec 2022). https://doi.org/10. 48550/arXiv.2107.07511, arXiv:2107.07511
-
[4]
International Journal of Forecasting16(4), 521–530 (2000)
Assimakopoulos,V.,Nikolopoulos,K.:Thethetamodel:Adecompositionapproach to forecasting. International Journal of Forecasting16(4), 521–530 (2000)
2000
-
[5]
https://doi.org/10.48550/arXiv.1905.02928
Barber, R.F., Candes, E.J., Ramdas, A., Tibshirani, R.J.: Predictive inference with the jackknife+ (May 2020). https://doi.org/10.48550/arXiv.1905.02928
-
[6]
https://doi.org/10.48550/arXiv.2202.13415, http://arxiv.org/abs/2202.13415, arXiv:2202.13415 [stat]
Barber, R.F., Candes, E.J., Ramdas, A., Tibshirani, R.J.: Conformal prediction beyond exchangeability (Mar 2023). https://doi.org/10.48550/arXiv.2202.13415, http://arxiv.org/abs/2202.13415, arXiv:2202.13415 [stat]
-
[7]
Holden-Day, San Francisco, 1st edn
Box, G.E.P., Jenkins, G.M.: Time Series Analysis: Forecasting and Control. Holden-Day, San Francisco, 1st edn. (1970)
1970
Show all 25 references
-
[8]
Mathe- matics of control, signals and systems2(4) (1989)
Cybenko, G.: Approximation by superpositions of a sigmoidal function. Mathe- matics of control, signals and systems2(4) (1989)
1989
-
[9]
In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence
Gammerman, A., Vapnik, V., Vovk, V.: Learning by transduction. In: Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence. Morgan Kaufmann, San Francisco, CA (1998)
1998
-
[10]
In: Advances in Neural Information Processing Systems
Gibbs, I., Candes, E.: Adaptive Conformal Inference Under Distribution Shift. In: Advances in Neural Information Processing Systems. vol. 34, pp. 1660–1672. Curran Associates, Inc. (2021)
2021
-
[11]
Resources, Conservation and Recycling168, art
Jacquin, L., Imoussaten, A., Trousset, F., Didier, P., Montmain, J.: Control of waste fragment sorting process based on MIR imaging coupled with cautious clas- sification. Resources, Conservation and Recycling168, art. 105258 (May 2021). https://doi.org/10.1016/j.resconrec.2020.105258
2021
-
[12]
https://www.kaggle.com/datasets/camnugent/ sandp500 (2018)
Nguyen, C.: S&p 500 stock data. https://www.kaggle.com/datasets/camnugent/ sandp500 (2018)
2018
-
[13]
In: Elomaa, T., Mannila, H., Toivonen, H
Papadopoulos, H., Proedrou, K., Vovk, V., Gammerman, A.: Inductive Confidence Machines for Regression. In: Elomaa, T., Mannila, H., Toivonen, H. (eds.) Machine Learning: ECML 2002. pp. 345–356. Springer, Berlin, Heidelberg (2002). https: //doi.org/10.1007/3-540-36755-1_29
2002 doi
-
[14]
Prinster, D., Stanton, S., Liu, A., Saria, S.: Conformal validity guarantees exist for any data distribution (and how to find them) (2024), https://arxiv.org/abs/2405. 06627
2024
-
[15]
In: Advances in Neural Information Processing Systems
Tibshirani, R.J., Foygel Barber, R., Candes, E., Ramdas, A.: Conformal Pre- diction Under Covariate Shift. In: Advances in Neural Information Processing Systems. vol. 32. Curran Associates, Inc. (2019), https://proceedings.neurips.cc/ paper/2019/hash/8fb21ee7a2207526da55a679f0...
2019
-
[16]
Springer, New York (2005)
Vovk, V., Gammerman, A., Shafer, G.: Algorithmic Learning in a Random World. Springer, New York (2005). https://doi.org/10.1007/978-3-031-06649-8
2005 doi
-
[17]
https://www.kaggle.com/datasets/ sumanthvrao/daily-climate-time-series-data (2017)
Vrao, S.: Daily climate time series data. https://www.kaggle.com/datasets/ sumanthvrao/daily-climate-time-series-data (2017)
2017
-
[19]
In: Proceedings of the 40th International Conference on Machine Learning
Xu, C., Xie, Y.: Sequential Predictive Conformal Inference for Time Series. In: Proceedings of the 40th International Conference on Machine Learning. pp. 38707– 38727. PMLR (Jul 2023), iSSN: 2640-3498
2023
-
[20]
In: Proceedings of the 39th International Conference on Machine Learning
Zaffran, M., Feron, O., Goude, Y., Josse, J., Dieuleveut, A.: Adaptive Conformal Predictions for Time Series. In: Proceedings of the 39th International Conference on Machine Learning. pp. 25834–25866. PMLR (Jun 2022), iSSN: 2640-3498 Relevance-Aware Thresholding in OCP for Tim...
2022
-
[21]
we prove the long-run coverage for the update in equation (11) with(si)i∈N∗ being any sequence of numbers in[−b, b]
-
[22]
This leads to the fact that the update in equation (12) satisfies the long-run coverage
we apply the point 1 onq′ t+1 =q t+1 −ˆqt+1 with(s i)i∈N∗ being any sequence of numbers in[−b 2 , b 2 ]and(ˆq i)i∈N∗ any sequence of numbers in[−b 2 , b 2 ]. This leads to the fact that the update in equation (12) satisfies the long-run coverage. Thus, proving the point 1 is e...
-
[23]
First of all, we propose a general idea that could be generalized to all the OCP methods
-
[24]
They set, in almost all the experiments,λ= 1
In ECI, the authors use the sigmoid functionσ(λ·x). They set, in almost all the experiments,λ= 1. With this function they haveσ(0) = 0.5whereas, we propose a family of functions for whichf ω,v,µ t (0) =α. Indeed, when st =q t (i.e. the ground truth is equal to one of the inter...
-
[25]
The functionsfω,v,µ t try to consider the scale of the dataset which is a crucial aspect. Indeed, let’s take the two following cases: –First,wehaveadatasetwhere,usually,thetargetedvaluesrangebetween 0and10.Let’ssayattimet,thealgorithmoutputstheinterval cCt = [5,7] whereas the ...
Reviewed August 4, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.