Conformalised Conditional Normalising Flows for Joint Prediction Regions in time series

Christoph Lippert; Eshant English

REVIEW 4 major objections 4 minor 21 references

Conformalised Conditional Normalising Flows for Joint Prediction Regions in time series

T0 review · 4 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Using a normalising flow's exact conditional density as a conformity score yields valid, adaptive joint prediction regions for multi-step time-series forecasts, and these regions can be disjoint when the predictive distribution is…

desk verdict A plausible conformal method for time series regions, but the central claim is unsupported because the reported coverage applies to an oracle level set, not the sampled regions users actually get. read the letter →

arxiv 2411.17042 v1 pith:F7WCI52F submitted 2024-11-26 stat.ML cs.LG

Eshant English , Christoph Lippert This is my paper · ORCID

classification stat.MLcs.LG

keywords conformalpredictionnormalisingflowstimeseriesforecastingregionsuncertaintyquantificationmultimodaldistributionsconditionaldensitycoverageguarantees

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes CCN-JPR, a method that attaches finite-sample conformal guarantees to conditional normalising flows for multi-step time-series forecasting. The core move is to use the flow's exact conditional density as a conformity score: a candidate future trajectory belongs to the prediction region if its density under the fitted model is at least the conformal quantile computed from calibration series. Because the density can be evaluated at any point, the region can be read off by sampling or grid search and has no imposed geometric shape; it can split into several disjoint components when the predictive distribution is multimodal. The paper reports coverages closer to the nominal 0.9 level than the baselines on two synthetic particle datasets and the COVID-UK dataset, and argues that no prior conformal time-series method produces such disjoint, adaptive regions.

What carries the argument

The load-bearing object is the conditional normalising flow $g_\theta: z \mapsto x$ with an RNN or transformer summarising the context into a hidden state $h^{(i)}_T$; the emission density $p(x^{(i)}_{T+1:T+H} \mid h^{(i)}_T)$ is obtained through the change-of-variables formula $\log p(x \mid c) = \log p_Z(f^{-1}(x, c) \mid c) + \log |\det J_{f^{-1}}(x, c)|$. The flow is trained by maximum likelihood on the training series. For each calibration or test series, the conformity score is exactly this conditional density evaluated at the true future window. The prediction region is the superlevel set of the density above the conformal threshold, which is what produces adaptive, potentially disjoint regions with no imposed geometry.

What would settle it

Calibrate the flow on early periods of a dataset and evaluate coverage on later periods where the regime has shifted, for example COVID-UK before versus during a case surge, keeping the fitted model fixed. If empirical coverage falls well below the nominal 0.9 level while the model is unchanged, the exchangeability assumption carrying the conformal guarantee has been violated.

Watch

Extended reading notes

Core claim

The central claim is that the probability density a fitted conditional normalising flow assigns to a candidate forecast trajectory is a valid conformity score for conformal prediction, and that this construction is especially useful for multi-step time-series forecasting. Each calibration series contributes the score $\alpha_i = p_{\hat{\theta}}(x^{(i)}_{T+1:T+H} \mid h^{(i)}_T)$, where the density is computed exactly by the change-of-variables formula; taking a conformal quantile $q_\epsilon$ of these scores gives a threshold, and the forecast region is the superlevel set $\{x : p_{\hat{\theta}}(x \mid h_T) \ge q_\epsilon\}$. Since the region is defined by a density threshold rather than by a residual norm or a geometric shape, it adapts to the history through the conditioning variable and can split into several disconnected clusters when the fitted predictive distribution is multimodal. The paper reports empirical coverages of 0.91, 0.89 and 0.87 at the nominal 0.9 level on two synthetic particle datasets and COVID-UK respectively.

Load-bearing premise

For the finite-sample coverage guarantee to hold, calibration and test time series must be exchangeable, so that the conditional-density scores of the true futures are interchangeable; the paper assumes independent multivariate time series but never states or stress-tests this condition.

Editorial extensions

If this is right

Prediction regions for multi-step forecasts no longer need to be bands, ellipsoids, or boxes: any superlevel set of the fitted conditional density is allowed, so a multimodal forecast can be covered by several small regions instead of one large one.
Coverage is finite-sample and distribution-free at the level $1-\epsilon$ whenever the usual conformal exchangeability condition holds, so practitioners can quote a coverage guarantee even when the flow model is misspecified.
The construction transfers directly to multivariate regression, since time-series forecasting with independent series can be seen as a special case of it.
The practical bottleneck is the cost of exploring the $H$-dimensional label space via grid or Monte Carlo sampling, so efficient sampling schemes would determine how large the forecast horizon can be.
Because the region is a density superlevel set, varying $\epsilon$ traces out a nested family of regions, which supports settings where a user wants a spectrum of uncertainty sets rather than a single one.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Any model with an exact or approximate conditional likelihood could in principle use the same density-threshold conformal recipe; the normalising flow is a clean instance, not a necessary one.
The efficiency advantage over single convex regions is only as strong as the flow's ability to learn the true modes; on data where the flow is unimodal but the truth is multimodal, the conformal region will be valid but may still be inefficient, so a direct volume comparison on synthetic bimodal series would quantify the gain.
The paper's coverage numbers are point estimates on three datasets; a stress test with temporal shift, calibrating early and testing late, would show where the exchangeability assumption binds.
The density-superlevel construction could be combined with other generative models that supply tractable likelihoods, such as autoregressive models with density evaluation, to obtain shape-free conformal regions outside the flow setting.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this paper combines conditional normalising flows with split conformal prediction in a way I haven't seen before for multi-step time series, using the conditional density as the conformity score. That is a legitimate extension. But the paper doesn't actually demonstrate the thing it claims—valid, efficient, disjoint prediction regions—because the regions are constructed by Monte Carlo sampling, and the coverage numbers are computed by checking the density threshold, not the sampled set. So the central claim is currently unsupported.

What's new: prior conformal methods for time series produce continuous bands, ellipsoids, or rectangular regions; using a CNF's conditional density as the conformity score can in principle produce disjoint clusters. The paper's literature review correctly distinguishes it from Colombo (2024), which uses flows on errors rather than density as score. The conformal machinery is standard and would give finite-sample coverage under exchangeability of the independent time series.

What it does well: the writing is clear, the method is simple to describe, and the authors are honest about the missing pieces—Section 6 admits the computational bottleneck and the lack of a multimodal dataset. The coverage numbers on the three datasets are close to nominal, which at least doesn't contradict the validity claim for the oracle rule.

Where it's soft: the stress-test note is right. Section 4 defines the region as the level set {y: p(y|x)>q}, then constructs it by grid or Monte Carlo sampling. With H=24 and D=2, the label space is 48-dimensional. A finite sample is not the level set. The split-conformal guarantee applies to the level set, not to the finite-sample approximation. No error bound or conservative correction is given. So the reported coverage (checking whether the true value passes the threshold) is not the coverage of the object a user receives. That's a load-bearing gap. On top of that, there is no volume or efficiency metric and no multimodal dataset, so the efficiency claim for disjoint regions is untested. These are fixable: state the exchangeability condition, prove or at least formalize the guarantee for the sampled region (or evaluate the sampled region directly), add region-size measurements, and run a synthetic multimodal example.

The exchangeability issue is minor—the paper says 'independent' series, which implies exchangeability if they're also identically distributed, but it never says so. Easy to fix.

Who it's for: people working on conformal prediction for time series, especially those interested in non-rectangular regions. It's a workshop-level contribution with a solid idea but incomplete evaluation. A serious referee could push it into a publishable form.

Recommendation: send it to review, but flag the sampling-versus-level-set gap as the main issue. It deserves a chance to be fixed.

Referee Report

4 major / 4 minor

Summary. The paper proposes a method (CCN-JPR) to construct joint prediction regions for multi-step time series forecasting by combining conditional normalising flows with split conformal prediction. The conditional density of the flow is used as a conformity score, and a prediction region is formed by taking the level set of that density above a calibrated quantile. The authors claim this yields adaptive and potentially disjoint regions that can improve efficiency for multimodal predictive distributions. They report coverage results on two synthetic and one real dataset, concluding that their method achieves coverage close to nominal.

Significance. The core idea is a natural and promising combination of two well-established frameworks: the exact density evaluation of normalising flows and the distribution-free validity of conformal prediction. If the validity and efficiency claims were rigorously established, the method could be a useful contribution to probabilistic forecasting, especially for multivariate or multi-step settings where existing conformal baselines produce conservative rectangular regions. However, the paper currently lacks a formal theorem, and the empirical evidence is far too thin to support the claimed advantages, particularly with respect to disjoint regions and multimodal predictive distributions. No code or reproducible implementation is provided, so the results cannot be independently checked.

major comments (4)

[Section 4 and Section 5] The conformal guarantee is described for the oracle level set {y: p(y|x) > q_epsilon}, but the actual output region is a finite set of Monte Carlo or grid samples that satisfy this threshold. The paper never proves coverage for this sampled region, nor does it provide a conservative correction. The empirical evaluation in Section 5 checks if the true value passes the density threshold, not whether it lies in the sampled region a user would actually receive. Consequently, the central claim of finite-sample valid prediction regions is unsupported for the object that the method actually deploys.
[Section 6] The paper explicitly acknowledges the lack of a standard dataset exhibiting multimodal predictive distributions, yet the abstract and introduction claim improved predictive efficiency in the presence of potential multimodal distributions. No volume or efficiency metric is reported anywhere, so the claimed benefit of disjoint regions is entirely untested. A paper making this claim needs at least one multimodal demonstration and a quantitative measure of region size or volume.
[Section 3.1 and Section 4] The paper does not state a formal theorem or explicitly specify the exchangeability condition required for the conformal guarantee. It should clearly state that the calibration and test series are exchangeable (or i.i.d.) and provide a formal statement of the coverage guarantee for the oracle level set, including a precise definition of the quantile q_epsilon (e.g., the ceil((l+1)(1-epsilon))/l-th order statistic of the calibration scores). Without this, the theoretical foundation is incomplete.
[Table 1] The empirical results report only a single coverage number per dataset, with no error bars, standard errors, or repeated trials, and the numbers of calibration and test series are not stated. Given that conformal prediction provides a finite-sample guarantee, the reported coverages should be accompanied by standard errors to demonstrate consistency with the nominal level. The table also uses the name "CCNF-JPR" while the method is introduced as "CCN-JPR" in Section 4.

minor comments (4)

[Section 3 title] The word "Preleminaries" should be "Preliminaries".
[Section 4] The sentence "we can use the calibration set to calculate the conditional density around the true prediction" would benefit from a precise description of how the density is evaluated for each calibration series (i.e., the flow density at the observed future vector conditioned on the context window).
[Related works] The comparison with English et al. (2024) could be more precise: both works propose adaptive conformity scores for time series, and the novelty of the current method lies in using the flow density itself as the score and in producing unrestricted region shapes.
[Section 3.1] A citation to a standard reference for conformal prediction (e.g., Vovk et al., 2005, or Angelopoulos and Bates, 2022) would help readers unfamiliar with the exchangeability assumption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the conformal threshold is computed from held-out calibration scores and the central claim is not forced by construction.

full rationale

The derivation chain is self-contained. The conformity score is the conditional density evaluated by a trained conditional normalising flow, and the threshold q_epsilon is the empirical quantile of calibration-set density scores. This is exactly split-conformal inversion (Section 3.1, Eq. 1, applied to the density score from Section 4), and the finite-sample coverage property is inherited from exchangeability of calibration and test scores, not from any parameter fitted to the test outcome. The paper's self-citations (English et al. 2023, English et al. 2024) appear only as contextual comparisons in Related Work and Discussion, not as load-bearing premises; no uniqueness theorem or fitted result is imported from them. The main validity caveat is that Section 5 reports coverage by 'checking if the true value is accepted by the conditional density threshold used as a conformity score', whereas a user actually receives a finite Monte-Carlo/grid approximation of the oracle level set in a 48-dimensional label space, and Section 6 acknowledges the computational bottleneck. This is a correctness or approximation gap, not a circular reduction: the sampled region is not constructed so as to make the coverage true by definition. No equation is shown to be equivalent to its own input, and no fitted parameter is renamed as a prediction. Therefore no circular step is exhibited.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The conformal guarantee rests on standard exchangeability and the flow being a valid density model. The paper introduces no new entities or exotic free parameters, but it does not report the model's fitted hyperparameters or the sampling budget.

free parameters (2)

Conditional normalising flow architecture and hyperparameters = not reported
The conformity score is the learned density, so the prediction regions depend on the trained model. The paper does not specify the number of coupling layers, learning rate, or other training details.
Monte Carlo or grid sampling budget for region construction = not reported
The paper says regions are formed by sampling points from the label space, but it does not state how many samples are used, which affects the region representation and computational cost.

assumptions (3)

domain assumption Exchangeability of calibration and test series
Section 4 assumes independent multivariate time series; conformal validity requires exchangeability of the conformity scores, which is not explicitly stated or verified.
standard math Change-of-variables formula for the conditional normalising flow
Section 3.2 relies on the density computed via an invertible transformation; this requires the flow to be differentiable and invertible.
standard math Inductive conformal prediction quantile construction
Equation 1 and the quantile step assume the standard machinery of inductive conformal predictors, including the rank-based p-value.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Conformalised Conditional Normalising Flows for Joint Prediction Regions in time series." pith.science (2026). https://pith.science/paper/F7WCI52F

@misc{pith2026241117042,
  author       = {Pith},
  title        = {Pith review of: Conformalised Conditional Normalising Flows for Joint Prediction Regions in time series},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F7WCI52F}},
  note         = {Machine review of arXiv:2411.17042}
}

read the original abstract

Conformal Prediction offers a powerful framework for quantifying uncertainty in machine learning models, enabling the construction of prediction sets with finite-sample validity guarantees. While easily adaptable to non-probabilistic models, applying conformal prediction to probabilistic generative models, such as Normalising Flows is not straightforward. This work proposes a novel method to conformalise conditional normalising flows, specifically addressing the problem of obtaining prediction regions for multi-step time series forecasting. Our approach leverages the flexibility of normalising flows to generate potentially disjoint prediction regions, leading to improved predictive efficiency in the presence of potential multimodal predictive distributions.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

21 extracted references · 3 canonical work pages

[1]

Angelopoulos and Stephen Bates

Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification, 2022

2022
[2]

Robust uncertainty quantification using conformalised monte carlo prediction, 2024

Daniel Bethell, Simos Gerasimou, and Radu Calinescu. Robust uncertainty quantification using conformalised monte carlo prediction, 2024

work page 2024
[3]

Pappas, and Lars Lindemann

Matthew Cleaveland, Insup Lee, George J. Pappas, and Lars Lindemann. Conformal prediction regions for time series using linear complementarity programming, 2024

2024
[4]

Normalizing flows for conformal regression, 2024

Nicolo Colombo. Normalizing flows for conformal regression, 2024. URL https://arxiv.org/abs/2406.03346

arXiv 2024
[5]

Kernelised normalising flows, 2023

Eshant English, Matthias Kirchler, and Christoph Lippert. Kernelised normalising flows, 2023

2023
[6]

Janet: Joint adaptive prediction-region estimation for time-series, 2024

Eshant English, Eliot Wong-Toi, Matteo Fontana, Stephan Mandt, Padhraic Smyth, and Christoph Lippert. Janet: Joint adaptive prediction-region estimation for time-series, 2024. URL https://arxiv.org/abs/2407.06390

arXiv 2024
[7]

Multi-scale attention flow for probabilistic time series forecasting, 2023

Shibo Feng, Chunyan Miao, Ke Xu, Jiaxiang Wu, Pengcheng Wu, Yang Zhang, and Peilin Zhao. Multi-scale attention flow for probabilistic time series forecasting, 2023

2023
[8]

Conformal approach to gaussian process surrogate evaluation with coverage guarantees, 2024

Edgar Jaber, Vincent Blot, Nicolas Brunel, Vincent Chabridon, Emmanuel Remy, Bertrand Iooss, Didier Lucor, Mathilde Mougeot, and Alessandro Leite. Conformal approach to gaussian process surrogate evaluation with coverage guarantees, 2024

work page 2024

Show all 21 references

[9]

Neural relational inference for interacting systems, 2018

Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard Zemel. Neural relational inference for interacting systems, 2018

2018
[10]

Prince, and Marcus A

Ivan Kobyzev, Simon J.D. Prince, and Marcus A. Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43 0 (11): 0 3964--3979, nov 2021. doi:10.1109/tpami.2020.2992934. URL https://doi.org/10.1109

2021
[11]

Understanding diffusion models: A unified perspective, 2022

Calvin Luo. Understanding diffusion models: A unified perspective, 2022

2022
[12]

Gaussianization flows, 2020

Chenlin Meng, Yang Song, Jiaming Song, and Stefano Ermon. Gaussianization flows, 2020

2020
[13]

Copula-based conformal prediction for multi-target regression, 2021

Soundouss Messoudi, Sébastien Destercke, and Sylvain Rousseau. Copula-based conformal prediction for multi-target regression, 2021

2021
[14]

Normalizing flows for probabilistic modeling and inference, 2021

George Papamakarios, Eric Nalisnick, Danilo Jimenez Rezende, Shakir Mohamed, and Balaji Lakshminarayanan. Normalizing flows for probabilistic modeling and inference, 2021

2021
[15]

Multivariate probabilistic time series forecasting via conditioned normalizing flows, 2021

Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multivariate probabilistic time series forecasting via conditioned normalizing flows, 2021

2021
[16]

Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations, 2021. URL https://arxiv.org/abs/2011.13456

2021 arXiv
[17]

Alaa, and Mihaela van der Schaar

Kamile Stankeviciute, Ahmed M. Alaa, and Mihaela van der Schaar. Conformal Time -series Forecasting . In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 6216--6228. Curra...

2021
[18]

Copula conformal prediction for multi-step time series prediction

Sophia Huiwen Sun and Rose Yu. Copula conformal prediction for multi-step time series prediction. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=ojIJZDNIBj

2024
[19]

Algorithmic learning in a random world, volume 29

Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005

2005
[20]

Conformal prediction for multi-dimensional time series by ellipsoidal sets, 2024

Chen Xu, Hanyang Jiang, and Yao Xie. Conformal prediction for multi-dimensional time series by ellipsoidal sets, 2024

2024
[21]

Conformalized adaptive forecasting of heterogeneous trajectories, 2024

Yanfei Zhou, Lars Lindemann, and Matteo Sesia. Conformalized adaptive forecasting of heterogeneous trajectories, 2024. URL https://arxiv.org/abs/2402.09623

2024 arXiv

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Angelopoulos and Stephen Bates

Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification, 2022

2022

[2] [2]

Robust uncertainty quantification using conformalised monte carlo prediction, 2024

Daniel Bethell, Simos Gerasimou, and Radu Calinescu. Robust uncertainty quantification using conformalised monte carlo prediction, 2024

work page 2024

[3] [3]

Pappas, and Lars Lindemann

Matthew Cleaveland, Insup Lee, George J. Pappas, and Lars Lindemann. Conformal prediction regions for time series using linear complementarity programming, 2024

2024

[4] [4]

Normalizing flows for conformal regression, 2024

Nicolo Colombo. Normalizing flows for conformal regression, 2024. URL https://arxiv.org/abs/2406.03346

arXiv 2024

[5] [5]

Kernelised normalising flows, 2023

Eshant English, Matthias Kirchler, and Christoph Lippert. Kernelised normalising flows, 2023

2023

[6] [6]

Janet: Joint adaptive prediction-region estimation for time-series, 2024

Eshant English, Eliot Wong-Toi, Matteo Fontana, Stephan Mandt, Padhraic Smyth, and Christoph Lippert. Janet: Joint adaptive prediction-region estimation for time-series, 2024. URL https://arxiv.org/abs/2407.06390

arXiv 2024

[7] [7]

Multi-scale attention flow for probabilistic time series forecasting, 2023

Shibo Feng, Chunyan Miao, Ke Xu, Jiaxiang Wu, Pengcheng Wu, Yang Zhang, and Peilin Zhao. Multi-scale attention flow for probabilistic time series forecasting, 2023

2023

[8] [8]

Conformal approach to gaussian process surrogate evaluation with coverage guarantees, 2024

Edgar Jaber, Vincent Blot, Nicolas Brunel, Vincent Chabridon, Emmanuel Remy, Bertrand Iooss, Didier Lucor, Mathilde Mougeot, and Alessandro Leite. Conformal approach to gaussian process surrogate evaluation with coverage guarantees, 2024

work page 2024