Pith. sign in

REVIEW 4 major objections 6 minor 19 references

On Uncertainty Prediction for Deep-Learning-based Particle Image Velocimetry

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper argues that a learned uncertainty neural network can put per-pixel error bars on deep-learning-based PIV velocities, scoring above ensemble and transform sampling on in-distribution data while sampling methods retain an edge on…

desk verdict The 'consistent best' claim for UNN is refuted by the paper's own Table 4, and the metric definitions are internally inconsistent; the benchmark framing has value but the paper cannot be accepted as is. read the letter →

arxiv 2507.20102 v1 pith:WGLXTXH6 submitted 2025-07-27 eess.IV

Wei Wang , Jeremiah Hu , Jia Ai , Yong Lee This is my paper · ORCID
classification eess.IV
keywords particleimagevelocimetryuncertaintyquantificationdeeplearningRAFTopticalflowU-netGaussiannegativelog-likelihoodsparsificationplotcoveragerate
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Particle image velocimetry (PIV) extracts flow velocities from pairs of particle images, and deep-learning versions of it are fast but give no error bars. This paper tries to fix that by comparing three ways of attaching uncertainty to the velocities predicted by a RAFT network: an uncertainty neural network (UNN) that regresses per-pixel uncertainty from the flow field and images, an ensemble of four models (MM), and a set of four rotated transforms of the input (MT). On the in-distribution CAI benchmark the paper reports that UNN gives the most accurate uncertainty estimates by all three metrics it uses—95% coverage, Spearman's rank correlation, and sparsification AUC—while MM and MT generalize better on unseen Sintel and rotating-flow images. A sympathetic reading of the central claim is that one-pass learned uncertainty can replace expensive sampling for in-distribution PIV, so long as the evaluation metrics are correctly calibrated.

What carries the argument

The load-bearing object is the UNN itself: a U-net regression head whose input is the concatenation of the RAFT-predicted velocity and the original particle image pair, and whose output is a per-pixel uncertainty $\sigma$. Training minimizes the Gaussian negative log-likelihood, so the network is rewarded for predicting large $\sigma$ exactly where the squared velocity error $e^2$ is large; this is the mechanism that lets a single forward pass produce error-sensitive uncertainty maps. The two comparators unpack the same idea statistically: MM averages four independently trained RAFT instances and takes their spread as uncertainty, MT feeds four rotations (0°, 90°, 180°, 270°) through one RAFT and takes the spread of the rotated-back fields as uncertainty. The evaluation machinery consists of three metrics, 95% coverage, Spearman's rank correlation, and sparsification AUC, whose exact forms matter for the ranking.

What would settle it

On the CAI test set, form the normalized error $e_i/\sigma_i$ for each method and compute the empirical fraction of points with $|e_i| < 1.96\sigma_i$; if UNN's fraction is no closer to 95% than MM's or MT's, the claim that UNN provides accurate 95% coverage fails. Independently, recompute the AUC by direct numerical integration of the sparsification curves shown in Figure 4 and compare the resulting rankings with Table 1; a changed ranking would show the headline comparison depends on the specific AUC formula in Eq. (4).

Watch

Extended reading notes

Core claim

The central discovery, stated on the paper's own terms, is that an end-to-end uncertainty neural network—a U-net that consumes the concatenation of a RAFT-predicted velocity field and the particle image pair and is trained with the Gaussian negative log-likelihood loss $L = -\frac{1}{N}\sum_{i=1}^{N}\left[\log\sigma_i + \frac{e_i^2}{2\sigma_i^2}\right]$—yields per-pixel velocity uncertainties that rank first on the CAI test set under 95% coverage, Spearman's rank correlation, and sparsification AUC, ahead of multiple-models and multiple-transforms sampling. The authors take this as evidence that learned single-pass uncertainty is a viable, computationally cheaper alternative to ensemble sampling for deep-learning PIV, while acknowledging that MM and MT generalize better on unseen Sintel and rotating-flow data and that all three methods degrade under strong Gaussian blur or noise.

Load-bearing premise

The comparison rests on the coverage formula in Eq. (2) counting an error as covered when the absolute error is below the predicted uncertainty and treating that rate as the 95% two-sigma target; if the network's predicted uncertainty is a one-standard-deviation quantity, the expected rate is about 68%, not 95%, and the ranking could be miscalibrated.

Editorial extensions

If this is right

  • A single UNN forward pass can attach per-pixel uncertainty to RAFT velocity fields on in-distribution PIV data, replacing four model evaluations or four image rotations and making uncertainty-aware flow measurement practical for real-time use.
  • The three-metric evaluation protocol—95% coverage, Spearman's CC, sparsification AUC against an Oracle curve—can serve as a standard benchmark for future PIV uncertainty methods.
  • Under mild Gaussian noise or blur the uncertainty estimates remain usable, but under strong degradation none of the three methods should be trusted, so error bars should carry a confidence flag tied to input quality.
  • For out-of-distribution flows, the paper's results imply that sampling-based uncertainty (MM or MT) is the safer default, while the single-pass UNN is the better default only when the test flow resembles the training distribution.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The reported in-distribution coverage values (0.83–0.90) sit below the nominal 95% target while the metric is defined as $|e| < \sigma$, which suggests recalibrating the threshold to $1.96\sigma$—or rescaling $\sigma$ post hoc—could move UNN closer to the target and might alter the method ranking; this is a direct, testable check of the headline claim.
  • Because the UNN loss assumes per-pixel Gaussian errors, swapping the negative log-likelihood for a heavy-tailed or mixture-density loss should improve coverage at flow boundaries and occlusions, the regions where PIV errors are largest and least Gaussian.
  • A deep ensemble of several UNN heads, trained with different initializations, would combine UNN's calibration with MM's generalization advantage; the paper neither tests nor rules out this hybrid.
  • Recomputing the AUC by trapezoidal integration of the sparsification curves in Figure 4, rather than applying Eq. (4), would verify whether UNN's reported advantage on that metric is robust to the formula's exact form.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript compares three uncertainty-quantification approaches for deep-learning-based particle image velocimetry (PIV): an Uncertainty Neural Network (UNN) that regresses per-pixel uncertainty from the RAFT velocity estimate and the image pair, a Multiple Models (MM) ensemble of four RAFT instances, and a Multiple Transforms (MT) method that averages predictions from four rotated versions of the input. The authors define three evaluation metrics—95% coverage, Spearman rank correlation, and sparsification area under the curve—and report results on a held-out test split of the CAI dataset, on CAI images corrupted by Gaussian noise and blur, and on unseen Sintel and rotating-flow images. The abstract and Section 2 claim that UNN consistently achieves the best uncertainty estimates, while Section 3.4 and the conclusion acknowledge that MM and MT generalize better on unseen data.

Significance. Uncertainty quantification for deep-learning-based PIV is an important and under-studied problem, and the paper usefully identifies three practical strategies and evaluates them on multiple datasets. The computational-efficiency argument for a single forward-pass UNN is attractive if it holds. However, the significance of the reported comparison is undermined by three problems: the 95% coverage metric is mis-specified, the AUC formula is not properly normalized and produces impossible values, and the headline claim of 'consistently best' is contradicted by the paper's own results in Table 4. The paper's conclusion is more careful than its abstract, which is a point in its favor, but the quantitative basis for the central claim is not reliable as written.

major comments (4)
  1. [Abstract and §3.4, Table 4] The claim that UNN consistently achieves the best performance is directly contradicted by Table 4: on Sintel alley 1, bamboo 1, and the rotating flow, UNN 95% coverage is 0.05, 0.12, and 0.26, whereas MM achieves 0.57, 0.67, and 0.79 and MT achieves 0.49, 0.63, and 0.84. Coverage near 0.05 means the predicted uncertainty intervals capture essentially none of the actual errors, so UNN's uncertainty estimates are not accurate on the generalization task the paper sets up. The conclusion itself states that MM and MT show better generalization, so the abstract's blanket claim is unsupported by the manuscript's own evidence.
  2. [§3.1.1, Eq. (2)] The 95% coverage definition is inconsistent with the text. Equation (2) counts the fraction of pixels with |e_i| < sigma_i, which for Gaussian errors has an expected value around 68%, not 95%; to obtain 95% the threshold should be c*sigma_i with c near 1.96, or the definition should explicitly use the 2-sigma interval. As written, the reported coverage values cannot be interpreted as 95% coverage, and any comparison based on closeness to 0.95 is not well defined.
  3. [§3.1.3, Eq. (4)] The AUC formula is not a valid normalized sparsification metric. It omits the division by the number of retained points (1/i) and therefore can exceed 1; Table 3 reports AUC values of 1.08 and 1.04, which are impossible for the normalized sparsification curve described in the text. This makes the AUC columns in Tables 1-4 incomparable and undermines the claim that UNN is best on CAI in terms of AUC.
  4. [§3.3, Table 2] Even on the in-distribution CAI test set under strong perturbation, UNN is not consistently best: at var = 10 the Spearman correlation is 0.28 for UNN versus 0.43 for MT and 0.37 for MM, and the AUC of UNN (0.80) is no better than MM's (0.80) and worse than MT's (0.76). Thus the 'consistently best' claim is false even before considering unseen data.
minor comments (6)
  1. [§2.3] The text describes MT as 'random sampling and statistical analysis based on the Monte Carlo method,' but the method uses four fixed rotations (0, 90, 180, 270 degrees); this is not random sampling and should be described as deterministic multi-transform ensembling.
  2. [§2.2] The manuscript does not specify how the four RAFT models differ (for example, initialization seeds or training subsets), nor does it state whether the 'uncertainty' is the standard deviation or variance of the four outputs; this should be clarified for reproducibility.
  3. [§3.2, Figure 4] The figure layout and caption do not clearly identify which subplot corresponds to ground truth, prediction, or uncertainty for each of the three samples; the caption should enumerate the panels explicitly.
  4. [General] No code or data availability statement is provided; given the issues with the evaluation metrics, an anonymous code release would be necessary for independent verification.
  5. [References] Several references have inconsistent formatting (for example, author initials are sometimes expanded and sometimes not), and the paper should use a consistent citation style throughout.
  6. [§3.4] The generalization experiments use only three test cases (two Sintel scenes and one rotating-flow set), with no error bars or repeated trials; the conclusions about generalization would be more convincing with more scenes and statistical confidence measures.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the uncertainty comparison is an empirical held-out benchmark, and the paper's internal metric inconsistencies are correctness issues rather than circular steps.

full rationale

The paper compares three uncertainty methods (UNN, MM, MT) on held-out CAI test images, perturbed CAI images, and unseen Sintel and rotating-flow images. UNN is trained on the errors of the pre-trained RAFT model on the CAI training split, but the CAI test set is disjoint and the other methods use the same velocity backbone, so the comparison is an empirical evaluation rather than a derivation that assumes its own conclusion. None of the evaluation equations (Eqs. 2–5) is defined in terms of a particular method, and no fitted parameter is renamed as a prediction. The self-citations in the references are background and contextual; none carries a load-bearing uniqueness claim or smuggles in the target result. The paper's own Table 4 and conclusion explicitly report that UNN generalizes poorly on unseen images, which contradicts the abstract's blanket 'consistently best' statement, but that is a factual/consistency limitation, not circularity. The 95% coverage definition in Eq. (2) is not consistent with the 2-sigma Gaussian interpretation in the text, and Eq. (4) is not the standard sparsification AUC and can produce values above 1, yet these issues apply equally to all three methods and do not make any method's advantage true by construction. The central claims therefore rest on empirical evidence, not on a self-referential chain, and no circular step can be exhibited from the paper's equations or citations.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. It relies on a Gaussian error assumption, a domain-transfer assumption, and hand-chosen perturbation levels. No new forces, particles, or conserved quantities are proposed.

free parameters (1)
  • Perturbation levels for robustness tests = noise var = 0, 5, 10; blur sigma = 0, 2.5, 5
    Hand-chosen thresholds define the 'mild perturbation' regime in which the paper claims all three methods perform well; the central claim is scoped to these values.
assumptions (3)
  • domain assumption PIV velocity errors are zero-mean Gaussian with pixel-wise variance sigma^2
    Eq. (1) maximizes a Gaussian likelihood; if errors are heavy-tailed or spatially correlated, the uncertainty estimates will be miscalibrated.
  • domain assumption RAFT-PIV errors on CAI are representative of errors in the deployment domain
    UNN is trained on the error between RAFT predictions and CAI ground truth; generalization to Sintel and rotating flow is assumed, but Table 4 contradicts this.
  • ad hoc to paper Coverage |e| < sigma is a valid 95% coverage criterion and the AUC formula in Eq. (4) is correct
    The text invokes the 2-sigma Gaussian rule while Eq. (2) uses a 1-sigma threshold; Eq. (4) is a nonstandard AUC definition. The method ranking depends on these metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of On Uncertainty Prediction for Deep-Learning-based Particle Image Velocimetry." pith.science (2026). https://pith.science/paper/WGLXTXH6

@misc{pith2026250720102,
  author       = {Pith},
  title        = {Pith review of: On Uncertainty Prediction for Deep-Learning-based Particle Image Velocimetry},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WGLXTXH6}},
  note         = {Machine review of arXiv:2507.20102}
}
read the original abstract

Particle Image Velocimetry (PIV) is a widely used technique for flow measurement that traditionally relies on cross-correlation to track the displacement. Recent advances in deep learning-based methods have significantly improved the accuracy and efficiency of PIV measurements. However, despite its importance, reliable uncertainty quantification for deep learning-based PIV remains a critical and largely overlooked challenge. This paper explores three methods for quantifying uncertainty in deep learning-based PIV: the Uncertainty neural network (UNN), Multiple models (MM), and Multiple transforms (MT). We evaluate the three methods across multiple datasets. The results show that all three methods perform well under mild perturbations. Among the three evaluation metrics, the UNN method consistently achieves the best performance, providing accurate uncertainty estimates and demonstrating strong potential for uncertainty quantification in deep learning-based PIV. This study provides a comprehensive framework for uncertainty quantification in PIV, offering insights for future research and practical implementation.

Figures

Figures reproduced from arXiv: 2507.20102 by the authors.

Figure 1
Figure 1. The process of UNN method. Our UNN predicts the measurement uncertainty via a neural [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The process of multiple models method. We use several different neural networks to deal with [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. The process of multiple transforms method. The flow images which are rotated to different [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: The performance of three methods on sample data. [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: The performance of unseen images in three groups, each group is consist of a ground flow [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

19 extracted references · 13 canonical work pages

  1. [1]

    (2021) A review of uncertainty quantification in deep learning: Techniques, applications and challenges

    Abdar M, Pourpanah F, Hussain S, Rezazadegan D, Liu L, Ghavamzadeh M, Fieguth P, Cao X, Khosravi A, Acharya UR et al. (2021) A review of uncertainty quantification in deep learning: Techniques, applications and challenges. Information fusion 76:243--297

    work page 2021

  2. [2]

    Applied optics 23:1690--1691

    Adrian RJ (1984) Scattering particle characteristics and their effect on pulsed laser measurements of fluid flow: speckle velocimetry vs particle image velocimetry. Applied optics 23:1690--1691

    work page 1984

  3. [3]

    Physics of Fluids 37

    Ai J, Chen Z, Li J, and Lee Y (2025) Rethinking asymmetric image deformation with post-correction for particle image velocimetry. Physics of Fluids 37

    work page 2025

  4. [4]

    Measurement Science and Technology 29:115301

    Bhattacharya S, Charonko JJ, and Vlachos PP (2018) Particle image velocimetry (piv) uncertainty quantification using moment of correlation (mc) plane. Measurement Science and Technology 29:115301

    work page 2018

  5. [5]

    Experiments in Fluids 60:1--16

    Cai S, Zhou S, Xu C, and Gao Q (2019) Dense motion estimation of particle images via a convolutional neural network. Experiments in Fluids 60:1--16

    2019

  6. [6]

    Nature Machine Intelligence 3:641--651

    Lagemann C, Lagemann K, Mukherjee S, and Schr \"o der W (2021) Deep recurrent optical flow learning for particle image velocimetry data. Nature Machine Intelligence 3:641--651

    work page 2021

  7. [7]

    Physics of Fluids 36

    Lee Y, Gu F, Gong Z, Pan D, and Zeng W (2024) Surrogate-based cross-correlation for particle image velocimetry. Physics of Fluids 36

    2024

  8. [8]

    IEEE Transactions on Instrumentation and Measurement 71:1--10

    Lee Y and Mei S (2021) Diffeomorphic particle image velocimetry. IEEE Transactions on Instrumentation and Measurement 71:1--10

    2021

Show all 19 references
  1. [9]

    Experiments in Fluids 58:1--10

    Lee Y, Yang H, and Yin Z (2017) Piv-dcnn: cascaded deep convolutional neural networks for particle image velocimetry. Experiments in Fluids 58:1--10

    2017

  2. [10]

    Measurement Science and Technology 32:104003

    Morrell MC, Hickmann K, and Wilson BM (2021) Particle image velocimetry analysis with simultaneous uncertainty quantification using bayesian neural networks. Measurement Science and Technology 32:104003

    2021

  3. [11]

    springer

    Raffel M, Willert CE, Scarano F, K \"a hler CJ, Wereley ST, and Kompenhans J (2018) Particle image velocimetry: a practical guide. springer

    2018

  4. [12]

    Measurement Science and Technology 30:092001

    Sciacchitano A (2019) Uncertainty quantification in particle image velocimetry. Measurement Science and Technology 30:092001

    2019

  5. [13]

    Experiments in fluids 53:1133--1147

    Timmins BH, Wilson BW, Smith BL, and Vlachos PP (2012) A method for automatic estimation of instantaneous local uncertainty in particle image velocimetry measurements. Experiments in fluids 53:1133--1147

    2012

  6. [14]

    in Proceedings of the IEEE international conference on computer vision

    Wannenwetsch AS, Keuper M, and Roth S (2017) Probflow: Joint optical flow and uncertainty estimation. in Proceedings of the IEEE international conference on computer vision. pages 1173--1182

    2017

  7. [15]

    Measurement Science and Technology 26:074002

    Wieneke B (2015) Piv uncertainty quantification from correlation statistics. Measurement Science and Technology 26:074002

    2015

  8. [16]

    Measurement Science and Technology 26:074001

    Xue Z, Charonko JJ, and Vlachos PP (2015) Particle image pattern mutual information and uncertainty estimation for particle image velocimetry. Measurement Science and Technology 26:074001

    2015

  9. [17]

    arXiv preprint arXiv:250414952

    Zhu Q, Wang J, Hu J, Ai J, and Lee Y (2025) Piv-flowdiffuser: Transfer-learning-based denoising diffusion models for piv. arXiv preprint arXiv:250414952

    2025

  10. [18]

    " * write output.state after.block = add.period write newline

    ENTRY address author booktitle chapter edition editor eid howpublished institution journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.senten...

  11. [19]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.