{"id":"6d34ef2d-d2d6-41a8-b341-f0240e64a46d","arxiv_id":"2507.20102","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"The paper compares an uncertainty neural network, a multi-model ensemble, and multi-transform averaging for uncertainty quantification in deep-learning PIV, finding the uncertainty network best on CAI but poorly generalizing to unseen flow images.","lead":"This paper tests three ways to put error bars on deep-learning-based particle image velocimetry (PIV) flow measurements. It found a trained uncertainty network gives the most reliable error bars on its home dataset, but ensemble and image-rotation methods generalize better to new images.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Abstract's 'consistently best' claim is contradicted by the paper's own Table 4, where UNN coverage drops to 0.05–0.26 on unseen images.","rationale":"The reader's REJECT verdict is well supported, and this stress-test does not change it. I agree that the metric definitions are problematic, but I see an even more direct load-bearing problem: the paper's own Table 4 falsifies the abstract's 'consistently best' statement, regardless of how the metrics are normalized. The reader's weakest_assumption focused on Eq. (2) and Eq. (4) as the foundation of the quantitative comparison; those issues are real and compound the problem, so my agreement is partial rather than complete. The paper does have virtues: it clearly describes three uncertainty strategies, constructs a reasonable benchmark setup, and reports results that honestly show UNN's poor generalization in Section 3.4. However, the abstract and the conclusion frame the contribution around UNN's superiority, which the reported numbers do not support. A rejection is appropriate because the central claim as stated fails on the paper's own evidence, and the metric inconsistencies prevent a reader from salvaging the claim by re-analysis. I recommend no change to the reader's verdict.","tokens_in":7077,"tokens_out":5659,"duration_ms":60007,"concrete_test":"Require the authors to release the raw RAFT velocity predictions and UNN uncertainties, then recompute all three metrics with two corrections: (a) use the 2-sigma coverage threshold |e_i| < 2*sigma_i if the target is truly 95% coverage, and (b) replace Eq. (4) with the standard sparsification AUC, AUC = (1/N) * sum_{i=1}^N (1/(i * e_bar)) * sum_{j=1}^i e_j. Then count, across Tables 2–4, how many cells UNN is strictly best. If UNN is not best in a majority of those cells, or if its Sintel coverage remains below 0.5, the abstract's 'consistently best' claim is false.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim requires UNN to be both best and calibrated across the datasets evaluated. Table 4 directly contradicts this: on Sintel alley 1, UNN coverage is 0.05 versus MM 0.57 and MT 0.49; on bamboo 1, UNN is 0.12 versus MM 0.67 and MT 0.63; on rotating flow, UNN is 0.26 versus MM 0.79 and MT 0.84. Coverage near 0.05 means the predicted uncertainty intervals capture almost none of the true errors, so the uncertainty estimates are not accurate outside the training distribution. The paper's own conclusion even states that MM and MT show better generalization, undercutting the abstract's blanket claim. The mismatch is not rescued by the evaluation metrics, because those are also internally inconsistent: Eq. (2) defines coverage as |e_i| < sigma_i while the text invokes the 2-sigma/95% Gaussian rule, and Eq. (4) omits the 1/i normalization needed for a sparsification AUC, which explains why Table 3 reports AUC values greater than 1, impossible for a standard sparsification curve. Even within CAI under strong perturbation, Table 2 (var = 10) shows UNN CC 0.28 versus MT 0.43, so 'consistently best' is false even before considering unseen data. Together these issues mean the headline claim is unsupported by the paper's own evidence and cannot be independently verified from the formulas provided.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript compares three uncertainty-quantification approaches for deep-learning-based particle image velocimetry (PIV): an Uncertainty Neural Network (UNN) that regresses per-pixel uncertainty from the RAFT velocity estimate and the image pair, a Multiple Models (MM) ensemble of four RAFT instances, and a Multiple Transforms (MT) method that averages predictions from four rotated versions of the input. The authors define three evaluation metrics—95% coverage, Spearman rank correlation, and sparsification area under the curve—and report results on a held-out test split of the CAI dataset, on CAI images corrupted by Gaussian noise and blur, and on unseen Sintel and rotating-flow images. The abstract and Section 2 claim that UNN consistently achieves the best uncertainty estimates, while Section 3.4 and the conclusion acknowledge that MM and MT generalize better on unseen data.","tokens_in":7370,"tokens_out":7621,"duration_ms":77421,"significance":"Uncertainty quantification for deep-learning-based PIV is an important and under-studied problem, and the paper usefully identifies three practical strategies and evaluates them on multiple datasets. The computational-efficiency argument for a single forward-pass UNN is attractive if it holds. However, the significance of the reported comparison is undermined by three problems: the 95% coverage metric is mis-specified, the AUC formula is not properly normalized and produces impossible values, and the headline claim of 'consistently best' is contradicted by the paper's own results in Table 4. The paper's conclusion is more careful than its abstract, which is a point in its favor, but the quantitative basis for the central claim is not reliable as written.","major_comments":[{"comment":"The claim that UNN consistently achieves the best performance is directly contradicted by Table 4: on Sintel alley 1, bamboo 1, and the rotating flow, UNN 95% coverage is 0.05, 0.12, and 0.26, whereas MM achieves 0.57, 0.67, and 0.79 and MT achieves 0.49, 0.63, and 0.84. Coverage near 0.05 means the predicted uncertainty intervals capture essentially none of the actual errors, so UNN's uncertainty estimates are not accurate on the generalization task the paper sets up. The conclusion itself states that MM and MT show better generalization, so the abstract's blanket claim is unsupported by the manuscript's own evidence.","section":"Abstract and §3.4, Table 4"},{"comment":"The 95% coverage definition is inconsistent with the text. Equation (2) counts the fraction of pixels with |e_i| < sigma_i, which for Gaussian errors has an expected value around 68%, not 95%; to obtain 95% the threshold should be c*sigma_i with c near 1.96, or the definition should explicitly use the 2-sigma interval. As written, the reported coverage values cannot be interpreted as 95% coverage, and any comparison based on closeness to 0.95 is not well defined.","section":"§3.1.1, Eq. (2)"},{"comment":"The AUC formula is not a valid normalized sparsification metric. It omits the division by the number of retained points (1/i) and therefore can exceed 1; Table 3 reports AUC values of 1.08 and 1.04, which are impossible for the normalized sparsification curve described in the text. This makes the AUC columns in Tables 1-4 incomparable and undermines the claim that UNN is best on CAI in terms of AUC.","section":"§3.1.3, Eq. (4)"},{"comment":"Even on the in-distribution CAI test set under strong perturbation, UNN is not consistently best: at var = 10 the Spearman correlation is 0.28 for UNN versus 0.43 for MT and 0.37 for MM, and the AUC of UNN (0.80) is no better than MM's (0.80) and worse than MT's (0.76). Thus the 'consistently best' claim is false even before considering unseen data.","section":"§3.3, Table 2"}],"minor_comments":[{"comment":"The text describes MT as 'random sampling and statistical analysis based on the Monte Carlo method,' but the method uses four fixed rotations (0, 90, 180, 270 degrees); this is not random sampling and should be described as deterministic multi-transform ensembling.","section":"§2.3"},{"comment":"The manuscript does not specify how the four RAFT models differ (for example, initialization seeds or training subsets), nor does it state whether the 'uncertainty' is the standard deviation or variance of the four outputs; this should be clarified for reproducibility.","section":"§2.2"},{"comment":"The figure layout and caption do not clearly identify which subplot corresponds to ground truth, prediction, or uncertainty for each of the three samples; the caption should enumerate the panels explicitly.","section":"§3.2, Figure 4"},{"comment":"No code or data availability statement is provided; given the issues with the evaluation metrics, an anonymous code release would be necessary for independent verification.","section":"General"},{"comment":"Several references have inconsistent formatting (for example, author initials are sometimes expanded and sometimes not), and the paper should use a consistent citation style throughout.","section":"References"},{"comment":"The generalization experiments use only three test cases (two Sintel scenes and one rotating-flow set), with no error bars or repeated trials; the conclusions about generalization would be more convincing with more scenes and statistical confidence measures.","section":"§3.4"}],"recommendation":"reject","confidential_remarks":"The manuscript was accepted for presentation at ISPIV 2025, and the version under review is a short conference paper. The metric errors are substantial, and the headline claim is contradicted by the paper's own Table 4. If the authors were to recompute the metrics and reframe the contribution as an in-domain comparison with a documented out-of-domain failure, the study could become a useful empirical note; as submitted, however, the central quantitative claims are not reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the comparison is worth having, but the headline claim is not supported by the paper's own numbers, and the evaluation metrics contain internal errors. Do not take the abstract at face value.\n\nWhat the paper actually does: it takes three off-the-shelf UQ ideas—deep ensembles (MM), test-time rotation averaging (MT), and learned heteroscedastic variance (UNN)—and runs them on the same RAFT-PIV model across CAI, two Sintel scenes, and a rotating flow. None of these methods is new, and the authors cite the relevant prior work (BCNN, ProbFlow). The contribution is the side-by-side comparison for PIV, which to my knowledge hasn't been done in exactly this form. That is useful for practitioners choosing between cheap single-pass UQ and expensive sampling-based UQ.\n\nWhat is actually good: the experimental structure is simple and reproducible in principle; the perturbation studies are a nice touch; and the conclusion is more honest than the abstract, noting that MM/MT generalize better. The figures tell a coherent story about sparsification on the CAI samples.\n\nNow the soft spots. First and most serious: the abstract says UNN 'consistently achieves the best performance.' Table 4 shows the opposite on every unseen dataset: coverage 0.05 on Sintel alley 1, 0.12 on bamboo 1, 0.26 on rotating flow, versus 0.49–0.84 for MM and MT. Even on the CAI noise test at var=10, UNN's correlation coefficient is 0.28 against MT's 0.43. So the central claim fails on the paper's own evidence. Second, Eq. (2) defines 95% coverage with the condition |e_i| < sigma_i. For a Gaussian error, that is a one-sigma interval, around 68% coverage. The 95% target described in the text requires roughly two sigma. As written, the coverage values are anchored to the wrong reference. Third, Eq. (4) is not a standard sparsification AUC; it lacks the proper normalization and can exceed 1, which is exactly what Table 3 reports (AUC 1.08, 1.04). So the numbers cannot be compared across methods or to the literature. Fourth, there are no error bars and no code; the unseen-image experiments are limited to three flow configurations.\n\nThe bottom line: the useful part is the benchmark structure and the acknowledgment of the generalization problem. The unsupported part is the 'consistently best' claim and the quantitative comparisons, which are undermined by the metric errors. I would not reject the topic; I would reject this version of the paper. A corrected revision with fixed equations, a more precise abstract, and code could be a reasonable contribution.\n\nWho should read it: PIV people who want a map of what existing UQ tricks look like on deep learning PIV. They should ignore the tables until the metrics are fixed.\n\nRecommendation: send to a serious referee. The flaws are specific, fixable, and the experiment has value. Not desk-reject, but not accept either.","headline":"The 'consistent best' claim for UNN is refuted by the paper's own Table 4, and the metric definitions are internally inconsistent; the benchmark framing has value but the paper cannot be accepted as is.","tokens_in":7908,"tokens_out":4150,"would_cite":false,"duration_ms":41290,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that a learned uncertainty neural network can put per-pixel error bars on deep-learning-based PIV velocities, scoring above ensemble and transform sampling on in-distribution data while sampling methods retain an edge on…","keywords":["particle image velocimetry","uncertainty quantification","deep learning","RAFT optical flow","U-net","Gaussian negative log-likelihood","sparsification plot","coverage rate"],"falsifier":"On the CAI test set, form the normalized error $e_i/\\sigma_i$ for each method and compute the empirical fraction of points with $|e_i| < 1.96\\sigma_i$; if UNN's fraction is no closer to 95% than MM's or MT's, the claim that UNN provides accurate 95% coverage fails. Independently, recompute the AUC by direct numerical integration of the sparsification curves shown in Figure 4 and compare the resulting rankings with Table 1; a changed ranking would show the headline comparison depends on the specific AUC formula in Eq. (4).","tokens_in":6825,"feed_emoji":"🌊","tokens_out":10405,"duration_ms":96861,"temperature":0.7,"pith_summary":"Particle image velocimetry (PIV) extracts flow velocities from pairs of particle images, and deep-learning versions of it are fast but give no error bars. This paper tries to fix that by comparing three ways of attaching uncertainty to the velocities predicted by a RAFT network: an uncertainty neural network (UNN) that regresses per-pixel uncertainty from the flow field and images, an ensemble of four models (MM), and a set of four rotated transforms of the input (MT). On the in-distribution CAI benchmark the paper reports that UNN gives the most accurate uncertainty estimates by all three metrics it uses—95% coverage, Spearman's rank correlation, and sparsification AUC—while MM and MT generalize better on unseen Sintel and rotating-flow images. A sympathetic reading of the central claim is that one-pass learned uncertainty can replace expensive sampling for in-distribution PIV, so long as the evaluation metrics are correctly calibrated.","feed_headline":"One neural net predicts PIV uncertainty better than four models","feed_subtitle":"A U-net with a Gaussian loss yields calibrated PIV error bars; sampling methods still win on unseen flows.","key_machinery":"The load-bearing object is the UNN itself: a U-net regression head whose input is the concatenation of the RAFT-predicted velocity and the original particle image pair, and whose output is a per-pixel uncertainty $\\sigma$. Training minimizes the Gaussian negative log-likelihood, so the network is rewarded for predicting large $\\sigma$ exactly where the squared velocity error $e^2$ is large; this is the mechanism that lets a single forward pass produce error-sensitive uncertainty maps. The two comparators unpack the same idea statistically: MM averages four independently trained RAFT instances and takes their spread as uncertainty, MT feeds four rotations (0°, 90°, 180°, 270°) through one RAFT and takes the spread of the rotated-back fields as uncertainty. The evaluation machinery consists of three metrics, 95% coverage, Spearman's rank correlation, and sparsification AUC, whose exact forms matter for the ranking.","core_discovery":"The central discovery, stated on the paper's own terms, is that an end-to-end uncertainty neural network—a U-net that consumes the concatenation of a RAFT-predicted velocity field and the particle image pair and is trained with the Gaussian negative log-likelihood loss $L = -\\frac{1}{N}\\sum_{i=1}^{N}\\left[\\log\\sigma_i + \\frac{e_i^2}{2\\sigma_i^2}\\right]$—yields per-pixel velocity uncertainties that rank first on the CAI test set under 95% coverage, Spearman's rank correlation, and sparsification AUC, ahead of multiple-models and multiple-transforms sampling. The authors take this as evidence that learned single-pass uncertainty is a viable, computationally cheaper alternative to ensemble sampling for deep-learning PIV, while acknowledging that MM and MT generalize better on unseen Sintel and rotating-flow data and that all three methods degrade under strong Gaussian blur or noise.","pith_inferences":["The reported in-distribution coverage values (0.83–0.90) sit below the nominal 95% target while the metric is defined as $|e| < \\sigma$, which suggests recalibrating the threshold to $1.96\\sigma$—or rescaling $\\sigma$ post hoc—could move UNN closer to the target and might alter the method ranking; this is a direct, testable check of the headline claim.","Because the UNN loss assumes per-pixel Gaussian errors, swapping the negative log-likelihood for a heavy-tailed or mixture-density loss should improve coverage at flow boundaries and occlusions, the regions where PIV errors are largest and least Gaussian.","A deep ensemble of several UNN heads, trained with different initializations, would combine UNN's calibration with MM's generalization advantage; the paper neither tests nor rules out this hybrid.","Recomputing the AUC by trapezoidal integration of the sparsification curves in Figure 4, rather than applying Eq. (4), would verify whether UNN's reported advantage on that metric is robust to the formula's exact form."],"forward_implications":["A single UNN forward pass can attach per-pixel uncertainty to RAFT velocity fields on in-distribution PIV data, replacing four model evaluations or four image rotations and making uncertainty-aware flow measurement practical for real-time use.","The three-metric evaluation protocol—95% coverage, Spearman's CC, sparsification AUC against an Oracle curve—can serve as a standard benchmark for future PIV uncertainty methods.","Under mild Gaussian noise or blur the uncertainty estimates remain usable, but under strong degradation none of the three methods should be trusted, so error bars should carry a confidence flag tied to input quality.","For out-of-distribution flows, the paper's results imply that sampling-based uncertainty (MM or MT) is the safer default, while the single-pass UNN is the better default only when the test flow resembles the training distribution."],"supporting_citations":[{"why":"Supplies the CAI dataset, the in-distribution benchmark whose test split carries the main comparison.","marker":"(Cai et al., 2019)"},{"why":"Provides the RAFT-PIV network used as the fixed velocity estimator that all three uncertainty methods share.","marker":"(Lagemann et al., 2021)"},{"why":"Supplies the Spearman correlation CC and the sparsification-plot AUC metrics, and the Oracle sparsification curve used as the ideal benchmark.","marker":"(Wannenwetsch et al., 2017)"},{"why":"Supplies the 95% coverage criterion used to judge whether predicted uncertainty bounds contain the true error.","marker":"(Timmins et al., 2012)"},{"why":"Motivates the UNN approach as the Bayesian-convolutional-network precedent that predicts uncertainty from network outputs, though at high computational cost.","marker":"(Morrell et al., 2021)"}],"fun_headline_variants":["UNN leads PIV uncertainty metrics, but sampling wins on unseen flows","Single-pass U-net beats ensembles for PIV error bars on CAI","PIV uncertainty: one neural net yields top confidence calibration","Deep-learning PIV gets calibrated uncertainty via UNN, not ensembles"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison rests on the coverage formula in Eq. (2) counting an error as covered when the absolute error is below the predicted uncertainty and treating that rate as the 95% two-sigma target; if the network's predicted uncertainty is a one-standard-deviation quantity, the expected rate is about 68%, not 95%, and the ranking could be miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["UNN leads PIV uncertainty metrics, but sampling wins on unseen flows","Single-pass U-net beats ensembles for PIV error bars on CAI","PIV uncertainty: one neural net yields top confidence calibration","Deep-learning PIV gets calibrated uncertainty via UNN, not ensembles"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000514,"raw_usage":{"total_tokens":2470,"prompt_tokens":891,"completion_tokens":1579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1504}},"tokens_in":507,"tokens_out":1579,"duration_ms":13158,"temperature":1.0,"reasoning_tokens":1504,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:49:21.814457+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On the CAI test set, form the normalized error $e_i/\\sigma_i$ for each method and compute the empirical fraction of points with $|e_i| < 1.96\\sigma_i$; if UNN's fraction is no closer to 95% than MM's or MT's, the claim that UNN provides accurate 95% coverage fails. Independently, recompute the AUC by direct numerical integration of the sparsification curves shown in Figure 4 and compare the resulting rankings with Table 1; a changed ranking would show the headline comparison depends on the specific AUC formula in Eq. (4).","supporting_citations":[{"cited_title":"Nature Machine Intelligence 3:641--651","cited_arxiv_id":null,"evidence_quote":"Provides the RAFT-PIV network used as the fixed velocity estimator that all three uncertainty methods share."},{"cited_title":"in Proceedings of the IEEE international conference on computer vision","cited_arxiv_id":null,"evidence_quote":"Supplies the Spearman correlation CC and the sparsification-plot AUC metrics, and the Oracle sparsification curve used as the ideal benchmark."},{"cited_title":"Experiments in fluids 53:1133--1147","cited_arxiv_id":null,"evidence_quote":"Supplies the 95% coverage criterion used to judge whether predicted uncertainty bounds contain the true error."},{"cited_title":"Measurement Science and Technology 32:104003","cited_arxiv_id":null,"evidence_quote":"Motivates the UNN approach as the Bayesian-convolutional-network precedent that predicts uncertainty from network outputs, though at high computational cost."}],"review_version":2}