{"id":"464ce65b-33a3-4322-8a47-7aee5eb08fdf","arxiv_id":"2505.09528","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Conformal prediction plus approximate posterior sampling yields guaranteed bounds on full-reference image quality metrics for imaging inverse problems.","lead":"This paper constructs guaranteed bounds on image quality scores like PSNR, SSIM, and LPIPS for reconstructed images, even when the true image is unknown. The method combines conformal prediction with posterior sampling and is demonstrated on image denoising and accelerated MRI.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified: the central conformal coverage claim is standard, correctly applied, and its key assumption is explicitly disclosed and empirically stress-tested.","rationale":"The reader's verdict is ACCEPT with high confidence, and this stress-test pass found no reason to change it. The central claim is a faithful application of split conformal prediction to a new target quantity, the FRIQ of a recovered image. The theoretical mechanism is standard: calibrated intervals of the form [hat z - lambda, inf) or (-inf, hat z + lambda] have monotone empirical miscoverage, and the calibration rule from Angelopoulos et al. (2022a) guarantees marginal coverage when the calibration and test pairs are exchangeable. The paper's adaptive bounds are also valid: the approximate posterior sampler merely defines the estimator f, and conformal calibration corrects any systematic bias at the marginal level. The weakest assumption is exchangeability, and the paper both states it and tests it under a realistic slice-location shift. The results in Section 4.3 show coverage degradation for some metrics, consistent with the assumption being load-bearing, but the paper does not overclaim robustness and discusses distributionally robust extensions as future work. The multi-round acceptance coverage being slightly below 1-alpha is a real practical caveat, but it is explicitly reported and is not part of the formal central claim. Overall, the paper is methodologically sound, honest about limitations, and reproducible via provided code and standard conformal tools. No adjustment to the reader's ACCEPT verdict is warranted.","tokens_in":34812,"tokens_out":14619,"duration_ms":166171,"concrete_test":"Run a synthetic split-conformal sanity check: draw n=100 i.i.d. calibration scores, compute the threshold via (3) with alpha=0.05, and verify over 10^5 test draws that the empirical coverage equals 0.9505, i.e., k/(n+1) with k=5. This directly confirms that the finite-sample correction in (3) is implemented correctly and that Equation (6) holds under exchangeability.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No load-bearing concern was identified against the central claim. Equation (6) is a standard split-conformal guarantee: given exchangeable FRIQ pairs, the calibration rule (3) with the monotone interval family (4)-(5) yields marginal coverage at level 1-alpha. The paper correctly keeps calibration data disjoint from training of the recovery h and of the regression predictor f, and the quantile and regression constructions are valid regardless of the accuracy of the approximate posterior sampler, since any fixed estimator f only affects tightness, not validity. The exchangeability assumption is the genuinely load-bearing condition, and the paper does not hide it: it is stated in Section 3, probed under slice-location shift in Section 4.3, and the resulting coverage degradation is reported. The limitations section additionally discloses the marginal nature of the guarantee and the need for calibration data similar to the test data. The only deviation from the advertised 1-alpha observed in the paper is the acceptance coverage in the multi-round protocol (e.g., 0.9323 for DISTS in Table 2), which is a conditional-coverage phenomenon caused by stopping on the bound; however, the authors explicitly report this quantity and do not claim a conditional guarantee for it, so it does not undermine Equation (6).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a conformal prediction framework for constructing one-sided bounds on full-reference image quality (FRIQ) metrics such as PSNR, SSIM, LPIPS, and DISTS in imaging inverse problems, without access to the true image. The authors introduce three variants: a non-adaptive bound that ignores the measurements, an adaptive quantile bound that summarizes approximate posterior samples of the image via an empirical quantile of posterior FRIQ values, and a regression bound trained with quantile regression on features derived from posterior samples. They prove that, under exchangeability of the FRIQ pairs, the calibration rule of Angelopoulos et al. yields marginal coverage of at least 1-α. Experiments on FFHQ denoising and accelerated multicoil fastMRI knee reconstruction, each with T=10,000 Monte Carlo trials, show empirical coverage very close to the nominal level, tighter bounds for the adaptive methods than the non-adaptive baseline, and a multi-round measurement protocol that attains higher average acceleration with adaptive bounds. The paper also includes an explicit distribution-shift study in which calibration on center knee slices is tested on increasingly peripheral slices, reporting coverage degradation for some metrics, and a limitations section that discloses the marginal nature of the guarantee.","tokens_in":35040,"tokens_out":6702,"duration_ms":68143,"significance":"If accepted, the paper provides a practically useful and theoretically sound method for communicating uncertainty about image quality in safety-critical imaging settings. The central coverage guarantee (Eq. 6) is a standard split-conformal result and is correctly applied: the interval family is monotone in lambda, calibration data are kept disjoint from training data, and the validity of the guarantee does not depend on the accuracy of the posterior sampler or the quantile-regression predictor. The empirical validation is unusually thorough, with 10,000 Monte Carlo trials, coverage checks across several metrics and values of c, and a deliberate stress test of the exchangeability assumption that reports honest degradation. The authors also openly state the limitation that the guarantee is marginal rather than conditional on a given test image or calibration set. These strengths, together with the released code, make the paper a solid contribution to uncertainty quantification for imaging inverse problems.","major_comments":[],"minor_comments":[{"comment":"In the paragraph after Table 12, the sentence \"This is can explained by the perception-distortion tradeoff\" should read \"This can be explained by the perception-distortion tradeoff.\"","section":"Appendix D"},{"comment":"The acceptance empirical coverage of 0.9323 reported for the quantile method is noticeably below the nominal 0.95; since this quantity is conditional on the stopping event and no conditional guarantee is claimed, the statement that it is \"very close to 1-α\" could be softened or qualified to avoid overstating the result.","section":"Table 2 and Section 4.2"},{"comment":"The text refers to a \"Flickr-Faces-HQ (FFHQ) validation dataset,\" but FFHQ does not have an official validation split; please clarify how this subset was defined.","section":"Section 4.1"}],"recommendation":"accept","confidential_remarks":"The paper is a technically sound application of conformal prediction to a new and relevant problem. The authors rely on their own prior tools (the CNF from Wen et al. 2023a and the task-driven UQ framework from Wen et al. 2024), but the FRIQ-bounding formulation is distinct, and the central coverage guarantee is independent of those tools. The empirical evaluation, including the deliberate distribution-shift analysis, is a model of transparency. I see no load-bearing technical issue. The only concerns are minor presentational, and I support acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe paper does what it says: for any recovery method and any full-reference image quality metric, it builds a conformal bound on the unknown FRIQ of a test reconstruction, with marginal coverage 1-α under exchangeability. The novelty is real but narrow. As far as I know, no prior work targets FRIQ metrics themselves; earlier work bounds pixel values or task outputs. The construction is a straightforward split-conformal wrapper around a posterior-sampling-based quantile estimate (or a quantile-regression estimate), with the one-sided interval family in (4)-(5). That is not a criticism: the paper correctly identifies the gap and fills it without overclaiming.\n\nThe theoretical core is standard and correctly applied. Calibration via (3) follows Angelopoulos et al., the interval family is monotone in λ, and (6) is a valid exchangeability-based marginal guarantee. The coverage guarantee is independent of the posterior sampler's accuracy; conformal calibration absorbs the bias. The experiments are extensive—T=10,000 Monte Carlo trials, four metrics, two tasks, multiple accelerations—and the empirical coverage is essentially exactly 0.95 in all in-distribution settings. The multi-round MRI protocol is a nice demonstration of practical value, and the paper honestly reports that the quantile bound beats the non-adaptive bound by roughly a factor of two in average acceleration.\n\nSoft spots, in proportion. The 'regression' bound from Sec. 3.4 adds complexity but delivers little gain over the simple empirical-quantile bound; the paper says so plainly, which is refreshing. The exchangeability assumption is load-bearing, and the distribution-shift study shows that coverage can drop below 1-α for some metrics when test slices are far from calibration slices. That is not hidden—it is in Sec. 4.3 and in the limitations—but it does temper any safety-critical claim. The acceptance coverage in the multi-round protocol is slightly below 0.95 (e.g., 0.9323 for DISTS), which is a conditional-coverage artifact; the authors report it without claiming a conditional guarantee, so I read that as honest.\n\nThe citation pattern is fine. The self-citations to their CNF and task-driven UQ are used as tools, and the conformal layer is independent of those tools. No circularity.\n\nWho is this for? Anyone doing uncertainty quantification for imaging inverse problems, especially medical imaging. A serious referee should engage; the paper likely needs only minor revisions, mostly tightening claims around shift robustness and clarifying that the regression bound's marginal benefit is task-dependent. I would take it.\n\nRecommendation: send to peer review.","headline":"A clean, honest application of split conformal prediction to a new target—full-reference image quality—with solid experiments; the novelty is modest but the execution is careful and the claims are appropriately scoped.","tokens_in":35579,"tokens_out":2248,"would_cite":true,"duration_ms":22181,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper constructs conformal bounds on full-reference image quality that hold with probability at least 1-alpha when the true image is unknown.","keywords":["conformal prediction","full-reference image quality","image recovery","inverse problems","uncertainty quantification","posterior sampling","accelerated MRI","image quality assessment"],"falsifier":"Calibrate the quantile bound on center knee slices at acceleration $R=8$ and evaluate it on edge slices from the same volumes; the paper's own experiment shows coverage for PSNR and LPIPS dropping below $1-\\alpha$ as the slice location increases. Any deployment in which a shifted test set produces empirical coverage clearly below the promised $1-\\alpha$ would falsify the practical claim that the bound holds in that setting.","tokens_in":34592,"feed_emoji":"🖼️","tokens_out":7548,"duration_ms":68566,"temperature":0.7,"pith_summary":"The paper asks how a user can know the quality of a recovered image when the true image is unavailable. It shows that split conformal prediction can convert a calibration set of known image/measurement pairs into a one-sided bound on any full-reference metric, with a user-chosen error probability. The bound adapts to the test measurements and reconstruction through an estimate of the metric built from approximate posterior samples. If the paper is right, users can get distribution-free guarantees on PSNR, SSIM, LPIPS, or DISTS without ever seeing the true image, and can decide when to trust a reconstruction or collect more measurements.","feed_headline":"Conformal bounds put a floor under image quality without the true image","feed_subtitle":"Posterior samples plus a calibration set guarantee coverage for PSNR, SSIM, LPIPS, or DISTS at a chosen error rate.","key_machinery":"The load-bearing object is a split-conformal one-sided prediction interval. For higher-preferred metrics like PSNR and SSIM, the interval is $C_\\lambda(\\hat{z}) = [\\hat{z} - \\lambda, \\infty)$, and for lower-preferred metrics like LPIPS and DISTS it is $(-\\infty, \\hat{z} + \\lambda]$. Calibration chooses $\\lambda$ as the smallest value for which the empirical miscoverage on held-out pairs stays at or below $\\alpha - (1-\\alpha)/n$. The estimate $\\hat{z}$ is produced by an adaptive estimator: the empirical $\\alpha$-quantile (or $(1-\\alpha)$-quantile) of FRIQ values $m(\\hat{x}, \\tilde{x})$ computed from approximate posterior samples of the true image, optionally refined by quantile regression. These posterior-derived estimates make the bound respond to the test measurement and reconstruction.","core_discovery":"The paper's central claim is that, for any recovery map $h$ and any full-reference metric $m$, the unknown quality $z_0 = m(\\hat{x}_0, x_0)$ can be bounded with guaranteed probability without observing $x_0$. Under exchangeability of the pairs $(z_i, \\hat{z}_i)$ across calibration and test, calibrating $\\lambda$ from the empirical miscoverage of one-sided intervals yields $\\Pr\\{Z_0 \\in C_{\\hat{\\lambda}(D_{\\mathrm{cal}})}(\\hat{Z}_0)\\} \\ge 1 - \\alpha$. The guarantee is marginal, over both calibration and test randomness. The paper demonstrates the construction on denoising and accelerated MRI, including adaptive bounds built from approximate posterior samples that track the true metric far better than constant bounds.","pith_inferences":["Whenever calibration data can be stratified by acquisition conditions such as slice location, scanner, or field strength, calibrating per stratum or using weighted conformal methods could restore coverage under covariate shift.","Because the bound adapts to posterior samples, it could serve as a hallucination flag: a tight low bound on SSIM or DISTS means the reconstruction is formally far from any plausible true image under the posterior.","Treating the posterior sampler as a black box means any improvement in posterior fidelity, including samplers that target epistemic uncertainty, should directly tighten the quantile and regression bounds.","Risk-controlling prediction sets could upgrade the marginal guarantee to a calibration-set-conditional guarantee with two error rates, which would make the bound more trustworthy for individual deployment."],"forward_implications":["A clinician could check whether an accelerated MRI reconstruction meets a preset DISTS or PSNR threshold before relying on it; the multi-round experiments stop measurement collection at an average acceleration near 4 instead of 2.","The same calibration machinery works for any recovery network and any full-reference metric, so bounds can be instantiated for a metric matched to a given application.","Because the coverage guarantee is marginal, the bounds are suited to population-level decisions rather than per-image certainty.","Adaptive bounds track the true metric much more closely than constant bounds, with Pearson correlation above 0.5 and up to about 0.7 in the denoising experiments.","The learned quantile-regression variant adds little over the empirical-quantile variant in the tested settings, suggesting the simple quantile bound is a strong default."],"supporting_citations":[{"why":"Supplies the split-conformal calibration rule based on empirical miscoverage that yields the coverage guarantee.","marker":"Angelopoulos et al. 2022a"},{"why":"Introduces conformal prediction, the framework the paper's bounds rely on.","marker":"Vovk et al. 2005"},{"why":"Provides the split conformal prediction formulation and the marginal coverage interpretation used throughout.","marker":"Angelopoulos & Bates 2023"},{"why":"Establishes the distribution-free coverage guarantee that the calibration step inherits.","marker":"Lei & Wasserman 2014"},{"why":"Inspires the learned adaptive bound via conformalized quantile regression.","marker":"Romano et al. 2019"},{"why":"Provides the DDRM posterior sampler used to generate FRIQ estimates in the denoising experiments.","marker":"Kawar et al. 2022a"},{"why":"Provides the conditional normalizing flow used as the posterior sampler in the MRI experiments.","marker":"Wen et al. 2023a"},{"why":"Provides the E2E-VarNet recovery network used as the reconstruction map in the MRI experiments.","marker":"Sriram et al. 2020a"},{"why":"Supplies evidence that DISTS matches radiologist ratings, motivating the choice of FRIQ metric in medical applications.","marker":"Kastryulin et al. 2023"}],"fun_headline_variants":["Guaranteed image-quality bounds without the true image","Conformal bounds certify PSNR, SSIM, LPIPS without true image","Image quality bounds guaranteed without seeing the original","Know your image quality bounds without the ground truth","Conformal prediction bounds PSNR, SSIM, LPIPS without the true image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the test image-quality pair and the calibration image-quality pairs behave like draws from the same joint distribution; if the test images come from a different distribution than the calibration set, the coverage guarantee can fail.","fun_headline_variants_meta":{"raw":{"variants":["Guaranteed image-quality bounds without the true image","Conformal bounds certify PSNR, SSIM, LPIPS without true image","Image quality bounds guaranteed without seeing the original","Know your image quality bounds without the ground truth","Conformal prediction bounds PSNR, SSIM, LPIPS without the true image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000425,"raw_usage":{"total_tokens":2136,"prompt_tokens":857,"completion_tokens":1279,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":473,"completion_tokens_details":{"reasoning_tokens":1193}},"tokens_in":473,"tokens_out":1279,"duration_ms":9160,"temperature":1.0,"reasoning_tokens":1193,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:28:23.525250+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Calibrate the quantile bound on center knee slices at acceleration $R=8$ and evaluate it on edge slices from the same volumes; the paper's own experiment shows coverage for PSNR and LPIPS dropping below $1-\\alpha$ as the slice location increases. Any deployment in which a shifted test set produces empirical coverage clearly below the promised $1-\\alpha$ would falsify the practical claim that the bound holds in that setting.","supporting_citations":[{"cited_title":"Algorithmic Learning in a Random World","cited_arxiv_id":null,"evidence_quote":"Introduces conformal prediction, the framework the paper's bounds rely on."},{"cited_title":"Image quality assessment for magnetic resonance imaging","cited_arxiv_id":null,"evidence_quote":"Supplies evidence that DISTS matches radiologist ratings, motivating the choice of FRIQ metric in medical applications."}],"review_version":1}