{"id":"8ae3ca8f-05a0-4c02-912d-ea74c329b893","arxiv_id":"1908.07709","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Point-wise GP registration uncertainty shows only a weak-to-moderate positive monotonic correlation with non-rigid registration error, and patch-wise correlation is low.","lead":"This pilot study tests whether uncertainty estimates from Gaussian process image registration can serve as a surrogate for registration error in brain surgery guidance. On two clinical ultrasound datasets, it finds a weak-to-moderate point-wise correlation and low patch-wise correlation, suggesting uncertainty alone is not yet reliable enough for clinical use.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported correlation strengths hinge on the GP kernel length-scale a in Eq. 3, which is never specified; without a sensitivity analysis the central empirical claim is not reproducible or robust.","rationale":"The paper is a cautious pilot study with modest claims, and the point-wise experimental design is reasonable: it uses held-out landmarks, Spearman's rank correlation instead of Pearson, and both a public dataset (RESECT) and a proprietary one (MIBS). The authors also explicitly acknowledge that the patch-wise results are low and that it is too early to draw definitive conclusions. The single most load-bearing weakness is the unspecified kernel length-scale a. Because the GP posterior variance in Eq. 3 is independent of the observed displacement values and depends only on the kernel and the training landmark positions, the uncertainty ranking is entirely determined by the chosen a. The paper never reports a value for a, nor any sensitivity analysis. This means the reported mean rho values of 0.29 and 0.40 are not reproducible from the text, and the quantitative conclusion is conditional on an unstated hyperparameter choice. A reader cannot tell whether a was selected deliberately to produce favorable correlations. This concern does not invalidate the paper; rather, it strengthens the case for a conditional verdict: the claim should be accepted only once the authors provide the length-scale value and show that the correlation is stable across a reasonable range of a. The reader's weakest_assumption identified the same issue, so I agree with that assessment. No other concern is more load-bearing: the lack of confidence intervals is real but secondary, and the patch-wise metric problems are already acknowledged by the authors. If the sensitivity test is run and the correlation is stable, the central claim would be supported by the data as reported.","tokens_in":7428,"tokens_out":4995,"duration_ms":53668,"concrete_test":"Obtain the exact value of a (or the code) from the authors and re-run the point-wise experiment on RESECT with k(x,x') = exp(-||x-x'||^2/a) for a = 1, 2, 5, 10, 20, and 50 voxels, using the same landmarks and error metric. Plot the mean and per-image Spearman rho against a. If the mean rho remains within the same weak-to-moderate band (roughly 0.2-0.4) across the entire range, the concern is minor; if it drops below 0.1 or changes sign for any plausible a, the central claim must be explicitly conditioned on the chosen length-scale.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1 defines the GP covariance as k(x,x') = exp(-x^2/a) and states that the same kernel is used for all displacement components, but no value for a is given anywhere in the paper, including the experiments. The posterior variance in Eq. 3 depends on a and on the training landmark positions, but not on the observed displacements, so the rank-ordering of uncertainty values, and hence the Spearman rho values in Fig. 5, is a function of a. As a tends to infinity, the posterior variance becomes nearly constant and rank correlation degenerates; as a tends to zero, uncertainty is near 1 except exactly at training landmarks. Intermediate values can produce weak, moderate, or even negative correlation depending on landmark geometry and the error distribution. Since a is unstated, the headline numbers (mean rho = 0.2899 for RESECT manual landmarks, 0.4014 for MIBS automatic landmarks) cannot be reproduced, and one cannot rule out that a was set to a favorable value post hoc. This is an addressable calibration gap rather than an internal inconsistency, but it makes the quantitative claim conditional on an unexamined hyperparameter.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper asks whether Gaussian-process (GP) registration uncertainty is monotonically associated with non-rigid registration error in image-guided neurosurgery. Using two clinical ultrasound datasets (RESECT and MIBS), the authors compute Spearman rank correlations between point-wise GP posterior uncertainties and landmark displacement errors, as well as patch-wise correlations using intensity dissimilarity measures (HI and SSD). They report weak-to-moderate positive point-wise correlations (mean rho = 0.2899 for manual landmarks, 0.4014 for automatic landmarks) and consistently low patch-wise correlations, concluding that GP uncertainty is not yet a reliable surrogate for registration error.","tokens_in":7618,"tokens_out":4850,"duration_ms":48269,"significance":"If the empirical result holds, it provides a useful cautionary quantification for clinicians considering GP uncertainty maps as error surrogates, and it opens a question that is indeed overlooked in the registration literature. The study has clear strengths: the point-wise test is direct, uses ground-truth displacements, and involves no circularity because the GP posterior variance is computed from landmark positions and kernel choices independent of the error measurement. The use of two clinical datasets is also a plus. However, the quantitative conclusions are conditional on an unstated kernel hyperparameter and lack any statistical uncertainty assessment, so the current weight of the evidence is limited.","major_comments":[{"comment":"The covariance kernel is written as k(x, x') = exp(-x^2/a), but no value or range for a is given anywhere in the manuscript. The posterior variance in Eq. (3) depends on a and the landmark positions, but not on the observed displacements D; consequently the rank ordering of uncertainty values and every Spearman coefficient in Fig. 5 is a function of this unstated parameter. The authors must report the exact value of a used and provide a sensitivity analysis over a range of a, since the headline means (0.2899 and 0.4014) cannot otherwise be reproduced and the influence of this hyperparameter on the conclusions cannot be assessed.","section":"Section 2.1, Eq. (3)"},{"comment":"The central claim of weak-to-moderate positive monotonic correlation rests only on mean rho values across image pairs. The authors do not report the number of image pairs and landmarks, the per-pair correlation values, confidence intervals, or significance tests, so the reader cannot judge whether the correlations are statistically distinguishable from zero or how stable they are across patients. Please provide this information, for instance with bootstrap confidence intervals for the mean rho and per-case results.","section":"Section 3.1, Fig. 5"},{"comment":"The patch-wise experiment is interpreted as showing consistently low correlation, but the authors themselves state that the intensity-based HI/SSD dissimilarity is 'sub-optimal' for estimating registration error under large deformation. Because the error surrogate is acknowledged to be flawed, the low patch-wise rho values cannot be cleanly attributed to the uncertainty-error relationship. The patch-wise conclusion should either be supported with a validated local error measure or explicitly downweighted and removed from the paper's conclusions.","section":"Section 3.2"},{"comment":"The experimental setup is not fully specified: the manuscript does not state how many interpolating landmarks were used per image, how they were selected, how many test points were evaluated, or how the automatic landmark detection was performed in MIBS. These details materially affect both the GP uncertainty and the error distribution and are needed for reproducibility of the reported correlations.","section":"Section 3"}],"minor_comments":[{"comment":"The kernel notation k(x, x') = exp(-x^2/a) is ambiguous because k takes two spatial arguments; the exponent should be written as e.g. exp(-||x - x'||^2 / a).","section":"Section 2.1"},{"comment":"The rank-transformation example is correct, but the authors should state how tied ranks are handled when computing Spearman's rho.","section":"Section 2.2"},{"comment":"The sentence 'while intensities over patches is label uncertainty' is unclear; patch intensities are observations used to define the dissimilarity, not label uncertainty, and the connection to the transformation-versus-label uncertainty distinction should be explained more carefully.","section":"Section 3.2"},{"comment":"The text says the automatic landmarks 'achieved an average rho of 0.4014, which can be categorized as a moderate-to-strong correlation,' but the correlation categories are not defined; the reader would benefit from an explicit threshold or reference for the labels 'weak-to-moderate' and 'moderate-to-strong.'","section":"Section 3.1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a short pilot study addressing a clear question. The missing kernel length-scale and lack of statistical inference are addressable within the paper's scope, so I see no need for rejection. If the authors provide the kernel setting, a sensitivity analysis, and confidence intervals, the paper would be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a pilot study with a genuinely new question, and a guarded answer, but the missing GP kernel length-scale in Eq. 3 makes the headline numbers impossible to reproduce. That is the one thing I would want fixed before trusting the quantitative claim.\n\nWhat is actually new: nobody has systematically asked whether registration uncertainty from GP registration tracks error pointwise in clinical data. The authors use Spearman's rho, which is appropriate since clinical usefulness only needs monotonicity, and they get mean rho around 0.29 for manual landmarks and 0.40 for automatic ones on two datasets, one public (RESECT) and one proprietary (MIBS). They are refreshingly cautious in the conclusion, explicitly saying it is too early to conclude it is safe to use uncertainty as a surrogate.\n\nThe paper is honest about its own soft spots. In Section 3.2 they state the intensity-based patch-wise error metric is sub-optimal, and they flag the transformation-vs-label uncertainty distinction. That is good scholarly behavior.\n\nWhere it gets soft: the length-scale a in k(x,x') = exp(-x^2/a) is never specified anywhere, including the experiments. This is not a nitpick. The posterior variance in Eq. 3 depends directly on a. As a grows large, the covariance becomes nearly constant, so uncertainty is flat and rank correlation degenerates; as a shrinks, uncertainty is concentrated around the interpolating landmarks. The rank ordering, and therefore every rho value in Fig. 5, depends on a. Without a value or a sensitivity analysis, the reported numbers cannot be reproduced, and one cannot rule out that the chosen a was cherry-picked. The stress-test note about this is on point.\n\nAlso missing: confidence intervals or significance tests. The point-wise correlations are computed over many landmarks, but we never learn whether rho = 0.29 is statistically distinguishable from zero. The patch-wise results are exploratory, and the authors admit the metric is sub-optimal, so those numbers carry little weight.\n\nProportionately: these are real but addressable flaws. The central qualitative conclusion -- that GP uncertainty has only weak-to-moderate monotonic association with error -- may survive once a is fixed, but the paper as written leaves it conditional on an unstated hyperparameter.\n\nFor whom: anyone working on uncertainty-driven error assessment in image-guided interventions. The paper is a reasonable pilot, not a definitive negative result.\n\nRecommendation: send it to peer review, but require the authors to report a and run a sensitivity analysis, plus some measure of uncertainty on the Spearman values. That would turn a conditional pilot into a useful benchmark.","headline":"A genuinely new empirical question about GP registration uncertainty and error, but the missing kernel length-scale parameter makes the headline correlations non-reproducible and the qualitative claim conditional.","tokens_in":8105,"tokens_out":2394,"would_cite":false,"duration_ms":26867,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Gaussian process registration uncertainty and non-rigid registration error are weakly to moderately monotonically correlated, and the association varies by how landmarks are chosen.","keywords":["registration uncertainty","registration error","Gaussian process registration","Spearman's rank correlation","image-guided neurosurgery","probabilistic image registration","brain shift","ultrasound registration"],"falsifier":"Recompute the point-wise Spearman's rho on the same landmarks while varying the kernel length-scale $a$ over a wide range; if the correlation swings from 0.29-0.40 to near zero or negative, the weak-to-moderate association is an artifact of the unstated kernel calibration rather than a robust property of GP registration uncertainty.","tokens_in":7246,"feed_emoji":"🧠","tokens_out":7644,"duration_ms":150901,"temperature":0.7,"pith_summary":"This paper asks a question clinicians implicitly rely on: when a probabilistic registration reports high uncertainty at a point, is the registration error at that point also large? Using Gaussian process registration on intraoperative ultrasound from 23 neurosurgery cases, the authors measure the monotonic association between point-wise transformation uncertainty and landmark-based registration error. They find Spearman rank correlations of about 0.29 for manual landmarks and 0.40 for automatically detected landmarks, a weak-to-moderate positive relationship, while patch-wise correlations using appearance-based error metrics stay consistently low. The paper concludes that at this stage using GP uncertainty as a surrogate for non-rigid registration error is not yet feasible in practice, and the question deserves further study.","feed_headline":"GP uncertainty only weakly tracks registration error","feed_subtitle":"Point-wise correlations hit 0.29-0.40; patch-wise links stay low, so uncertainty maps cannot yet flag errors.","key_machinery":"The object carrying the argument is the posterior covariance of a Gaussian process displacement field. Registration uncertainty at a voxel is the diagonal entry of $\\Sigma_* = K_{**} - K_*^T K^{-1} K_*$, computed with kernel $k(x,x') = \\exp(-x^2/a)$; larger posterior variance means less confidence in the interpolated displacement. The authors compare this uncertainty to error using Spearman's rank correlation $\\rho_s$, chosen because it tests monotonic rather than linear relationships and is robust to outliers. The comparison is done point-wise, using known landmark displacements as ground truth, and patch-wise, using appearance-based dissimilarity metrics (histogram intersection and SSD) as error surrogates.","core_discovery":"The central claim is that there is a positive monotonic association between point-wise GP registration uncertainty and non-rigid registration error, but it is weak-to-moderate, not strong enough to support clinical use as a surrogate. In the point-wise posterior predictive checking experiment, the mean Spearman's rho is 0.2899 for manual landmarks from the RESECT dataset and 0.4014 for automatically extracted landmarks from the MIBS dataset. In the patch-wise experiment, where error is approximated by histogram intersection and sum of squared differences over image patches, the correlations are consistently low for both datasets. The authors infer that transformation uncertainty and appearance-based error may be inherently weakly coupled, and that the distribution of landmarks matters for the strength of the association.","pith_inferences":["The paper studies only Gaussian process registration; other probabilistic registration approaches may show a different uncertainty-error association, so the practical verdict should not be read as a general statement about registration uncertainty.","Because the kernel length-scale is not reported, varying it in a follow-up experiment would show how much of the measured correlation is intrinsic to GP uncertainty and how much is an artifact of calibration.","A synthetic phantom with a known dense deformation field could test the same question without relying on sparse landmarks or appearance-based patch error, giving a cleaner ground-truth comparison.","The weak patch-wise correlation may reflect the difference between transformation uncertainty and label uncertainty; a model that estimates label uncertainty directly might yield clinically useful error maps where this one does not."],"forward_implications":["Surgeons cannot yet treat GP uncertainty maps as reliable indicators of where non-rigid registration may be wrong, particularly in patch-level assessments.","Point-wise uncertainty carries some ranking information about error, but the strength is too low to set confidence thresholds for clinical decisions.","The higher correlation for automatic landmarks suggests that the spatial distribution of features used for interpolation modulates the uncertainty-error link.","Registration uncertainty should not be used as a proxy for error without further calibration or validation against dense ground-truth deformation."],"supporting_citations":[{"why":"Introduces probabilistic diffeomorphic registration with uncertainty, one of the probabilistic registration foundations whose uncertainty concept the study adopts.","marker":"[20]"},{"why":"Describes the feature-driven GP-based active framework for ultrasound brain-shift compensation that supplies the registration setup used in this study.","marker":"[27]"},{"why":"Investigates feature-based non-rigid registration with Gaussian processes, another basis for the GP interpolation used here.","marker":"[29]"},{"why":"Establishes the distinction between transformation uncertainty and label uncertainty, which motivates limiting the study to transformation uncertainty.","marker":"[31]"},{"why":"Provides the definition and rationale for Spearman's rank correlation as a non-parametric monotonic association measure.","marker":"[32]"},{"why":"Supplies the posterior predictive checking framework that the point-wise experiment adapts to compare predicted and observed landmark locations.","marker":"[33]"},{"why":"Provides the RESECT public clinical dataset with manual landmarks used in the point-wise experiment.","marker":"[34]"},{"why":"Provides the automatic feature detection and matching method used to generate landmarks in the proprietary MIBS dataset.","marker":"[35]"},{"why":"Shows that histogram intersection is an effective local image dissimilarity measure for registration quality, which is used as the patch-wise error surrogate.","marker":"[36]"}],"fun_headline_variants":["GP uncertainty weakly tracks error, not a safe surrogate","Uncertainty-error link too weak for neurosurgery use","Point-wise GP uncertainty: weak positive link to error","GP uncertainty and error: monotonic but weak link"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Gaussian process posterior variance used as 'registration uncertainty' is a faithful measure of confidence; the kernel length-scale $a$ is not reported, so if a different $a$ re-ranks the uncertainties, the reported correlations could be an artifact of calibration rather than a property of the method.","fun_headline_variants_meta":{"raw":{"variants":["GP uncertainty weakly tracks error, not a safe surrogate","Uncertainty-error link too weak for neurosurgery use","Point-wise GP uncertainty: weak positive link to error","GP uncertainty and error: monotonic but weak link"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000242,"raw_usage":{"total_tokens":1502,"prompt_tokens":899,"completion_tokens":603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":539}},"tokens_in":515,"tokens_out":603,"duration_ms":110039,"temperature":1.0,"reasoning_tokens":539,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:58:00.342109+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the point-wise Spearman's rho on the same landmarks while varying the kernel length-scale $a$ over a wide range; if the correlation swings from 0.29-0.40 to near zero or negative, the weak-to-moderate association is an artifact of the unstated kernel calibration rather than a robust property of GP registration uncertainty.","supporting_citations":[{"cited_title":"In: WBIR’14","cited_arxiv_id":null,"evidence_quote":"Introduces probabilistic diffeomorphic registration with uncertainty, one of the probabilistic registration foundations whose uncertainty concept the study adopts."},{"cited_title":"In: MICCAI’18","cited_arxiv_id":null,"evidence_quote":"Describes the feature-driven GP-based active framework for ultrasound brain-shift compensation that supplies the registration setup used in this study."},{"cited_title":"et al.: Investigation of Feature-Based Nonrigi d Image Registration Using Gaussian Process","cited_arxiv_id":null,"evidence_quote":"Investigates feature-based non-rigid registration with Gaussian processes, another basis for the GP interpolation used here."},{"cited_title":"and Frisken, S.: O n the applicability of registration uncer- tainty","cited_arxiv_id":null,"evidence_quote":"Establishes the distinction between transformation uncertainty and label uncertainty, which motivates limiting the study to transformation uncertainty."},{"cited_title":"and Foreman, D.I.: Nonparametric Statisti cs: A Step-by-Step Ap- proach, Wiley","cited_arxiv_id":null,"evidence_quote":"Provides the definition and rationale for Spearman's rank correlation as a non-parametric monotonic association measure."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the posterior predictive checking framework that the point-wise experiment adapts to compare predicted and observed landmark locations."},{"cited_title":"Phys.4 4(7): 3875-3882 (2017)","cited_arxiv_id":null,"evidence_quote":"Provides the RESECT public clinical dataset with manual landmarks used in the point-wise experiment."},{"cited_title":"Int J CARS, 13 (10): pp.1525-1538","cited_arxiv_id":null,"evidence_quote":"Provides the automatic feature detection and matching method used to generate landmarks in the proprietary MIBS dataset."},{"cited_title":"et al.: Visualization of deformable ima ge registration quality using local image dissimilarity","cited_arxiv_id":null,"evidence_quote":"Shows that histogram intersection is an effective local image dissimilarity measure for registration quality, which is used as the patch-wise error surrogate."}],"review_version":1}