{"id":"09082ac5-eaed-47b4-9ec6-9905bd910081","arxiv_id":"2505.09197","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A Bayesian framework generalizes imaging biomarker repeatability analysis to non-normal (Dirichlet-Multinomial) biomarkers and, applied to whole-body diffusion MRI of metastatic prostate cancer, estimates an approximately 70% individual-tumor response rate across two studies.","lead":"Researchers propose a Bayesian statistical framework for measuring the precision of imaging biomarkers, including non-numeric biomarkers like tumor habitat maps, and use it to detect which individual bone tumors respond to treatment in whole-body MRI of prostate cancer patients. Applied to two small studies of metastatic prostate cancer, the framework estimates that about 70% of individual tumors respond to therapy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Transfer of double-baseline measurement error from Study 3 to Studies 1/2 is untested and internally flagged (Sec 4.3), directly threatening the ~70% response-rate claim.","rationale":"The methodology is coherent: the analytic equivalence to Bland-Altman is proven (Theorem 7.1), the Dirichlet-Multinomial extension is plausible, and the simulation study supports the implementation. The most vulnerable link in the clinical application is not the math but the inference that Study 3's test-retest noise applies to Studies 1 and 2. The acquisition differences and the self-reported τ discrepancy in Section 4.3 make this a real, testable risk to the 70% response rate rather than a purely speculative concern. Because the reader's verdict already conditions acceptance on reproducibility and tempering claims, my independent read supports the conditional verdict; no new verdict change is needed.","tokens_in":27898,"tokens_out":10142,"duration_ms":103938,"concrete_test":"Refit the habitat mixture model (Stan code in Supplementary 7.4) for Study 1 and Study 2 after rescaling the repeat-baseline precision: set τ fixed at the Study 3 posterior median multiplied by f for f in {0.5, 1, 2}, and likewise scale the prior scale (prec_prior) so the data still inform τ under the perturbed model. Record the posterior median of λ under each f. If the median moves by more than ±0.05 from the reported ~0.70, the response-rate claim is sensitive to the transferability assumption and the conclusion should be conditional on an external repeatability study with the treatment-cohort protocols. Run 3 chains of 5000 post-burn-in samples and confirm R-hat < 1.05.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline result — 'approximately 70% response rate among individual tumors across both studies' — is the posterior median of the mixing weight λ in the mixture model of Section 2.5. The null model M0 determines each lesion's posterior odds through the Dirichlet-Multinomial measurement precision τ, which is estimated exclusively from the double-baseline cohort (Study 3: 10 patients, 73 lesions, Section 3.2). Study 3 was acquired with 6 mm slice thickness, in-plane resolution 1.7 mm², and three b-values; Study 1 used 5 mm slices with some patients on two b-values; Study 2 used 5 mm slices at 1.3–1.5 mm² in-plane resolution. Lesion volumes are also very different (median 5.1 ml for Study 3 vs 12.5 and 22.0 ml for Studies 1 and 2). Section 2.5 itself states the double-baseline data 'should be representative of the biomarker measurements made within the main study,' but no evidence or sensitivity analysis supports this representation for habitat proportions. The paper provides internal evidence that the assumption may fail. In Section 4.3 (Figure 11), the prior-sensitivity analysis reports that for habitat biomarkers in Study 1, the posterior of the measurement precision τ shifts when post-treatment data are included, 'suggest[ing] some misalignment between the assumed model and the acquired post-treatment data.' The authors recommend relying on repeat-baseline-only estimates — but that is exactly the Study 3-derived estimate whose transferability to Studies 1 and 2 is unverified. If the actual measurement error in the treatment cohorts is larger or smaller than Study 3's, every M0 likelihood is misspecified, biasing PO10 and λ. Thus the ~70% figure and the per-lesion classifications rest on an unvalidated cross-protocol assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Bayesian framework for imaging-biomarker repeatability that extends conventional Bland-Altman-style analysis to biomarkers that are not univariate normal, in particular to Dirichlet-Multinomial distributed 'habitat' proportions on the simplex. The authors derive an analytical link between Bayesian anomaly detection and the conventional repeatability coefficient, propose posterior odds (PO10) via Bayesian mixture modeling to classify individual lesions as responding or not, and apply the framework to whole-body diffusion-weighted MRI data from two mCRPC treatment cohorts (Studies 1 and 2) using a separate double-baseline cohort (Study 3) to estimate measurement error. They report an approximately 70% response rate among individual tumors across both studies, based on the posterior median of the mixing weight λ. The manuscript includes simulation studies, a prior sensitivity analysis, and Stan code in the supplementary material.","tokens_in":28259,"tokens_out":4680,"duration_ms":44043,"significance":"If the methodological and clinical claims are valid, the paper provides a genuinely useful generalization of repeatability assessment to non-normal, simplex-valued imaging biomarkers, with a principled Bayesian alternative to Bland-Altman for such data. The theoretical derivations in the supplement are internally consistent, and the simulation study demonstrates low bias and good coverage in most parameter regions, which supports the computational implementation (modulo the Stan-code issues noted below). The framework also gives a concrete, interpretable way to quantify inter-lesion response heterogeneity, which is clinically relevant to mCRPC. However, the key clinical quantity -- the 70% response rate -- rests on an untested assumption about the transferability of measurement error across cohorts, and the printed Stan code for the habitat model is not syntactically executable as written; these issues currently limit the reliability and reproducibility of the central claims.","major_comments":[{"comment":"The transferability of the double-baseline measurement error from Study 3 to Studies 1 and 2 is a load-bearing premise for the per-lesion posterior odds and the reported ~70% response rate. The three studies differ in slice thickness (6 mm vs 5 mm), in-plane resolution, b-value sampling, and lesion volume (median 5.1 ml vs 12.5 and 22.0 ml), yet Section 2.5 only asserts that the double-baseline data 'should be representative' without providing evidence or a sensitivity analysis. Moreover, the paper's own prior-sensitivity analysis in Section 4.3 (Figure 11) shows that for habitat biomarkers in Study 1, the posterior of the measurement precision τ shifts when post-treatment data are included, which the authors attribute to 'some misalignment between the assumed model and the acquired post-treatment data.' The recommendation to rely on repeat-baseline-only estimates does not resolve the issue because those estimates come from a different cohort. I ask the authors to provide a direct assessment of how the posterior odds and λ estimates would change under plausible perturbations of τ (e.g., a sensitivity interval over τ), or to present an empirical demonstration that measurement repeatability is similar across the three acquisition protocols.","section":"Sections 3.2 and 2.5; Figure 11"},{"comment":"The Stan code for the Dirichlet-Multinomial habitat model is syntactically malformed and, as printed, cannot be executed. The custom function `dir_mult` is declared with a 2D array argument (`array[,] int y`) and separate `vector mu` and `real prec` arguments, but in both the model block and the generated-quantities block it is called with `yp[n]`, which is a 1D array of length K, and with a single expression `mu0 * prec` (a vector) rather than separate `mu` and `prec` arguments. Additionally, the likelihood statement `yb ~ dirichlet_multinomial(mu0, prec);` does not reference the custom `dir_mult_lpmf`; if it relies on Stan's built-in `dirichlet_multinomial` distribution, the custom function is redundant and the inconsistency still prevents the code from running as written. Because the habitat results are a central contribution, the authors must correct the code and confirm that the reported results are reproducible from the provided model code.","section":"Section 7.4 (Supplementary Stan code)"},{"comment":"The headline 'approximately 70% response rate among individual tumors across both studies' is stated without any uncertainty quantification. The posterior distribution of λ is shown in Figure 9, but the text reports no numerical estimate, credible interval, or posterior predictive check for this quantity. Given that the 70% figure is a posterior median based on the mixture model, the authors should report the posterior median and 95% credible interval for λ for each study and each biomarker type, and should moderate the claim if the posterior is wide or if the credible interval includes values far from 0.7.","section":"Abstract and Section 4.2"}],"minor_comments":[{"comment":"The manuscript contains numerous typographical and spelling errors, including 'Hamiltonean', 'disitributed', 'paramters', 'lections', 'seperated' (Figure 7 caption), 'respoonse', and 'depcted'. A careful proofread is needed before resubmission.","section":"Throughout"},{"comment":"The caption of Figure 6 states 'median ADC in this instance' even though the figure describes the Dirichlet-Multinomial habitat model; this is misleading and should be corrected.","section":"Section 4.1 / Figure 6"},{"comment":"The color-coding description in the text refers to bins defined by posterior odds values, but the last bin is labeled 'BF10 > 10 (dark green)' while the rest are labeled with PO10; for consistency, all bins should be labeled as PO10.","section":"Section 4.2 / Figure 7"},{"comment":"The assumption that the baseline habitat proportion is known and fixed at μ0 = (0.1, 0.8, 0.1) is ad hoc; please add a sentence acknowledging this as a modeling assumption and, if possible, report a sensitivity analysis or at least describe how deviations from this prior value would affect inference.","section":"Section 2.6"},{"comment":"In the simulation study, the measurement precision τ is fixed to 11.54 and the voxel-size parameters are fixed to values derived from the clinical studies; because the transferability of τ across cohorts is a central concern (see Major Comment 1), the simulation design should include a grid over τ or otherwise acknowledge this limitation.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid methodological contribution with clean theory and a clinically relevant application, but the two load-bearing issues -- the untested transfer of measurement error from Study 3 to Studies 1/2 and the non-executable Stan code for the habitat model -- need to be addressed before the central claims can be accepted. The referee report focuses on these points; the authors should also be encouraged to report posterior uncertainty for λ explicitly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The genuinely new piece is the Dirichlet-Multinomial repeatability model for simplex-valued habitat biomarkers, with a mixture model that yields per-lesion posterior odds and a posterior mixing weight λ. That fills a recognized gap in the imaging biomarker roadmap. The analytical equivalence between Bland-Altman and Bayesian novelty detection under normality (Theorems 7.1, 7.1.1) is clean and correctly derived, though mathematically elementary. The simulation studies are honest: low bias, reasonable coverage in most regions, with expected degradation near the identifiability boundary (μΔ→0, η→0.5). Credit where due: the paper ships derivations and code (with a caveat) and is self-aware about some limitations.\n\nNow the soft spots. First, the headline clinical number—~70% lesion-level response rate across both studies—is the posterior median of λ, and every lesion's likelihood under M0 inherits the measurement precision τ estimated from Study 3, the double-baseline cohort. Study 3 used 6mm slices, 1.7mm² in-plane, median lesion volume 5.1ml, while Studies 1/2 used 5mm slices, different resolutions, and much larger lesions. The paper states in Sec 2.5 that the repeat-baseline data 'should be representative' but offers no sensitivity analysis. Worse, Sec 4.3 (Fig 11) reports that for habitat biomarkers in Study 1, including post-treatment data shifts the posterior of τ, and the authors recommend relying on repeat-baseline-only estimates—exactly the ones whose transfer is unverified. That's not a refutation of the method, but it's a real robustness gap in the clinical claim. The abstract's 'validating the clinical relevance' overstates two small cohorts with no external outcomes.\n\nSecond, the printed Stan code for the DM model looks syntactically off: `dir_mult_lpmf` is defined for a 2D array but called on rows `yp[n]` with a single precision argument; as printed it wouldn't compile. That's a reproducibility issue, not a conceptual one, but it matters for a methods paper.\n\nThird, a minor point: the posterior odds are estimated via categorical sampling of labels, which can be noisy; the correlation analysis caps PO10 at 80,000, a detail the paper notes but could handle more transparently.\n\nVerdict: the methodological core holds up. The framework is a useful generalization, and the DM repeatability model is the actual contribution. The clinical application is a proof-of-concept that needs validation of the transfer assumption before the response-rate claim is taken at face value. I'd send this to a serious ref in imaging biomarkers or Bayesian methods; it deserves referee time. I'd cite the DM model if I work in this area. Bring to reading group? Yes.","headline":"Genuinely new simplex-biomarker repeatability model and a clean BA-equivalence proof, but the ~70% response-rate claim leans on unvalidated transfer of measurement error across scanner protocols.","tokens_in":28842,"tokens_out":2789,"would_cite":true,"duration_ms":25937,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes a Bayesian framework that generalizes imaging-biomarker repeatability studies to non-normal biomarkers, and reports an ~70% individual-tumor response rate in two mCRPC cohorts.","keywords":["Bayesian inference","imaging biomarkers","repeatability","Dirichlet-Multinomial","habitat mapping","diffusion-weighted MRI","metastatic prostate cancer","response heterogeneity"],"falsifier":"Acquire true test-retest data using the exact acquisition parameters of Study 1 and Study 2 (same b-values, in-plane resolution, and slice thickness), fit the same null model, and compare the resulting posterior predictive credible regions with those derived from Study 3. If the regions differ materially, or if lesions known by an independent clinical or biological marker to be stable are classified as responders at a rate above the nominal level, the transfer of measurement error from Study 3 is the breaking point.","tokens_in":1490,"feed_emoji":"🩻","tokens_out":1906,"duration_ms":80255,"temperature":0.7,"pith_summary":"The paper proposes a Bayesian framework for judging whether an imaging biomarker changed between two scans, replacing the usual assumption that biomarkers are single, normally distributed numbers. It frames each lesion's response as a choice between a null model (no true change beyond measurement noise) and an alternative model (change), and derives posterior odds for each lesion from a mixture model. Applied to whole-body diffusion-weighted MRI of metastatic castrate-resistant prostate cancer, the method classifies individual bone lesions and reports that roughly 70% of tumors responded, while the rest showed no evidence of change or changed in the opposite direction. The authors' aim is to make repeatability assessment, and per-lesion response classification, available for any biomarker type, including habitat proportions on the unit simplex.","feed_headline":"Bayesian scan method finds 70% of prostate tumors respond","feed_subtitle":"A Bayesian repeatability framework classifies each lesion as responding or not and puts a number on response heterogeneity.","key_machinery":"The load-bearing object is the posterior predictive distribution under the null model M0, written as p(y|y0, yb, M0), approximated by Hamiltonian Monte Carlo samples and turned into a credible region by a kernel-distance criterion. Posterior odds PO10 come from a Bayesian mixture model over lesion labels (change versus no change), with a mixing weight lambda that estimates the population response rate. For habitat biomarkers the likelihood is a Dirichlet-Multinomial model on unnormalised voxel counts, with baseline proportions fixed at (0.1, 0.8, 0.1) from ADC percentiles; this is what lets the framework handle bounded, compositional biomarker data.","core_discovery":"On the paper's own terms, the central discovery is that conventional repeatability analysis is a special case of Bayesian novelty detection: with normally distributed biomarkers and intraclass correlation tending to one, the posterior predictive null distribution reduces to the classical Bland-Altman result, so the Bayesian credible region and the Repeatability Coefficient coincide. Once this is in place, the same machinery extends to arbitrary biomarker likelihoods. For habitat biomarkers, which are counts of voxels falling into ADC-defined tissue classes, the paper models the null and alternative cases with Dirichlet-Multinomial distributions and estimates posterior odds for every lesion via Bayesian mixture modeling. In two mCRPC treatment studies, the posterior odds reveal inter- and intra-lesion heterogeneity and yield an approximately 70% response rate among individual tumors.","pith_inferences":["An implication the paper leaves implicit is that the 70% figure is a posterior summary of the two cohorts, not a universal response rate; applying it to a new trial requires re-estimating the measurement-error and mixing parameters for that trial's protocol.","Because the paper notes that including post-treatment data changed the habitat measurement-precision estimate in Study 1, a simulation study varying the baseline proportions and precision priors would show how sensitive the 70% and per-lesion labels are to model misspecification.","A natural next step, which the paper lists as future work, is to add per-patient response-rate parameters lambda_i; one could then correlate those per-patient posterior distributions with survival endpoints such as progression-free or overall survival.","The same anomaly-detection machinery could be applied to other constrained biomarkers, such as fat fraction bounded on [0,1] or count-based radiomic features, by substituting the appropriate likelihood and rerunning the repeatability study."],"forward_implications":["Repeatability studies no longer need to assume real-valued normal biomarkers; any biomarker with a defined likelihood, such as counts, proportions, or bounded values, can get a credible region and a per-subject response decision.","Classical Bland-Altman repeatability coefficients and the Student-t form with Nb degrees of freedom emerge as limiting cases, so the framework is backward-compatible with existing precision results.","Posterior odds computed from the mixture model give a per-lesion classifier and a population-level response-rate estimate; in the two mCRPC studies this rate is about 70%.","Because posterior odds can exceed the 95% specificity ceiling of p-value-based testing, lesion classifications can in principle be more specific than conventional thresholding.","Habitat-style simplex biomarkers, previously without a repeatability method, can be monitored in the same decision framework."],"supporting_citations":[{"why":"Defines the Bland-Altman repeatability analysis and Repeatability Coefficient that the Bayesian framework reproduces as a special case.","marker":"Bland and Altman [1986]"},{"why":"Supplies the standard treatment of measurement error and change interpretation for quantitative imaging biomarkers.","marker":"Obuchowski [2018]"},{"why":"Establishes the imaging biomarker roadmap requiring precision evaluation at every validation stage.","marker":"O'Connor et al. [2017]"},{"why":"Provides the double-baseline repeatability study design whose data type the framework consumes as measurement-error evidence.","marker":"Winfield et al. [2017]"},{"why":"Sets the Prostate Cancer Working Group criteria that make mixed response documentation a required endpoint.","marker":"Scher et al. [2016]"},{"why":"Demonstrates whole-body DWI response assessment in bone metastases, the clinical application the method builds on.","marker":"Blackledge et al. [2014]"}],"fun_headline_variants":["Bayesian repeatability generalizes to non-normal biomarkers","Bayes extends biomarker precision to Dirichlet-Multinomial","Generalized repeatability: Bayesian inference for imaging","Heterogeneous response: Bayesian MRI analysis of prostate"],"cache_read_input_tokens":30848,"weakest_assumption_plain":"The classifications and the ~70% response rate depend on using Study 3's double-baseline data (10 patients, 73 lesions) as the measurement-error model for Studies 1 and 2, even though the acquisition protocols differ; if the noise in the treatment studies is not the same, every posterior odds estimate is miscalibrated.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian repeatability generalizes to non-normal biomarkers","Bayes extends biomarker precision to Dirichlet-Multinomial","Generalized repeatability: Bayesian inference for imaging","Heterogeneous response: Bayesian MRI analysis of prostate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00088,"raw_usage":{"total_tokens":3815,"prompt_tokens":965,"completion_tokens":2850,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":2787}},"tokens_in":581,"tokens_out":2850,"duration_ms":23399,"temperature":1.0,"reasoning_tokens":2787,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:38:05.770399+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Acquire true test-retest data using the exact acquisition parameters of Study 1 and Study 2 (same b-values, in-plane resolution, and slice thickness), fit the same null model, and compare the resulting posterior predictive credible regions with those derived from Study 3. If the regions differ materially, or if lesions known by an independent clinical or biological marker to be stable are classified as responders at a rate above the nominal level, the transfer of measurement error from Study 3 is the breaking point.","supporting_citations":[],"review_version":1}